OrchestAI: Enterprise Multi-LLM Orchestration with Signed Audit and On-Premise Deployment

Enterprise AI adoption faces a governance gap. Organizations deploy AI tools ad-hoc — a department uses ChatGPT here, another uses Claude there, a third experiments with open-source models on personal machines. There is no central visibility into what AI agents are doing, what data they process, which provider handles which request, or how much it all costs. When the CISO asks 'can you prove what your AI did last Tuesday?', the answer is silence. When the CFO asks 'how much are we spending on AI?', no one knows. This is shadow AI — and for regulated enterprises, it is an existential compliance risk.

The Challenge

A regional utility company with over 1,500 employees identified three critical problems with their AI adoption strategy: First, shadow AI — departments were independently subscribing to AI services with no central governance, creating data leakage risks and compliance blind spots. Second, vendor lock-in — early AI investments were tightly coupled to a single provider, creating business continuity risks and preventing cost optimization. Third, audit requirements — their regulatory framework demanded demonstrable proof that AI systems were operating within defined boundaries, with tamper-evident records of every decision and action.

Existing solutions addressed one or two of these problems, but never all three together. Multi-LLM routers existed but lacked audit capabilities. Compliance platforms existed but forced cloud deployment. Governance tools existed but did not support on-premise installation. The organization needed a single platform that combined all three — and that platform did not exist.

The Solution: OrchestAI Architecture

OrchestAI was designed from the ground up to solve the governed enterprise AI problem. The architecture rests on four pillars:

• Multi-LLM Router: Every AI request passes through a central router that selects the optimal provider based on configurable strategies — cheapest, local-first, cloud-first, or preferred — with automatic fallback. Applications connect to OrchestAI's unified API and never need to know which provider is serving their request.
• Signed Audit Chain: Every agent action is signed with HMAC-SHA256 and chained to the previous action. This creates a tamper-evident log that external auditors can verify independently. If any entry in the chain is modified after the fact, the cryptographic chain breaks and the tampering is immediately detectable.
• On-Premise Deployment: The entire platform — API (FastAPI), frontend (Next.js), database (PostgreSQL), vector store (ChromaDB) — deploys via docker-compose or Helm/Kubernetes on the client's own infrastructure. No data ever leaves their datacenter.
• Versioned Agent Catalog: Agents are managed with promotion gates (dev → staging → prod), evaluation benchmarks before promotion, and continuous canary deployments every 15 minutes. This brings software engineering discipline to AI operations.

Technical Architecture

The OrchestAI platform comprises 22 SQLAlchemy models across 39 database tables. The backend is Python/FastAPI with full type safety. The frontend is Next.js for the management dashboard. PostgreSQL handles relational data while ChromaDB provides vector storage for agent memory persistence and retrieval. The system supports SSO via OIDC with domain allowlist, and all credentials are encrypted with Fernet AES.

The testing infrastructure includes approximately 574 pytest tests covering the backend API and business logic, plus 41 Playwright end-to-end tests validating the full user journey from agent creation through deployment and audit verification. Backup/restore procedures have been drilled with a tested RTO of approximately 30 seconds.

Business Case: Utility Profile

For a regional utility with 1,500 employees using AI across operations, customer service, and predictive maintenance, the economics are compelling. Unmanaged multi-provider AI spend typically runs USD 8,000 or more per month. With OrchestAI's intelligent routing — particularly the local-first strategy that routes appropriate requests to on-premise Ollama models instead of expensive cloud APIs — organizations can expect to reduce AI infrastructure costs by 60-70%, bringing monthly spend down to USD 2,000-3,000 while maintaining the same or better service quality for each use case.

Results and Current Status

• Multi-LLM routing across Claude, GPT, Gemini, and Ollama with configurable strategies and automatic fallback
• HMAC-SHA256 signed audit chain — tamper-evident, independently verifiable by external auditors
• Full on-premise deployment via docker-compose or Helm/Kubernetes — data never leaves the client datacenter
• Versioned agent catalog with promotion gates (dev → staging → prod) and continuous canary deployments every 15 minutes
• Per-agent cost quotas with real-time dashboards and alerts for budget governance
• 60-70% estimated cost reduction through intelligent provider routing
• Audit compliance achievable in approximately 1 day with the signed chain evidence
• ~30s tested backup RTO for business continuity

Client Reference

OrchestAI is currently in active pilot preparation with EPEC (Empresa Provincial de Energia de Cordoba), a regional utility company in Argentina. The demo has been approved and the pilot phase is being structured. EPEC represents the ideal OrchestAI customer profile: a large regulated enterprise with distributed AI adoption, strict audit requirements, and a clear need for on-premise deployment to maintain data sovereignty.

Differentiator

OrchestAI is the only platform that combines versioned agents + multi-LLM routing + signed audit chains + on-premise deployment in a single product. Competitors offer one or two of these capabilities, but never all four together. This combination is what regulated enterprises need — and it is what OrchestAI was purpose-built to deliver.