Policy Mesh

What it is

Policy Mesh is an open-source AI control plane designed to provide governed, observable, and policy-driven access to both local and cloud-based Large Language Models.

Core Goal

Enable self-service AI consumption without sacrificing governance, auditability, or cost control.

Why I built it

Throughout my career, I have been drawn to platform engineering challenges where centralized capabilities reduce complexity and improve consistency across teams.

Design Goals

• Governance by default: Policies should be enforced automatically rather than relying on developers to remember rules and procedures.
• Provider abstraction: Applications should interact with a consistent interface regardless of the underlying AI provider.
• Explainability: Routing decisions should be understandable, traceable, and auditable.
• Cost awareness: Organizations should be able to optimize AI usage based on cost, performance, and capability requirements.
• Security and privacy: Sensitive requests should be routed according to organizational data handling requirements.
• Observability: AI consumption should be visible, measurable, and reportable.

The Problem

Without a centralized control plane, AI adoption pushes routing, governance, and provider decisions into individual applications.

• Inconsistent governance: Different teams implement different controls, leading to fragmented policy enforcement.
• Cost sprawl: Applications may default to expensive models even when less costly alternatives are sufficient.
• Limited visibility: Organizations struggle to understand how AI services are being consumed.
• Provider lock-in: Applications become tightly coupled to specific model vendors.
• Duplicate logic: Every application must independently implement routing, governance, and provider integrations.

The Solution

Policy Mesh acts as an intelligent decision layer between applications and AI providers. Applications submit requests through a single API, and the control plane evaluates the request before selecting the destination.

• Data sensitivity.
• Request characteristics.
• Cost considerations.
• Organizational policies.
• Provider capabilities.

Applications remain unaware of the routing logic while governance remains centralized and consistent.

Architecture

Policy Mesh is built as a modular control plane with distinct responsibilities.

• API layer: FastAPI and OpenAPI provide request intake, validation, authentication, and response normalization through a unified interface.
• Policy engine: Policy evaluation, routing decisions, governance enforcement, and explainable decision generation determine how each request should be handled.
• Provider layer: Provider integrations abstract OpenAI, Anthropic, Ollama, and future model targets without requiring application changes.
• Audit layer: Decision logging, request metadata, audit trails, and governance reporting provide traceability while avoiding unnecessary storage of sensitive request content.
• Observability layer: Prometheus metrics and structured logging provide request tracking, usage visibility, cost insight, and operational monitoring.

Current Capabilities

• Policy-based AI request routing.
• Local and cloud model support.
• Provider abstraction.
• Audit logging.
• Explainable decisioning.
• Cost-aware routing.
• API-driven consumption.
• Observability and metrics.

Example Routing Scenarios

• Sensitive internal data: Requests containing sensitive business data can be routed to approved local models so data remains within organizational boundaries.
• Cost optimization: General-purpose requests can be routed to lower-cost models unless advanced capabilities are required.
• Provider resilience: Requests can fall back to alternative providers when primary providers become unavailable.

Roadmap

• Dynamic policy management: External policy definitions, policy versioning, and policy testing.
• Cost analytics: Spend reporting, cost optimization recommendations, and consumption forecasting.
• Compliance reporting: Governance dashboards, audit reporting, and policy compliance metrics.
• AI platform services: Model catalogs, self-service AI onboarding, developer portals, and usage scorecards.
• Multi-model orchestration: Workflow chaining, capability-based routing, and agent support.

Key Learnings

• AI governance is a platform problem: Many AI adoption challenges are fundamentally platform engineering challenges rather than model challenges.
• Centralization reduces complexity: Moving governance decisions into a shared control plane simplifies application development and improves consistency.
• Explainability matters: Organizations need visibility into why AI decisions are being made, not just the outcomes.
• Cost becomes a platform concern: As AI adoption grows, routing and governance decisions increasingly affect organizational spending.
• Platform patterns apply to AI: The same principles used in cloud platforms, Internal Developer Platforms, and self-service infrastructure can be applied to AI consumption.

Why it matters