← All posts

How enterprises are wiring AI in 2026: models, AI gateways, MCP, A2A, LLMOps and DLOps

2026-10-13

Originally published on LinkedIn. The two Kubernetes diagrams in section 3 are shown here in their full-size draw.io versions. Two years ago most AI conversations started with "which model should we use?". Today, in most of the teams I work with or talk to, the model is the easy part. Projects stall on the plumbing: who is allowed to call what, how tools and data are exposed, how agents talk to each other, how a prompt change is tested before it reaches a customer, and who pays for the GPUs.

I have been on both sides of that plumbing. At IAG I led the architecture of an AI assistant for aircraft operations: an ETL pipeline with AI steps for entity extraction from technical documentation, RAG on Amazon Aurora PostgreSQL with pgvector, hierarchical LangGraph agents, and Langfuse for tracing and evaluation. At Multiverse Computing I led up to four teams building custom LLMs and SLMs for clients, compressing models so they are cheaper to run and fit on devices, and contributed to the definition and design of the benchmark suite used to evaluate them before release.

This article is the map I wish I had at the start: the reference architecture that keeps repeating across sectors, the protocols and gateways that glue it together, how the emphasis changes by industry, a complete worked example for an insurer on Azure, and the two "Ops" disciplines that decide whether any of it survives production: LLMOps and DLOps.

1. One reference architecture, five layers

Reference architecture

Whether it is a bank, a hospital group or an airline, new AI systems converge on the same five layers:

  1. Channels and identity. Web and mobile apps, copilots inside Teams, Slack, the IDE or the CRM, APIs and events, and back-office batch jobs. All of them go through the corporate API gateway and SSO first. The key question here is "who is calling, and on whose behalf?", because an agent acting for a user must inherit that user's permissions, not a super-user's.
  2. Agents and orchestration. An agent runtime with explicit state: LangGraph, Google ADK, Microsoft Agent Framework, the OpenAI Agents SDK or Bedrock AgentCore. Typical building blocks are a supervisor/router that delegates to specialised agents, checkpointed state in Postgres or Redis, and human approval for risky actions.
  3. The AI gateway. One control plane for every model call, tool call and agent call (more on this below).
  4. Capabilities. Models (frontier APIs such as Azure OpenAI, Bedrock and Vertex AI, open-weight models and SLMs served on your own GPUs, and compressed models at the edge), context (RAG over pgvector, Azure AI Search or OpenSearch, plus MCP servers in front of CRM, ERP and data products) and other agents reached through A2A.
  5. Foundation. The data platform (lakehouse, ETL with AI steps, embedding pipelines, data contracts) and the operational disciplines: LLMOps and, when you own models, DLOps.

And a column that cuts across everything: observability, evaluation, guardrails, agent identity, FinOps and governance. In my experience this column, not the model, is what decides whether a pilot becomes a product.

2. MCP is vertical, A2A is horizontal

MCP vs A2A

The two protocols people mix up most often solve different problems:

In practice you need both: MCP to give an agent hands, A2A to let agents delegate to each other across team and vendor boundaries. Importantly for enterprise buyers, in August 2026 A2A joined MCP under the Linux Foundation's Agentic AI Foundation, a neutral home that already lists support in major clouds and in platforms such as ServiceNow, Salesforce, Atlassian and SAP, and in frameworks like LangGraph, CrewAI, Pydantic AI, AG2 and BeeAI (A2A blog).

3. The AI gateway: one control plane, three kinds of traffic

AI gateway

The AI gateway is the API gateway idea of the last decade, made aware of tokens, tools and agents. Everything that calls a model, a tool or another agent goes through it, and it gives you:

Two of these are worth a closer look, because they are where most of the savings and most of the uptime come from.

What I see companies actually choose:

AI gateways compared

The comparison above summarises what each of them documents today. A few readings of it:

Self-hosted AI on Kubernetes, in detail: network, node pools, namespaces and pods, and the path of a request

Zoom in: draw.io viewer · full-resolution PNG · editable .drawio file

Show the original LinkedIn version of this diagram

Self-hosted models on Kubernetes

How that self-hosted stack splits the work, following the numbered path in the diagram:

Notes on the agent layer: one graph or many, A2A, kmcp, kagent and Agent Stack

The diagram draws the simplest setup: one LangGraph service and three MCP servers as ordinary workloads. Seven notes on the variations.

The same cluster with A2A between agents. The second diagram shows what changes when each agent becomes its own service. A concierge agent is the only one that faces users: it understands the request and delegates to the specialist agents (intake, coverage, fraud), each with its own Deployment, its own Agent Card and its own release cycle. The delegation goes through the AI gateway, like every other call that leaves an agent, so identity, limits and traces are applied in one place; agents of other companies are reached the same way. That route is a choice, not a rule of the protocol: A2A is plain HTTP, so agents of the same team in the same cluster can also call each other directly, service to service, with mTLS and network policies, and save a hop. The gateway is what you should not skip at the border, between teams or with other companies, where you need to control who calls whom. Here kagent manages the agents and serves their A2A endpoints; without it, each agent serves its own. Everything below the gateway (MCP servers, llm-d, KServe, vLLM) stays the same. What you pay is more moving parts: several deployments, state per agent, and a network hop where there was a function call.

Self-hosted AI on Kubernetes, with A2A between agents

Zoom in: draw.io viewer · full-resolution PNG · editable .drawio file

Show the original LinkedIn version of this diagram

Self-hosted AI on Kubernetes with A2A between agents

Routing easy requests to small models

How do easy requests end up on small models? In three ways, often combined:

  1. The application says so (the most common in enterprises): each step of the application asks for an alias such as mini or reasoning, as the LangGraph supervisor of the worked example in section 6 does. Predictable, cheap and auditable.
  2. The gateway decides with rules or a small triage model: LiteLLM's complexity router scores each prompt (length, code, reasoning markers, or a small classifier model) and sends it to a tier; Kong AI Proxy Advanced has a semantic routing algorithm.
  3. A managed, trained router behaves like one more model: Microsoft Foundry's model router picks the underlying model per prompt in Balanced, Cost or Quality mode, and Amazon Bedrock Intelligent Prompt Routing chooses between two models of the same family by predicted response quality.

Use the first as the default, and the second or third for open, heterogeneous traffic such as an employee chat, after checking with your evals that hard questions are not being sent to the small model.

A very common pairing in new projects: LangGraph for orchestration + LiteLLM or Azure API Management as the gateway + Langfuse or LangSmith for tracing and evaluation. LangGraph simply points its model client to the gateway's endpoint, so swapping providers or adding guardrails does not touch the agent code. Section 6 shows, on a concrete example, how the LangGraph supervisor and the gateway split the work.

4. Same building blocks, different emphasis by sector

Sector patterns

The layers are the same everywhere; what changes is which part has to be bullet-proof:

5. New projects vs. existing systems

Greenfield vs brownfield

Most of the AI work I see is not greenfield. The patterns differ:

Greenfield (agent-native). Start from the user journey and the decisions to automate; build on an agent runtime with explicit state; expose every tool through MCP from day one; send all model, tool and agent calls through the gateway; make evals part of CI so no prompt or model change ships without passing; and switch tracing on from the very first prototype.

Brownfield (AI on the existing estate). Wrap existing REST or SOAP services as MCP tools (several gateways can now generate MCP servers from APIs they already manage); add AI steps inside current ETL and document flows to extract, classify and validate; use an event-driven sidecar where Kafka events trigger agents and results flow back as events; embed copilots in the tools people already use; and replace manual steps one by one (the strangler pattern) while keeping the system of record. Above all, reuse the existing IAM, logging and data governance. A parallel "AI world" with its own identities and logs is the fastest way to fail an audit.

6. Worked example: an AI claims assistant for an insurer on Azure

Patterns are easier to judge on a concrete case, so here is one end to end. Imagine a mid-sized European insurer (motor and home) that already runs on Microsoft and Azure. It wants an assistant inside its mobile app and its website that can do two things: let a customer report a claim (first notice of loss), and answer coverage questions about the customer's own policy. The constraints are typical for the sector: data stays in the EU, the core policy and claims systems must not be rewritten, every decision must be auditable, a human approves anything that pays out money, and cost per conversation has to be visible.

The design is drawn twice, once per use case: the same architecture in both diagrams, with the numbered path of the request and the components that journey does not touch greyed out. The first use case touches almost every component, so it doubles as the full picture.

Use case 1: report a claim from the mobile app

The technology choices, and why

Following one interaction through the system

Interaction sequence

Use case 1, mobile: "I had a car accident." The customer opens the app, describes what happened and attaches photos. The step numbers match the first diagram.

Use case 2: coverage question on the website

Use case 2, web: "Is water damage from my neighbour covered?" A different customer asks on the website. The step numbers match the second diagram, and the greyed-out boxes show how much of the platform this journey never touches: no MCP tools, no documents, no Service Bus, no core-system writes.

Every hop emits OpenTelemetry spans with latency, tokens, cost, prompt version and evaluation score, so one trace shows the whole journey, from the tap on the phone to the core system and back.

Inside the claims assistant: who routes what

The assistant has two "routers": the LangGraph supervisor inside the Claims assistant API, and the AI gateway policies in API Management.

A question I get often: if the supervisor already decides which agent and which model handles each request, why does the design need an AI gateway as well? Because they route at different levels. The supervisor decides what work to do and who does it; the gateway decides how each model or tool call leaves the building, and under which rules. They do not replace each other; they stack.

The LangGraph supervisor is application logic, Python code inside your service:

The AI gateway is shared infrastructure, a proxy that every application and agent calls:

What the LangGraph graph looks like: flat supervisor or hierarchical?

LangGraph graph

For this claims assistant, a flat supervisor is enough: one supervisor node and a handful of workers. A hierarchical design (a top supervisor whose workers are themselves supervisor graphs, for example a claims team, a policy team and a billing team) pays off only when you have many agents across several domains, owned by different teams; each team then ships its own subgraph and the top level only routes between teams. Start flat and split into a hierarchy when one supervisor has too many workers to route between reliably. Both are still one LangGraph application; the step after that, one service per agent talking over A2A, is covered in the notes under the Kubernetes diagram in section 3.

Two design choices matter more than flat versus hierarchical:

The code below follows the steps of use case 1 above (step numbers in the comments). It is a sketch to show the structure, not production code: helpers such as latest_claim_draft and fraud_score, error handling and tests are left out.

1. Models through the AI gateway (steps 7–8 and 12–13). The supervisor and the agents never call Azure OpenAI directly; they call the gateway's OpenAI-compatible endpoint with an alias, and the gateway picks the deployment, applies token limits and fails over between regions.

import os
from langchain_openai import ChatOpenAI

GATEWAY = "https://apim.insurer.eu/ai/v1"     # OpenAI-compatible endpoint of the AI gateway
PRODUCT_KEY = os.environ["APIM_PRODUCT_KEY"]  # identifies the product: quotas, budgets, metrics

router_llm = ChatOpenAI(base_url=GATEWAY, api_key=PRODUCT_KEY, model="mini", temperature=0)
writer_llm = ChatOpenAI(base_url=GATEWAY, api_key=PRODUCT_KEY, model="reasoning")

2. Tools through the MCP gateway (steps 9–10). The core APIs are exposed as MCP servers behind APIM, and the agent calls them with the customer's own token, so it can only see that customer's policy and claims.

from langchain_mcp_adapters.client import MultiServerMCPClient

def mcp_client(customer_token: str) -> MultiServerMCPClient:
    auth = {"Authorization": f"Bearer {customer_token}"}
    return MultiServerMCPClient({
        "policy": {"transport": "streamable_http", "url": "https://apim.insurer.eu/mcp/policy", "headers": auth},
        "claims": {"transport": "streamable_http", "url": "https://apim.insurer.eu/mcp/claims", "headers": auth},
    })

3. Retrieval scoped to the customer's policy (step 11; step 6 in use case 2). The policy id comes from the validated request context, never from the model, so a prompt cannot make the agent read someone else's policy.

from dataclasses import dataclass
from azure.identity import DefaultAzureCredential
from azure.search.documents import SearchClient
from langchain.tools import tool, ToolRuntime

@dataclass
class Ctx:
    customer_id: str
    policy_id: str            # taken from the validated token by the API, not from the prompt

search = SearchClient(os.environ["SEARCH_URL"], "policy-wordings", DefaultAzureCredential())

@tool
def search_policy_wording(query: str, runtime: ToolRuntime[Ctx]) -> str:
    """Search the clauses of the customer's own policy wording."""
    hits = search.search(search_text=query, top=5,
                         filter=f"policy_id eq '{runtime.context.policy_id}'",
                         query_type="semantic", semantic_configuration_name="default")
    return "\n\n".join(f"[{h['clause_id']}] {h['text']}" for h in hits)

4. The worker agents (steps 9–13). Each worker is a small tool-calling agent with its own prompt and tools.

from langchain.agents import create_agent

def build_workers(mcp_tools: dict):
    intake = create_agent(
        model=writer_llm,
        tools=[mcp_tools["get_policy"], mcp_tools["create_claim_draft"]],
        system_prompt="Collect the first notice of loss. Ask only for missing data. Never promise a payout.",
        context_schema=Ctx)
    coverage = create_agent(
        model=writer_llm,
        tools=[search_policy_wording],
        system_prompt="Answer only from the retrieved clauses and cite their ids. If unsure, offer a person.",
        context_schema=Ctx)
    return intake, coverage

5. The supervisor (step 7). It classifies the intent with the small model and hands over with Command; this is the "routing" that belongs to the application.

from typing import Literal
from pydantic import BaseModel
from langgraph.graph import MessagesState
from langgraph.types import Command

class ClaimsState(MessagesState):
    intent: str | None
    claim_id: str | None
    decision: dict | None

class Route(BaseModel):
    intent: Literal["report_claim", "coverage_question", "other"]

def supervisor(state: ClaimsState) -> Command[Literal["intake", "coverage", "handoff_to_human"]]:
    route = router_llm.with_structured_output(Route).invoke(state["messages"])
    goto = {"report_claim": "intake", "coverage_question": "coverage"}.get(route.intent, "handoff_to_human")
    return Command(goto=goto, update={"intent": route.intent})

6. Fraud signals and human approval (steps 14 and 17–18). A deterministic node posts a review task through the Claims MCP server and pauses the graph; the handler's decision resumes it, and the last node publishes the decision for the push notification.

from langchain.messages import AIMessage
from langgraph.types import interrupt

def make_fraud_check(mcp_tools: dict):
    async def fraud_check(state: ClaimsState) -> Command[Literal["human_review", "__end__"]]:
        claim = latest_claim_draft(state["messages"])   # parsed from the create_claim_draft result
        if claim.amount_eur > 5_000 or fraud_score(claim) > 0.7:          # rules + score: flags only
            await mcp_tools["create_review_task"].ainvoke({"claim_id": claim.id})  # -> Service Bus
            return Command(goto="human_review", update={"claim_id": claim.id})
        return Command(goto="__end__", update={"claim_id": claim.id})
    return fraud_check

def handoff_to_human(state: ClaimsState):
    return {"messages": [AIMessage("I'll put you in touch with a member of our team.")]}

def human_review(state: ClaimsState):
    decision = interrupt({"claim_id": state["claim_id"], "reason": "high value or fraud signal"})
    return {"decision": decision}                       # runs again when the handler resumes

def make_notify(mcp_tools: dict):
    async def notify(state: ClaimsState):
        await mcp_tools["publish_decision"].ainvoke({"claim_id": state["claim_id"], **state["decision"]})
        return {"messages": [AIMessage(f"Claim {state['claim_id']}: {state['decision']['status']}.")]}
    return notify

7. Assembling the graph (the checkpoint of step 14). The graph is built per request because the MCP tools carry the customer's token; the PostgreSQL checkpointer is what lets a claim wait for a person for hours.

from langgraph.graph import StateGraph, START, END

async def build_graph(customer_token: str, checkpointer):
    mcp_tools = {t.name: t for t in await mcp_client(customer_token).get_tools()}
    intake, coverage = build_workers(mcp_tools)

    g = StateGraph(ClaimsState, context_schema=Ctx)
    g.add_node("supervisor", supervisor)
    g.add_node("intake", intake)
    g.add_node("coverage", coverage)
    g.add_node("fraud_check", make_fraud_check(mcp_tools))
    g.add_node("human_review", human_review)
    g.add_node("notify", make_notify(mcp_tools))
    g.add_node("handoff_to_human", handoff_to_human)
    g.add_edge(START, "supervisor")
    g.add_edge("intake", "fraud_check")
    g.add_edge("coverage", END)
    g.add_edge("human_review", "notify")
    g.add_edge("notify", END)
    g.add_edge("handoff_to_human", END)
    return g.compile(checkpointer=checkpointer)

8. The API around it (steps 3, 16 and 17). One call runs a turn and streams tokens back through APIM and Front Door; the claims handler workbench later resumes the same conversation thread with its decision.

from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver

async def chat_turn(conversation_id, text, customer_token, ctx: Ctx):        # step 3
    async with AsyncPostgresSaver.from_conn_string(os.environ["PG_URI"]) as saver:
        graph = await build_graph(customer_token, saver)
        config = {"configurable": {"thread_id": conversation_id}}
        async for chunk, _meta in graph.astream({"messages": [("user", text)]}, config,
                                                context=ctx, stream_mode="messages"):
            yield chunk.content                                               # step 16

async def handler_decision(conversation_id, decision: dict, service_token, ctx: Ctx):  # step 17
    async with AsyncPostgresSaver.from_conn_string(os.environ["PG_URI"]) as saver:
        graph = await build_graph(service_token, saver)
        config = {"configurable": {"thread_id": conversation_id}}
        await graph.ainvoke(Command(resume=decision), config, context=ctx)     # -> notify, step 18

On Kubernetes instead of Container Apps, this same service would simply run as the agent pods of the self-hosted diagram in section 3; nothing in the graph changes.

Notice what is not in this code: no API keys of the model provider, no retry-on-429 logic, no token counting, no list of regions. All of that lives in the AI gateway, so the platform team can change models, limits or guardrails without touching the agents, and the product team can change the graph without touching the platform.

What this design buys the insurer

Start smaller: the same design as a proof of concept

Nobody should build the full picture first. Before asking for the production budget, the insurer needs a first version that shows both use cases working, in front of the people who decide, in a few weeks. The proof of concept (PoC) is not a throwaway demo; it is the same design with everything removed that only real customers, real money and real scale make necessary.

Proof-of-concept architecture

What makes the cuts possible is one rule: synthetic data and internal test users only. With no customer data and no payouts, most of the security, resilience and governance components can wait.

Five things I would not cut, even in a PoC, because adding them later means rewriting the application: the gateway URL with model aliases, the MCP tool contract, checkpoints with human approval, traces with an evaluation set, and real sign-in in an EU region.

What the PoC has to answer before anyone funds production: are the coverage answers grounded in the right clauses on the evaluation set, do test users complete a claim report without help, what does a conversation cost and how long does it take, and do the claims handlers trust what they are asked to approve. Agree the thresholds before the first demo, not after it.

The path to production is then additive: the real core APIs behind the MCP servers and Service Bus, Front Door with the WAF, the second region, the semantic cache and prompt shields, the extraction job, AI Search, private endpoints, Purview and the evaluation gate in CI/CD. None of those steps changes the graph.

The same design on AWS, Google Cloud, or a cloud-independent stack

Stack substitutions

Nothing in this architecture is Azure-specific except the product names. The infographic maps every component to its equivalent in three other stacks:

What stays the same in every column is telling: LangGraph, PostgreSQL, Langfuse and the CI/CD evaluation gate. Those are the pieces that carry the business logic, the state and the quality evidence, so they are worth keeping portable from day one.

How to choose between the columns:

7. LLMOps: the loop that keeps AI applications honest

LLMOps loop

In a classic application, a release changes code. In an AI application, the behaviour changes when someone edits a prompt, re-indexes documents, adds a tool or the provider updates a model snapshot. LLMOps is the loop that makes those changes safe:

Two lessons from practice: redact PII before traces are stored, not after, and treat cost (including cached tokens) as a first-class metric, otherwise your FinOps dashboard will lie to you.

8. DLOps: when you own the models

DLOps

LLMOps is enough while you consume models through APIs. The moment you fine-tune, compress or self-host, you need the operations of a model builder. I call that DLOps (deep-learning operations):

How the ops family fits together: MLOps for classic models, LLMOps for applications on foundation models, DLOps for building and running the models themselves, and an emerging AgentOps for agents in production (tool permissions, agent identity, MCP and A2A governance). Most companies need LLMOps from day one; DLOps only when they fine-tune, compress or self-host, but then it is not optional.

9. The tooling market: who leads in MLOps, LLMOps, DLOps and AgentOps

Ops tooling rankings

Every discipline above has its own crowded market, and the vendor pages all claim to do everything. Below is my ranking for each one as of October 2026. It is based on adoption signals (PyPI and npm downloads in the last month, GitHub stars) and on how often each tool shows up in enterprise stacks. Each entry also says how it is priced: free open source, freemium, pay-as-you-go cloud billing or paid. Download counts are a rough guide: langsmith is installed with every LangChain project, and Datadog's tracer counts all of its APM users.

MLOps (classic models: experiment tracking, registry, pipelines, monitoring)

  1. MLflow: the de facto standard for experiment tracking and the model registry, and the one most other platforms embed: it is managed inside Databricks, Azure Machine Learning and SageMaker. Around 45 million monthly downloads, counting the lightweight client. Free, open source.
  2. Databricks (Mosaic AI): the natural choice when data already lives in the lakehouse; features, training, registry and serving sit in one governed place. Pay-as-you-go cloud billing.
  3. SageMaker AI, Vertex AI, Azure Machine Learning: the default for teams standardised on one hyperscaler. Pay-as-you-go cloud billing.
  4. Weights & Biases: the favourite for experiment tracking in deep-learning teams; now part of CoreWeave. Freemium: free tier, paid plans to scale.
  5. Kubeflow, Airflow, Metaflow: pipelines, depending on whether you live on Kubernetes or already run a scheduler. Free, open source.
  6. Evidently (open source) and Arize: drift and performance monitoring once models are in production. Free to self-host; hosted cloud with a free tier.

Neptune.ai was acquired by OpenAI, so anyone still on it should plan a migration.

LLMOps (applications on foundation models: tracing, evaluations, prompt versions, cost)

  1. LangSmith: the most widely deployed, because it comes with LangChain and LangGraph. It is framework-agnostic and runs in the cloud, hybrid or self-hosted. Freemium: free tier, paid plans to scale.
  2. Langfuse: the most used open-source option (MIT licence, about 35,000 GitHub stars and 23 million monthly downloads). It is the usual pick when traces must stay in your own EU tenant. Part of ClickHouse since January 2026. Free to self-host; hosted cloud with a free tier.
  3. Datadog LLM Observability: the pragmatic choice when the company already runs Datadog: LLM traces sit next to the APIs and infrastructure they depend on. Paid, usage-based, after a free trial.
  4. Arize AX and Phoenix (open source): OpenTelemetry-native tracing and strong evaluations; a good fit where classic ML monitoring also lives. Free to self-host; hosted cloud with a free tier.
  5. Braintrust: evaluation-first, popular with product teams that ship prompts weekly. Freemium: free tier, paid plans to scale.
  6. MLflow 3, Opik (Comet) and W&B Weave: the GenAI extensions of the MLOps tools you may already own. Free to self-host; hosted cloud with a free tier.
  7. promptfoo, DeepEval and Ragas: open-source test frameworks for evaluations and red teaming in CI, used alongside any of the platforms above. Free open-source core; paid enterprise or managed edition.

DLOps (building and running your own models)

  1. vLLM: the default open inference engine (about 93,000 GitHub stars). Free, open source.
  2. SGLang: the main alternative, very strong on throughput and on structured and agentic workloads. Free, open source.
  3. NVIDIA TensorRT-LLM with Dynamo: when you want the last bit of performance from NVIDIA GPUs. Free, open source.
  4. KServe + llm-d: the Kubernetes layer that turns vLLM replicas into a governed, KV-cache-aware service. Free, open source.
  5. Ray (and Anyscale): distributed training, tuning and batch inference. Free open-source core; paid enterprise or managed edition.
  6. DeepSpeed and Megatron-LM: large-scale pre-training and full fine-tuning. Free, open source.
  7. Unsloth, LLaMA-Factory and Axolotl: LoRA and QLoRA fine-tuning on a handful of GPUs. Free open-source core; paid enterprise or managed edition.
  8. Kueue, Slurm and NVIDIA Run:ai: GPU scheduling, quotas and sharing. Free open-source core; paid enterprise or managed edition.

AgentOps (agents in production: multi-step traces, tool calls, agent evaluations)

  1. LangSmith: the most used for LangGraph agents, combining tracing with agent deployment. Freemium: free tier, paid plans to scale.
  2. Microsoft Foundry observability: for agents on Azure, with agent-specific evaluators (tool-call accuracy, task completion), red teaming and Application Insights dashboards. Pay-as-you-go cloud billing.
  3. Amazon Bedrock AgentCore Observability: for agents on AWS, with OpenTelemetry traces and CloudWatch dashboards for sessions, latency, tokens and errors. Pay-as-you-go cloud billing.
  4. Datadog LLM Observability: one view for agents, the APIs they call and the infrastructure underneath. Paid, usage-based, after a free trial.
  5. Arize AX and Phoenix: agent-as-a-judge evaluations and automatic detection of recurring failures. Free to self-host; hosted cloud with a free tier.
  6. Langfuse: open-source agent tracing you can self-host. Free to self-host; hosted cloud with a free tier.
  7. AgentOps (agentops.ai): despite the name, a small developer SDK next to the platforms above. Freemium: free tier, paid plans to scale.

What the rankings say together:

10. Regulation is part of the architecture

In Europe the EU AI Act turns several of these components (audit logs, evaluation evidence, human oversight, model inventories) from good practice into evidence you must be able to produce. The June 2026 Digital Omnibus agreement moved the obligations for stand-alone high-risk systems to 2 December 2027 and for AI embedded in regulated products to 2 August 2028 (Sidley). That is more time, not a reason to wait: the gateway logs, eval reports and model registry described above are exactly the artefacts auditors will ask for.

11. Ten questions to ask before your next AI project

  1. Which decisions are we automating, and where must a human stay in the loop?
  2. Does every agent act with the identity and permissions of the user it serves?
  3. Do all model, tool and agent calls go through one gateway with quotas and audit?
  4. Are our tools exposed once, through MCP, instead of re-implemented per agent?
  5. Will agents from other teams or vendors need to collaborate (A2A)?
  6. What is our golden dataset, and which eval scores block a release?
  7. Are traces standardised (OpenTelemetry GenAI) and PII-redacted before storage?
  8. Do we know the cost per conversation or document, including caching?
  9. Will we fine-tune, compress or self-host? If yes, who owns DLOps?
  10. Which EU AI Act risk tier applies, and can we produce the evidence today?

The model will keep changing every few months. A clear architecture, a gateway, standard protocols and a solid LLMOps/DLOps loop are what let you swap it without starting over.

I'm curious: which layer is the bottleneck in your organisation right now: identity, gateway, evaluation, or GPUs?

Diagrams: my own. Sources linked inline.

#AI #GenAI #AIArchitecture #LLMOps #MLOps #AIAgents #MCP #A2A #AIGateway #EnterpriseAI #CloudArchitecture #EUAIAct