Proving each ASIST release is better

ASIST is an assistant for aircraft maintenance engineers at IAG airlines, starting with Aer Lingus. When a fault grounds an aircraft, an engineer describes it in plain text. ASIST searches past AOG cases and the fixes engineers recorded, falls back to the Airbus manuals when history is thin, and drafts resolution steps with references.

It is built as a LangGraph agent on Amazon Aurora PostgreSQL with pgvector, and this page shows how it would be evaluated with Langfuse. The team keeps improving it: models are retrained on new data, retrieval is tuned, and the agent gains new steps. Each change raises the same question. Is the new version actually better, and is it still safe? This page sets out a way to prove it.

AOG

Aircraft on Ground: an issue that leaves the aircraft inoperative, so it cannot fly until it is fixed. Every hour costs the airline money.

MRO

Maintenance, repair and overhaul.

AMM

Aircraft Maintenance Manual: Airbus's approved maintenance procedures.

TSM

Troubleshooting Manual: Airbus's fault-isolation procedures.

ECAM

The cockpit system that shows fault messages, such as "NWS FAULT".

NWS · BSCU

Nose wheel steering, and the brake and steering control unit. Used in the examples.

LLM

Large language model, the part that writes the answer.

RAG

Retrieval-augmented generation: find relevant documents first, then have the LLM answer from them.

LangGraph

A framework for building an LLM agent as a graph of steps, called nodes.

pgvector

A PostgreSQL extension that stores text as vectors so similar cases can be found.

Langfuse

An open-source platform for tracing and evaluating LLM apps. Here it records every ASIST run and holds the test sets, experiments and scores.

Trace

The recorded path of one request through the system, step by step, with timings and costs.

Fine-tuning (SFT, LoRA, DPO)

Further training of a model on your own examples or on engineers' preferences.

v1, v2

Successive versions of ASIST. v2 is the candidate that must prove it beats v1.

Everything runs in AWS. The agent is Python with LangGraph, the chat interface is Next.js, and both run as containers on ECS. An ETL built on AWS Step Functions and Lambda loads the AOG history and the Airbus manuals into Aurora PostgreSQL with pgvector, calling an LLM in Amazon Bedrock to extract the problem and the solution from each AOG record. Langfuse is self-hosted on the same ECS cluster, and the test sets are versioned in S3.

core the stack as described for ASIST, including Langfuse for its evaluation · added needed for the evaluation approach on this page · to confirm depends on choices not covered here.

The ETL that fills the knowledge store

Step names are illustrative. In production the ETL keeps loading new AOGs into the live index. The notes under the steps apply when it builds a frozen index for evaluation.

On ECS the unit of deployment is a task or service; "pod" is the Kubernetes term. A self-hosted Langfuse runs as two services, web and worker, and needs its own PostgreSQL, ClickHouse, Redis and S3 storage, kept separate from ASIST's Aurora database.

Every node in the graph gets its own metric (amber tags). An end-to-end score tells you something changed. Per-node scores tell you where.

ASIST as a LangGraph graph, simplified. ASIST abstains and escalates to an engineer in three cases: the question is out of scope; neither past AOGs nor the manuals hold anything relevant; or the guardrail check fails. "Nothing relevant" is decided by a relevance threshold, because vector search always returns its nearest results, even irrelevant ones. The abstain path skips the guardrail because it gives no technical content: a fixed message, at most listing the closest AOG references, with no values or steps. Whether ASIST abstains at the right times is measured separately, by correct abstention and over-refusal. The trajectory tag covers the whole run: node path, loops, latency and cost. In Langfuse each run is recorded as a trace with one span per node, which is where these per-node metrics are read (Part 10). Every metric name is explained in the glossary.

1. 1Offline · data

   Build validation and sealed sets

   The newest Aer Lingus AOGs, held back from training and labelled by mechanical engineers.

2. 2Offline · metrics

   Score every node

   Classifier, both retrievers, fallback decision, generation, guardrail, trajectory. Run as Langfuse experiments.

3. 3Offline · stats

   Compare v_N vs v_N+1

   Both versions on the same new sealed set, same judge, 3–5 repeated runs, paired significance tests, slice by category.

4. 4Offline · attribution

   Ablate each change

   On the validation set, test the new LLM, embeddings and reranker one at a time to see which one earned the gain.

5. 5CI · gate

   Pass the release gates

   No metric drop beyond threshold, zero critical safety errors, latency budget held.

6. 6Online · production

   Shadow, canary, roll out

   Confirm with real engineers, then repeat per airline with its own sets.

For every version, v1 included, take a snapshot of all the AOG history available, then split it by date at the training cutoff. Older AOGs build the system. The newest AOGs are labelled by engineers and split again: a validation set the team practises on, and a sealed set used only to decide whether the version ships. Both sets exist before launch.

How ground truth grows from v1 to v2

Ground truth is never all used to build a version. Each release builds on everything up to its cutoff and holds back the newest slice to test on. While v1 is in service, new ground truth keeps arriving from four sources.

Annotators are qualified mechanical engineers who write down the correct answer for each case. "Annotate" means to add notes; in machine learning the notes are the labels that say what the system should output. They are also called labellers or raters. First, a case writer, often one of the same engineers, turns each AOG record into the question an engineer would have typed.

Anatomy of one case

The route is the path through the graph; the reference steps are the procedure given to the engineer. A case can take the right route and still give the wrong procedure, so they are scored separately. Most cases are sampled from real AOGs so the set stays representative. A small share of abstain cases is added on purpose: questions ASIST has no history or manual coverage for.

Who fills in each field, and how

Annotators own every expected output, for every case in both the validation and the sealed set. They rarely start from a blank page: the AOG record already holds much of the answer, so a large part of the work is checking and correcting a pre-filled draft.

Effort levels, time per case and pooling are general practice in RAG evaluation, not a record of how ASIST was labelled.

Each term in plain words, with an ASIST example. All example numbers are illustrative.

All numbers in this section are illustrative, to show what the report looks like.

v2 shipped newer data and several improvements at once. Adding them one at a time, on the validation set, attributes the gain and exposes a change that does nothing or hurts.

AOG Recall@5 on the validation set, illustrative. Ablation needs many runs, so it never touches the sealed set. The first step runs v1's pipeline on v2's newer data, to separate the gain from more history from the gain from better techniques. The fine-tuned LLM does not move retrieval, which is expected: its effect shows up in the generation metrics.

Part 10 shows how to run all of this on Langfuse; LangSmith, Ragas or DeepEval are alternatives. Version everything that changes a result: prompts, model IDs, the embedding model per pgvector index, and the sets themselves. A new embedding model means a new index, never mixed vectors in one table.

Langfuse is an open-source platform for tracing and evaluating LLM applications. It plugs into LangGraph through a callback handler, so every ASIST run becomes a trace with one span per node. On top of those traces it holds the datasets, runs the experiments and stores the scores this page describes.

Annotators and evaluators are different jobs

Annotators write the answer key before any testing. Evaluators mark ASIST's answers against that key every time a test runs. Think of an exam: teachers write the answer key, a machine marks the multiple-choice questions, a teaching assistant marks the essays, and the teacher spot-checks the assistant.

An LLM is never one of the annotators. It may pre-fill a draft (Part 4), but engineers decide the answer key. Otherwise ASIST would be graded against another model's opinion.

Who marks how much

Where each part of this page lives in Langfuse

# Langfuse Python SDK v3. Check names against the current docs.
from langfuse import get_client, Evaluation
from langfuse.langchain import CallbackHandler
from asist import build_graph, untraceable_values

langfuse = get_client()   # self-hosted URL and keys from env
sealed = langfuse.get_dataset("asist-sealed-2026-10")

def make_task(version):
    graph = build_graph(version)   # frozen v1 or v2
    def task(*, item, **kwargs):
        state = graph.invoke(
            item.input,
            config={"callbacks": [CallbackHandler()]},
        )
        return {"answer": state["answer"],
                "sources": state["sources"],
                "top5": state["retrieved_aogs"][:5],
                "route": state["route"]}
    return task

def aog_recall_at_5(*, output, expected_output, **kwargs):
    expected = set(expected_output["expected_aogs"])
    if not expected:   # manual or abstain case
        return Evaluation(name="aog_recall@5", value=None)
    found = expected & set(output["top5"])
    return Evaluation(name="aog_recall@5",
                      value=len(found) / len(expected))

def critical_errors(*, output, **kwargs):
    bad = untraceable_values(output["answer"], output["sources"])
    return Evaluation(name="critical_errors",
                      value=len(bad), comment=", ".join(bad))

for version in ["v1", "v2"]:
    for run in range(5):
        sealed.run_experiment(
            name=f"{version}-sealed-run{run}",
            task=make_task(version),
            evaluators=[aog_recall_at_5, critical_errors],
        )

Each technique targets one layer, so you know in advance which metric should move and what could get worse.