$ cat ai-engineering/human-feedback-loops.md

Human-in-the-Loop Feedback Loops for LLM Systems

Turning thumbs, edits, and re-rolls into a data flywheel: capturing user feedback, sampling traces for review, label hygiene, and selective annotation.

Jatin Bansal@blog:~/ai-engineering$ open human-feedback-loops

A team ships a customer-support assistant behind a thumbs-up/down widget. Three months in, the dashboard shows 14,000 thumbs-up and 1,200 thumbs-down. The team has never looked at a single one of those 1,200 rows. The eval suite is at 89% on the golden set, the drift detector is green, the observability stack holds three weeks of traces, and the most expensive curated signal in the entire system — the user told you it was wrong — is sitting in a column of a Postgres table that nobody queries. When the next regression lands, the team will find it via a support escalation. When the fine-tuning vendor asks for “a representative dataset of failures,” the team will email a CSV exported from that table without ever having read it. This is the modal failure mode of human-in-the-loop in production LLM systems: not absence of signal, but absence of a loop that closes on it.

Opening bridge

Yesterday’s piece on drift detection closed the control-loop side of evaluation: input/output/concept drift, the reference window, the regression-testing protocol for model upgrades. It implicitly assumed the judged score — the pinned LLM-as-judge from two days ago — was the ground truth against which production was measured. That assumption is doing real load-bearing work, because the judge itself was calibrated against a human reviewer somewhere upstream and will need to be recalibrated every time the corpus or the user mix shifts. Today’s piece is the layer that makes that calibration honest, and that turns the trace store into something more than a forensic archive: the human-feedback loop that captures explicit and implicit signals from production, samples the right rows for human review, runs them through an annotation surface with label discipline, and feeds the labelled output back into the eval suite, the prompts, the retriever, and — when worth it — the model. This is the article that closes the Evaluation subtree by making it a closed loop.

Definition

A human-in-the-loop feedback loop for an LLM system is the discipline of capturing structured signals from production users and human reviewers, sampling the highest-value rows for annotation under a versioned codebook, and routing the labelled output back into the eval suite, retrieval index, prompts, or model — on a recurring cadence, not as a one-off. Three properties separate it from “we collect feedback.” First, the signal is structured and trace-linked — a thumbs-down is attached to a specific trace ID with the prompt, retrieved context, tools called, and model output preserved next to it; an edit is captured as a diff against the original output; an implicit signal (regenerate, abandon, copy-without-edit) is logged with the same provenance. The raw value is the trace context, not the verdict. Second, the sampling is deliberate — random for representativeness, stratified for category coverage, uncertainty- or disagreement-based for active learning, targeted to the slice where the drift detector just fired. The annotation budget is finite and the queue’s ordering is the load-bearing decision. Third, the loop closes — annotated rows land in the eval suite as new rows, in the retriever as new examples or chunks, in the prompt as new examples or rules, in the fine-tuning corpus as new preference pairs. The metric of the loop is not how many labels you collected; it is how many production failures the loop prevented from recurring.

The naming convention matters. RLHF — reinforcement learning from human feedback — is one specific use of this signal at training time, on the model. The rest of the loop (evals, prompts, retrieval, error analysis) exists for application teams who never fine-tune anything. Most production LLM teams in 2026 sit in the latter group; the feedback loop matters anyway, because the model is only one of the surfaces the signal can flow back into.

Intuition

The mental model: production feedback is the only oracle that can’t be gamed, but it arrives as a noisy stream that needs aggressive filtering before it’s worth a human’s attention. A thumbs-down is not a verdict; it is a hint. A user might down-vote because the answer was wrong, because the formatting was ugly, because the assistant refused a reasonable request, because the user is frustrated about something unrelated, or because their cat walked on the keyboard. The signal-to-noise ratio on raw explicit feedback is usually well under 30% — meaning the majority of thumbs-downs you’d review are actionable, but a substantial minority will be noise on inspection. The unprocessed signal is most valuable as an ordering over the trace store: traces with negative feedback should be sampled at a much higher rate than traces with no feedback, but the conversion from “negative feedback” to “real bug” is the annotation step that humans do, not the signal itself.

The complementary frame: implicit feedback dominates explicit feedback by an order of magnitude in volume, and is consequently both more representative and noisier per unit. A 2025 study of human–LLM dialogues found that under 5% of users leave explicit ratings, but nearly every user leaves implicit signals: regenerate clicks, conversational pivots (“no, I meant…”), copy-without-edit (positive), copy-with-edit (positive-with-correction), abandonment mid-stream (negative), dwell time (ambiguous). The right architecture captures both: explicit feedback as the high-precision, low-recall signal, implicit feedback as the low-precision, high-recall signal. The two streams are merged at the trace store, weighted appropriately, and fed into the sampler that selects the annotation queue.

The third frame, borrowed from Hamel Husain and Shreya Shankar’s evals work: the highest-value annotation in the early weeks of a product is not “label our backlog of thumbs-down” but error analysis on a stratified random sample of production traces. User feedback is informative about what users complained about; it is uninformative about the failures users didn’t notice or didn’t bother to flag. A team that only annotates user-flagged traces will build an eval suite that catches the failures users yelled about, and will silently ship the failures users absorbed without yelling. Both samples — flagged and unflagged — need to be on the annotation queue, with the unflagged stratified random sample as the floor below which the queue should not starve.

The distributed-systems parallel

The closest analogue is a feedback loop in a control system, with the additional constraint that the sensors are unreliable and the controller is slow. Classical control loops measure system output against a setpoint, compute an error, and apply a correction; the LLM feedback loop measures system output against user preference, computes a discrepancy through annotation, and applies a correction through eval rows, prompt edits, retrieval reconfiguration, or fine-tuning. The loop’s bandwidth — how fast a single observed failure can be turned into a structural fix — is the operational SLO. The NVIDIA AI Blueprints data-flywheel and the Agent-in-the-Loop framework from Liu et al. both report production deployments where the cycle from observation to fix shortened from months to weeks; that compression is what the discipline is for.

The deeper parallel is observability with humans in the loop, structured the same way a microservice on-call rotation is. A microservice generates traces that exceed any single engineer’s reading bandwidth; the on-call surface routes a subset to humans — pages on SLO breaches, alerts on high-cardinality error spikes, weekly review of slow-query top-Ks. The LLM feedback loop is structured identically: production generates traces that exceed any annotator’s bandwidth; the routing surface (the annotation queue) selects a subset by the same sampling primitives — high-recall on flagged failures, periodic stratified sweeps, targeted pulls when drift fires, random spot-checks. The annotation queue’s design is essentially a paged-alerting design problem in a different vocabulary.

A real disanalogy worth flagging. A microservice’s control loop measures objective quantities; the LLM feedback loop measures preferences, which are subjective, individually noisy, and often inconsistent across annotators. The classical control-theory advice (“more samples, narrower CI, tighter setpoint”) doesn’t translate cleanly because the setpoint itself is contested — what counts as a good answer depends on who’s asking and who’s judging. The loop has to model annotator disagreement as a first-class signal (via Cohen’s kappa or Krippendorff’s alpha on a held-out double-labelled subset), and either resolve disagreements through a benevolent-dictator pass or accept that a row with low inter-annotator agreement is a row whose rubric is unstable and needs to be split.

Mechanics: the signals worth capturing

The signals factor into three categories. A production loop that doesn’t capture all three is at least one axis blind.

Explicit feedback. What the user says directly. Three sub-modes: binary verdicts (thumbs up/down — high recall on intent, low precision on what was actually wrong), structured categories (a follow-up “what went wrong?” dropdown — hallucination, refused, too long, missed the point, formatting — the highest-signal-per-row data the system gets, but only ~10–30% of thumbs-downs actually fill it in), and free-text edits (the user rewrites the response, and the diff between original and edited is a perfect supervised signal). The edit signal is the most valuable per row: it is a directional correction, not a verdict; it is grounded in the same context the model saw; and it is exactly the shape of data a fine-tuner or preference-pair generator wants. Capture all three with the same trace ID, and prioritize edits at the top of the annotation queue.

Implicit feedback. What the user does. Six common signals: regenerate (the user asked for another try — high precision negative), follow-up clarification (the user immediately pivoted to “no, I meant…” — high precision negative), abandonment (the user closed the session mid-stream — ambiguous but trend-significant), copy-without-edit (the user copied the output to clipboard verbatim — high precision positive), copy-with-edit (the user pasted, edited externally, returned — positive-with-correction; great supervised signal if you can capture the diff), and dwell time (the user spent significantly more or less time on a response than the per-category median — ambiguous but useful in conjunction). The 2025 analysis of human–LLM dialogues found implicit signals are noisier than explicit feedback per row but vastly higher volume; the right architecture exposes them as ordered ranks over the trace store, not as standalone labels.

Reviewer annotations. What internal humans say. The structural signal: a domain expert reviewing a sampled trace, scoring it against the same rubric the LLM judge uses, and either confirming the judge’s verdict (calibration data) or overriding it (new failure-mode candidate). This is the source of truth the loop calibrates against. Volume is constrained by human bandwidth — a reviewer can sustain 50–200 labels per day depending on rubric complexity — so the sampling policy is the load-bearing design decision (next section).

A subtler signal worth mentioning: public-channel feedback — support tickets, internal Slack threads about the assistant misbehaving, customer-success notes. This is structurally similar to explicit feedback but arrives without a trace ID, and the work of re-linking a complaint back to a specific turn is non-trivial. Tools like Langfuse’s user-feedback API and LangSmith’s feedback API include explicit traceId parameters so the application can attach feedback at the moment of capture, but support tickets and Slack threads usually don’t carry that ID, and the re-linking pass is what makes them usable. Worth doing for incidents; not worth doing at steady state.

Mechanics: sampling policies for the annotation queue

A reviewer with 60 minutes a day and a 12,000-trace-per-day stream can label maybe 80 traces. The sampler is the difference between 80 of the most informative traces and 80 of the loudest. Five policies, picked in combination by what the loop is trying to learn.

Random sampling. A uniform random subset of all production traces. The floor below which the queue should never starve, because every other policy is biased toward what the system already knows is broken; only random sampling discovers unknown unknowns. Eugene Yan’s recommendation to aim for “a 50:50 split of passes and fails that spans the distribution of inputs” relies on random sampling as the recall mechanism. Allocate 10–30% of the annotation budget to random, even when the system is on fire and the targeted pulls are screaming.
Stratified sampling. Random within each error-analysis category so the long-tail categories aren’t drowned out by the dominant ones. The dominant category in a customer-support assistant might be “order status” at 40% of traffic; stratifying ensures the long-tail “billing dispute” or “refund processing” categories — which are usually where the failures concentrate — get proportional annotation time.
Negative-feedback sampling. Every trace with an explicit thumbs-down or a high-precision implicit negative (regenerate, abandon, immediate-pivot) goes into the queue with a high prior. The conversion rate from “user said it was bad” to “actually bad” is well under 100% but well above the random base rate; this is the cheapest place to find real failures. Cap the rate so a single noisy user can’t dominate the queue.
Uncertainty-based active learning. The LLM judge’s confidence on a trace — or the score’s distance from a threshold — is a usable uncertainty signal. Traces where the judge’s score sits at the rubric boundary (e.g. 2 on a 1–3 scale) are the rows where the model labels are most ambiguous and the human label most informative. The LLM-based active learning literature reports 50–80% reductions in annotation budget for the same downstream quality versus random sampling on this approach; the catch is that “informative for the judge” is not the same as “informative for the application,” and uncertainty sampling alone over-indexes on edge cases.
Targeted sampling on drift signals. When the drift detector flags a category or cluster as drifted, pull the next N traces from that slice into the queue ahead of the steady-state policies. This is the equivalent of an SRE on-call following the alert to the cluster that’s degrading — same shape, applied to the annotation queue.

The RLTHF paper (Wang et al., 2025) reports that combining LLM-based initial labelling with selective human review on the uncertain rows achieves full-human-annotation alignment quality with only 6–7% of the human annotation effort on the HH-RLHF and TL;DR benchmarks. The result is specific to RLHF preference-pair labelling, but the principle generalises: an LLM-first pass that pre-labels the queue and routes only the uncertain rows to humans is the production pattern that scales when annotation bandwidth is the bottleneck.

A practical mix that holds up: 25% random, 25% stratified by category, 30% negative-feedback (capped), 10% uncertainty-based, 10% targeted on the day’s drift alerts. Re-tune the weights quarterly based on which slice’s annotations turned into the most caught failures.

Mechanics: label hygiene that survives contact with reality

The label is the metric. If the labels are inconsistent across annotators or across days, the downstream eval suite, judge calibration, and fine-tuning corpus all inherit the inconsistency. Four practices that production teams converge on.

Codebook discipline. The rubric is a versioned artifact — not a wiki paragraph that drifts with each onboarded annotator. Each label category has a written definition, a positive example, a negative example, and an edge case that’s explicitly not in the category. Versions are stamped on every label batch so that comparisons across time know which rubric version produced the labels. When the rubric changes, the affected categories’ historical labels are quarantined or re-labelled under the new codebook — not silently re-interpreted.

Benevolent-dictator workflow. Hamel Husain’s recommendation for most teams: appoint a single domain expert as the final judge of label quality and the rubric owner. Distributed annotation efforts without a single point of authority tend to drift into Cohen’s-kappa-below-0.5 territory within a quarter. The benevolent dictator’s job is to spot-check labels, resolve disagreements, edit the codebook, and own the inter-annotator agreement metric. When multiple annotators are necessary — for volume or for legitimately ambiguous categories — periodic alignment sessions with kappa measurement on a doubly-labelled subset keep the process honest.

Inter-annotator agreement as a process metric. Double-label 5–10% of the queue (the same trace labelled independently by two reviewers), compute Cohen’s kappa for two annotators or Krippendorff’s alpha for more, and track it as a dashboard line. Kappa below 0.6 on a category means the rubric is unstable — split the category, write better examples, or accept that the category is genuinely subjective and weight it differently in downstream metrics. The categories with the highest disagreement are usually the ones whose evals are most volatile and whose judge is most unreliable; treating IAA as a debugging tool catches the problem upstream of the eval suite.

Annotation latency budget. A label that lands 12 weeks after the trace is logged is too late to feed back into a meaningful product cycle. The target is single-digit days from “trace generated” to “label applied,” which constrains both the queue size and the policy mix. Queues that grow past their throughput will silently age into uselessness; cap the queue, drop the lowest-priority overflow, and ship the missing labels as a known limitation rather than letting the queue rot.

Mechanics: closing the loop — what the labels actually feed back into

A labelled trace is half the work. The other half is the routing decision: where does the label go, and who acts on it?

Into the eval suite. A confirmed failure that recurs in the wild is a new row on the golden set. The provenance — trace ID, date, annotator — gets stamped on the row, and the row gates merges until the underlying behaviour is fixed. This is the highest-leverage routing: it makes the failure expensive to re-introduce.
Into the judge calibration set. Every labelled trace is also a calibration data point for the LLM judge. Computing judge-human agreement on a rolling window catches judge drift before the eval suite numbers go bad. When agreement drops below a threshold (kappa < 0.6 against the human reference), the judge needs a re-tune or a model swap.
Into the prompt. Edits where the user rewrote the response are a direct prompt-improvement signal. The diff often clusters into a small number of recurring corrections — “use bullet points,” “cite the source,” “don’t apologise” — that are cheap to add to the system prompt as explicit rules.
Into the retriever. A negative trace where the right document was in the index but didn’t make it into the top-K is a retrieval failure, not a generation failure. The labelled trace becomes an example row for retriever evaluation, and over time the corpus of such rows is what drives changes to chunking, reranking, or hybrid search weights.
Into the model. Confirmed-failure-confirmed-fix pairs are the raw material for preference-pair fine-tuning. Most application teams don’t fine-tune anything in 2026, but for teams that do, this is the cleanest source of training data — preference pairs grounded in real production failures, validated by a domain expert against a versioned rubric.

The loop’s cycle time — from observation to a structural fix that prevents the failure recurring — is the metric the team should be tracking, not the raw label volume. Eugene Yan’s framing is right: whoever turns the data flywheel faster wins. A team that labels 5,000 traces a quarter and ships zero fixes is doing labelling-theatre; a team that labels 500 traces a quarter and ships 30 structural fixes is running the loop.

Code: capturing structured user feedback against a Langfuse trace in Python

The harness below captures explicit feedback (thumbs and structured reason) and implicit feedback (regenerate, dwell time, edit diff) against a Langfuse trace ID, stamps each with the trace context, and exposes a simple FastAPI endpoint. Install: pip install fastapi uvicorn langfuse pydantic.

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
# pip install fastapi uvicorn langfuse pydantic
import os
from datetime import datetime, timezone
from typing import Literal, Optional

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from langfuse import get_client

app = FastAPI()
langfuse = get_client()

# --- schemas ---

class ExplicitFeedback(BaseModel):
    trace_id: str
    user_id: str
    rating: Literal["thumbs_up", "thumbs_down"]
    reason_category: Optional[
        Literal[
            "hallucination", "refused", "too_long", "too_short",
            "missed_point", "formatting", "other",
        ]
    ] = None
    free_text: Optional[str] = Field(default=None, max_length=2000)


class EditFeedback(BaseModel):
    trace_id: str
    user_id: str
    original_output: str
    edited_output: str
    edit_kind: Literal["copy_and_edit", "in_place_edit"]


class ImplicitFeedback(BaseModel):
    trace_id: str
    user_id: str
    event: Literal[
        "regenerate", "abandon_mid_stream", "copy_without_edit",
        "follow_up_pivot", "long_dwell", "short_dwell",
    ]
    dwell_ms: Optional[int] = None
    follow_up_text: Optional[str] = None


# --- routes ---

@app.post("/feedback/explicit")
def post_explicit(fb: ExplicitFeedback):
    # Score: -1 for thumbs_down, +1 for thumbs_up. Langfuse stores arbitrary
    # numeric scores against a trace_id; downstream queries filter by name.
    langfuse.create_score(
        trace_id=fb.trace_id,
        name="user_rating",
        value=1.0 if fb.rating == "thumbs_up" else -1.0,
        data_type="NUMERIC",
        comment=fb.free_text or "",
        metadata={
            "user_id": fb.user_id,
            "reason_category": fb.reason_category,
            "signal_type": "explicit",
            "captured_at": datetime.now(timezone.utc).isoformat(),
        },
    )
    return {"status": "ok"}


@app.post("/feedback/edit")
def post_edit(fb: EditFeedback):
    # An edit is supervisor signal. Stamp the diff as a score-with-data so
    # the trace store keeps the original + corrected pair together.
    langfuse.create_score(
        trace_id=fb.trace_id,
        name="user_edit",
        value=1.0,  # presence of an edit; magnitude is in the metadata
        data_type="NUMERIC",
        comment=fb.edited_output,
        metadata={
            "user_id": fb.user_id,
            "original_output": fb.original_output,
            "edit_kind": fb.edit_kind,
            "signal_type": "explicit_correction",
            "captured_at": datetime.now(timezone.utc).isoformat(),
        },
    )
    return {"status": "ok"}


@app.post("/feedback/implicit")
def post_implicit(fb: ImplicitFeedback):
    if fb.event == "long_dwell" and fb.dwell_ms is None:
        raise HTTPException(400, "dwell_ms required for dwell events")

    # Implicit signals carry a prior on satisfaction direction.
    polarity = {
        "regenerate": -1.0,
        "abandon_mid_stream": -1.0,
        "follow_up_pivot": -1.0,
        "short_dwell": -0.3,  # weaker negative
        "copy_without_edit": 1.0,
        "long_dwell": 0.5,    # weaker positive
    }[fb.event]

    langfuse.create_score(
        trace_id=fb.trace_id,
        name=f"implicit_{fb.event}",
        value=polarity,
        data_type="NUMERIC",
        metadata={
            "user_id": fb.user_id,
            "dwell_ms": fb.dwell_ms,
            "follow_up_text": fb.follow_up_text,
            "signal_type": "implicit",
            "captured_at": datetime.now(timezone.utc).isoformat(),
        },
    )
    return {"status": "ok"}


@app.get("/")
def root():
    return {"ok": True}


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8080)

Four things to flag. First, the trace ID is required, not optional — feedback without a trace ID is feedback that can’t be investigated, and the API should reject it. Second, explicit, edit, and implicit signals land under different score names (user_rating, user_edit, implicit_*) so downstream queries can filter cleanly; mixing them under one name will make the dashboards lie. Third, the polarity weights for implicit signals are starting points, not laws; recalibrate them quarterly against labelled traces (high-precision implicit signals should correlate strongly with confirmed-failure annotations; if they don’t, the weight is wrong). Fourth, the metadata is structured, not free-text — reason_category and signal_type are enums in the schema, which is what makes the trace store queryable a year from now. Free-text fields are useful for support escalation but can’t power dashboards.

Code: a sampler that builds an annotation queue from a week of traces in TypeScript

The harness below pulls a week of production traces from a trace store, applies the five-policy mix (random + stratified + negative-feedback + uncertainty + drift-targeted), and emits a deduplicated queue of N trace IDs ordered by priority. Install: npm install @anthropic-ai/sdk zod.

typescript

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
// npm install @anthropic-ai/sdk zod
import { z } from "zod";

// --- types ---

const TraceSchema = z.object({
  traceId: z.string(),
  timestamp: z.string(),
  category: z.string(), // pre-classified at ingest
  judgeScore: z.number().min(0).max(1).nullable(), // null if not judged
  userFeedback: z.enum(["thumbs_up", "thumbs_down", "edit", "none"]),
  implicitSignal: z
    .enum(["regenerate", "abandon", "copy", "follow_up_pivot", "none"])
    .default("none"),
  driftFlagged: z.boolean().default(false),
});

type Trace = z.infer<typeof TraceSchema>;

interface QueueEntry {
  traceId: string;
  priorityScore: number;
  policy: string;
  category: string;
}

// --- policy weights (sum to ~1.0) ---

const POLICY_BUDGET = {
  random: 0.25,
  stratified: 0.25,
  negativeFeedback: 0.3,
  uncertainty: 0.1,
  driftTargeted: 0.1,
} as const;

// --- helpers ---

function sampleRandom(traces: Trace[], n: number): Trace[] {
  // Uniform without replacement
  const pool = [...traces];
  const picked: Trace[] = [];
  while (picked.length < Math.min(n, pool.length)) {
    const i = Math.floor(Math.random() * pool.length);
    picked.push(pool.splice(i, 1)[0]);
  }
  return picked;
}

function sampleStratified(traces: Trace[], n: number): Trace[] {
  const byCat: Record<string, Trace[]> = {};
  for (const t of traces) (byCat[t.category] ||= []).push(t);
  const categories = Object.keys(byCat);
  const perCat = Math.max(1, Math.floor(n / Math.max(1, categories.length)));
  const picked: Trace[] = [];
  for (const cat of categories) {
    picked.push(...sampleRandom(byCat[cat], perCat));
  }
  return picked.slice(0, n);
}

function sampleNegativeFeedback(traces: Trace[], n: number): Trace[] {
  // Anything with an explicit negative or a high-precision implicit negative.
  const negative = traces.filter(
    (t) =>
      t.userFeedback === "thumbs_down" ||
      t.userFeedback === "edit" ||
      t.implicitSignal === "regenerate" ||
      t.implicitSignal === "abandon" ||
      t.implicitSignal === "follow_up_pivot",
  );
  // Per-user cap so a single frustrated user can't dominate the queue.
  return sampleRandom(negative, n);
}

function sampleUncertainty(traces: Trace[], n: number): Trace[] {
  // Score-band rows are the most ambiguous to the judge and most informative
  // to the human. The "rubric boundary" is rubric-specific; here we treat
  // scores in [0.45, 0.55] as the high-uncertainty band on a 0-1 judge.
  const uncertain = traces.filter(
    (t) =>
      t.judgeScore !== null &&
      t.judgeScore >= 0.45 &&
      t.judgeScore <= 0.55,
  );
  return sampleRandom(uncertain, n);
}

function sampleDriftTargeted(traces: Trace[], n: number): Trace[] {
  const flagged = traces.filter((t) => t.driftFlagged);
  return sampleRandom(flagged, n);
}

// --- main sampler ---

export function buildAnnotationQueue(
  traces: Trace[],
  totalBudget: number,
): QueueEntry[] {
  const dedupe = new Set<string>();
  const entries: QueueEntry[] = [];

  const policies: Array<[keyof typeof POLICY_BUDGET, (t: Trace[], n: number) => Trace[]]> = [
    ["driftTargeted", sampleDriftTargeted],
    ["negativeFeedback", sampleNegativeFeedback],
    ["uncertainty", sampleUncertainty],
    ["stratified", sampleStratified],
    ["random", sampleRandom],
  ];

  // Highest-priority policies first so dedupe favors them.
  for (const [name, fn] of policies) {
    const n = Math.floor(totalBudget * POLICY_BUDGET[name]);
    const picked = fn(traces, n * 2); // over-sample to absorb dedupe losses
    let added = 0;
    for (const t of picked) {
      if (added >= n) break;
      if (dedupe.has(t.traceId)) continue;
      dedupe.add(t.traceId);
      entries.push({
        traceId: t.traceId,
        priorityScore:
          name === "driftTargeted" ? 1.0
            : name === "negativeFeedback" ? 0.8
            : name === "uncertainty" ? 0.6
            : name === "stratified" ? 0.4
            : 0.2,
        policy: name,
        category: t.category,
      });
      added += 1;
    }
  }

  // Sort by priority descending.
  entries.sort((a, b) => b.priorityScore - a.priorityScore);
  return entries.slice(0, totalBudget);
}

// --- example wiring (replace with your real trace fetch) ---

async function main() {
  // In production this is a query against Langfuse, LangSmith, Phoenix, or
  // your own trace store. The Trace shape is the projection your sampler needs.
  const traces: Trace[] = [
    // ... pulled from the trace store, last 7 days
  ];

  const queue = buildAnnotationQueue(traces, 400); // 400 labels for the week
  console.log(JSON.stringify(queue.slice(0, 20), null, 2));
}

main().catch(console.error);

Three things to flag. First, the policies are applied in priority order with dedupe, so a trace that’s both drift-flagged and has a thumbs-down counts once at the higher priority. The downstream annotator only sees each trace once, and the queue ordering reflects which policy surfaced it — which matters for downstream metrics (does negative-feedback sampling actually convert to confirmed failures at a higher rate than random? — only measurable if you stamp the policy on each entry). Second, the per-user cap on negative-feedback sampling is omitted for brevity but is non-optional in production; without it, a single frustrated user generating 50 thumbs-downs in one session will eat the queue. Third, the policy budget is a tunable — the 25/25/30/10/10 split is a starting point that should be re-tuned against the metric of “which policy’s labels turned into the most caught failures.” Most teams move to a higher negative-feedback weight once the loop is running, because the conversion rate from “user flagged” to “confirmed failure” is high enough to dominate.

Trade-offs, failure modes, gotchas

The biggest failure mode is not signal absence; it is loop incompleteness. Teams collect thumbs-downs for 18 months without ever annotating them or feeding the annotations back anywhere. The metric to track is not labels-collected; it is fixes-shipped-attributable-to-labels. A loop that doesn’t close on a fix every week is doing labelling-theatre, and the labelled data ages out before it produces value. Treat the loop’s cycle time as the SLO and instrument it explicitly — date of trace, date of label, date of fix, mean time across the cohort.

Implicit signals are correlates, not measurements. A regenerate click correlates with dissatisfaction at ~70% in most studies; the other 30% is users curious about variation, hitting the wrong button, exploring the output space. Treating regenerate as a hard negative will flood the annotation queue with false positives. Treat all implicit signals as priors that bias the sampler, not as labels in their own right; the actual label comes from the annotator looking at the trace.

Annotator drift is a real problem. A single annotator labelling the same kind of trace over months will drift in their interpretation of the codebook — what counted as “hallucination” in week 2 isn’t the same as “hallucination” in week 12. Mitigate by periodically re-labelling a held-out subset of old traces under the current codebook, computing kappa against the original labels, and re-aligning when it drops. The benevolent-dictator workflow makes drift easier to catch but doesn’t eliminate it; the held-out re-labelling is the safety net.

LLM-pre-labelling introduces its own bias. The RLTHF pattern (LLM labels everything, humans correct the uncertain rows) saves enormous annotation budget, but it biases the final dataset toward what the LLM judge considered ambiguous — which is not the same as what the application considers important. Run a quarterly audit where a stratified sample of the high-confidence LLM labels gets human-reviewed; the discovery rate of disagreement on the “easy” rows is what tells you whether the LLM judge is calibrated or just confident-and-wrong.

Privacy and consent are not afterthoughts. Production traces contain PII; user feedback often contains PII (the user’s edits reference their account, their addresses, their data); annotation surfaces expose all of it to reviewers who may be third parties. The memory-privacy article covers the same boundary for stored memories; the annotation surface is the same boundary at a different layer. PII scrubbing at the trace-store ingest, role-based access on the annotation surface, audit logs on reviewer access, and explicit consent collection at the feedback widget are the minimum.

The annotation surface is product, not infrastructure. Hamel Husain’s field guide makes this explicit: a one-click “correct” or “wrong” button next to the trace, with the relevant context already loaded, beats a four-field form by a factor of 5–10 in labels-per-hour. The annotation tool’s UX directly determines the throughput of the loop; treat it as a first-class product surface, not as a spreadsheet with extra steps.

Edit-based feedback has a legal surface most teams miss. When a user edits a model’s output, the edited version is a derivative work that the user authored — and depending on the product’s terms of service, the user may or may not have granted you the right to use that edit as training data, eval data, or even retrieval examples. Get this reviewed once at the start; otherwise the most valuable signal in the loop is the one your legal team makes you delete a year later.

The flywheel is multiplicative, not additive. Each turn of the loop should produce a structural fix — a new eval row, a prompt edit, a retriever change, a model retune — not just one more entry in a backlog. The systems that win are the ones where each labelled failure makes the next failure of the same shape less likely; the systems that lose are the ones where labelled failures accumulate without compounding into capability. Audit quarterly: of the failures labelled six months ago, what percentage of the failure category has reappeared in the trace store? A high recurrence rate means the labels aren’t feeding back into structural changes; a low rate means the flywheel is turning.