CCAF Reference

Module 5

Structured Data Extraction

Enforce structure with a schema, stop fabrication at its source, model the real world honestly, and review by risk.

Fix root cause vs. patch symptom

Use a tool/function schema, not prompt-and-parse

Confirmed in official docs
Plain language

If output needs to strictly match a given structure, the reliable way to get it is to define a 'tool' whose input fields are exactly your target structure, and read back the structured result — the model's own tool-calling mechanism enforces the types and required fields for you. Just asking nicely for 'valid JSON' in the prompt is only ever advisory, and it will occasionally produce broken output; trying to regex-parse free text is even more fragile.

Technical

Define a tool/function whose input schema is the target data structure; read the resulting tool_use input rather than parsing free text. Types and required fields are enforced by the tool-use mechanism itself, by construction. Prompt-only instructions to 'output valid JSON' are advisory and can still produce non-conformant output; regex-parsing free text is brittle.

Why it matters

This turns a probabilistic formatting request into a structurally-enforced one — the same guarantee-vs-influence logic as hooks, just applied to output shape instead of behavior.

Tool use for structured output — docs.claude.com

Stop the model from fabricating

Per study guide
Plain language

Models invent plausible-sounding values when a field is marked as required but the information just isn't in the source document. The fixes, roughly from most-direct to least: make the field genuinely optional so it's allowed to be skipped; explicitly say 'return null if this isn't stated'; and for a specific recurring quirk (like informal recipe measurements), show a couple of examples of exactly how you want it handled. Retrying the request over and over only helps with formatting mistakes — it can never invent data that was never there to begin with.

Technical

In order of directness: make genuinely-optional fields optional in the schema; explicitly instruct 'return null if not stated'; for recurring domain-specific quirks, provide few-shot examples of the exact desired handling. Retries resolve formatting failures only — they cannot conjure information absent from the source.

Why it matters

A required field forces an answer; making it optional removes the pressure that causes fabrication in the first place, which is more reliable than trying to catch fabricated values after the fact.

Model the real world (amendments, totals)

Per study guide
Plain language

Sometimes a value genuinely changes over time — a contract amendment overriding the original term — and forcing the extraction into a single answer throws away real information. Instead, design the schema to hold multiple values with the date each one took effect. Similarly, when numbers are supposed to add up (invoice line items vs. a stated grand total), capture both the computed sum and the stated total and flag it for a human when they disagree — never silently 'fix' financial figures yourself.

Technical

When a value legitimately changes (e.g., contract amendments), the schema should capture multiple values with source location and effective dates rather than forcing a single answer. When values must reconcile (line items vs. stated total), capture both the computed and stated values and flag mismatches for human review rather than silently auto-correcting.

Why it matters

Flattening a genuinely historical or reconciliation-requiring value into one number destroys information a downstream human or system may need — the schema should mirror the domain's real shape, not a simplified fiction of it.

Human review should follow risk, and be measured

Per study guide
Plain language

If reviewers can only look at a small slice of output, spend that slice on the cases most likely to be wrong — low-confidence extractions and ambiguous or contradictory source material — not a random sample. And before trusting a 'high confidence' bucket to run unreviewed, check accuracy broken down by document type and field, not just as one big average number, because an overall-good number can hide specific weak spots. To keep catching rare errors in the stuff you've already decided to trust, run an ongoing stratified random sample as a standing audit.

Technical

With limited reviewer capacity, route low-confidence and ambiguous/contradictory cases for review, not a random sample. Before automating high-confidence cases, verify accuracy per document type and per field — an aggregate accuracy number can mask weak segments. To catch and measure rare high-confidence errors over time, use stratified random sampling as an ongoing audit, distinct from the risk-based routing used for day-to-day review.

Why it matters

Risk-based routing and stratified sampling solve two different problems — routing maximizes value from scarce day-to-day review capacity, while sampling is what lets you detect and measure drift in the cases you've already decided not to routinely review.

Retries fix formatting, never missing data

Per study guide
Plain language

If a validation error comes back because a value is the wrong shape (a string instead of a number, a nested object instead of a flat list), feeding that specific error back to the model and asking it to retry usually fixes it in a try or two. But if the actual information was simply never present in the source document, no amount of retrying will make it appear — that's a data problem, not a formatting problem, and needs a different fix (e.g., an optional field or a null).

Technical

Retry-with-error-feedback effectively resolves reformatting failures (wrong type, wrong nesting, wrong date format) because the underlying data exists and simply needs reshaping. It cannot resolve failures where the required information does not exist anywhere in the input — no retry count fixes a missing-source-data problem.

Why it matters

Distinguishing 'the data exists but is misshapen' from 'the data was never provided' determines whether retrying is the right lever at all, versus needing a schema or instruction change instead.

Worked examples for this module

See all 58 →

Situation

When source documents lack certain specs, the model invents plausible values to satisfy required schema fields — for example, fabricating a specific weight when the source only lists dimensions.

Recommended approach

Change the fields that may legitimately be absent from required to optional, letting the model omit them.

Why it works

The model fabricates because the schema forces a value — a required field must be filled, so it makes one up. Making those fields optional removes that pressure: the model can legitimately omit what isn't in the source. This fixes the incentive at its root rather than catching the fabrication afterward.

Weaker approaches people try — and why they fall short

  • Add a self-reported confidence field and filter out low-confidence values afterward.Self-reported confidence is unreliable and doesn't stop the model from inventing the value in the first place.
  • Instruct the model to use placeholder text for missing values.Placeholder text still writes something into a required field and can itself pollute downstream data.
  • Add semantic validation that checks each value against the source afterward.Catches errors after the fact but doesn't remove the incentive to fabricate in the first place.
Takeaway — Models fabricate to satisfy required fields. Make genuinely-optional data optional so omission is allowed.

Plain-language notes on the technical terms above

Schema
A precise description of what shape a piece of data must have — which fields exist, what type each one is, and which are required.
Schema

Situation

Extracting resume data (name, contact info, skills, experience, education) that must strictly conform to a JSON schema, where downstream validation currently fails on missing or mistyped fields.

Recommended approach

Define a tool whose input schema is the target JSON structure, and read the resulting tool-use input directly.

Why it works

Defining a tool whose input schema is the target structure makes the model emit data that conforms to that schema by construction — types and required fields are enforced by the tool-use mechanism itself, making this the most reliable path to schema conformance.

Weaker approaches people try — and why they fall short

  • Make two calls: extract as free text first, then reformat that text as JSON.Two calls add cost and a second failure point, and the reformatting step can still drift from the schema.
  • Give detailed JSON instructions and a template in the system prompt, and ask for JSON only.Prompt instructions are advisory — the model can still produce malformed or non-conformant JSON.
  • Regex-parse the free-text response, with retry logic on failure.Regex parsing of free text is brittle, and retry loops paper over a problem that's avoidable in the first place.
Takeaway — For strict schema conformance, use a tool/function schema and read the structured tool-use output — don't hand-parse text.

Plain-language notes on the technical terms above

Tool/function schema (for extraction)
Defining a fake 'tool' whose only job is to hold your desired output shape, so the model's own tool-calling mechanism enforces the structure for you.
Schema
A precise description of what shape a piece of data must have — which fields exist, what type each one is, and which are required.
Tool/function schema (for extraction)Schema
#33

Validate High-Confidence Accuracy Per Segment

Module 5 · Structured Data ExtractionFix root cause vs. patch symptom

Situation

After three months of full human review, extractions with 90%+ confidence show 97% accuracy overall. Before automating those high-confidence cases, what validation matters most?

Recommended approach

Break the accuracy number down by document type and field to confirm it holds consistently across every segment, not just in aggregate.

Why it works

An aggregate 97% can hide pockets of poor performance — some document types or fields might sit at 80% while others hit 99%. Confirming accuracy holds across every segment prevents auto-approving exactly the slices where the model is actually weak.

Weaker approaches people try — and why they fall short

  • Verify that 97% meets all downstream requirements.Assumes the 97% is uniformly valid across segments — which is exactly what hasn't been checked yet.
  • Compare accuracy at several different confidence thresholds to find an optimal cutoff.Threshold tuning optimizes a single number that could still mask segment-level failures.
  • Run a two-week pilot routing a quarter of high-confidence cases downstream and monitor for errors.Useful, but launching it before checking segment consistency risks pushing an already-weak slice straight to production.
Takeaway — Never trust an aggregate accuracy number for automation — break it down by document type and field to expose weak segments first.

Situation

Event-metadata extraction uses an all-nullable schema, but the model still invents plausible values for fields not actually present in the source article — like a specific attendee count with no attendance information given.

Recommended approach

Add an explicit prompt instruction: return null for any field not directly stated in the source.

Why it works

The schema already allows null, so the fix is simply telling the model to use it. This directly authorizes omission and is the simplest effective correction when the schema is already nullable but the model still isn't using that option.

Weaker approaches people try — and why they fall short

  • Make all fields required with strict validation.The opposite of the fix — it forces the model to invent values instead of allowing omission.
  • Add a second model call to verify each value actually exists in the source.Adds cost and latency to patch behavior that a simple instruction can prevent.
  • Upgrade to a more capable model tier.A bigger model may hallucinate less, but it doesn't address the missing instruction, and it's more expensive without fixing the root cause.
Takeaway — Nullable schema plus fabrication → tell the model explicitly to return null when information isn't stated.

Plain-language notes on the technical terms above

Schema
A precise description of what shape a piece of data must have — which fields exist, what type each one is, and which are required.
Schema

Situation

A contract's original terms say '30-day payment,' but an amendment says '45 days.' The model inconsistently returns one value or the other, with no indication of which one currently applies.

Recommended approach

Redesign the schema so amended fields capture multiple values, each with its source location and effective date.

Why it works

The real information here is that the value changed over time. A schema storing multiple values with source location and effective date captures that reality faithfully, letting downstream consumers see both the original and amended terms and know which applies when.

Weaker approaches people try — and why they fall short

  • Add post-extraction pattern matching to detect amendments and flag them for manual review.Flagging for manual review scales poorly and still discards the structured history.
  • Preprocess documents to classify and remove superseded sections before extraction.Deleting superseded sections destroys information that may still be legally required to retain.
  • Instruct the model to always extract the most recent amendment and ignore superseded terms.'Take the latest' loses the original term and can be wrong when amendments are conditional or partial.
Takeaway — When values legitimately change (amendments), model the history — multiple values with effective dates — rather than forcing one answer.

Plain-language notes on the technical terms above

Schema
A precise description of what shape a piece of data must have — which fields exist, what type each one is, and which are required.
Schema
#36

Few-Shot Examples for Domain-Specific Handling

Module 5 · Structured Data ExtractionFix root cause vs. patch symptom

Situation

Recipe extraction hits informal measurements like 'a handful' or 'a splash' — the model either invents a specific amount or leaves the field empty, accounting for about a quarter of all corrections needed.

Recommended approach

Add few-shot examples showing informal measurements extracted verbatim — kept as-is, not converted to a number or dropped.

Why it works

The model doesn't inherently know the desired convention for this recurring case. Few-shot examples that demonstrate the exact behavior wanted — keep 'a handful' as written rather than inventing a number or dropping it — teach the convention cheaply and directly.

Weaker approaches people try — and why they fall short

  • Add a measurement-type field (precise vs. informal) to the schema.Labels the type but doesn't actually tell the model to preserve the phrase verbatim.
  • Add post-processing pattern matching to detect informal phrases and fill in empty values.Pattern matching is brittle across the endless variety of informal phrasing.
  • Fine-tune a model on hundreds of corrected extractions.Heavy, slow, and overkill for a fix that a few well-chosen examples solve directly.
Takeaway — A specific, recurring behavior gap is usually best closed with few-shot examples of the exact desired output.

Plain-language notes on the technical terms above

Few-shot examples
A couple of worked examples shown to the model demonstrating exactly how you want a tricky, recurring case handled.
Few-shot examples

Situation

About 12% of extractions fail schema validation — for example, a quantity field expecting a number instead receives a range like '2 to 3.' Retrying without any change just reproduces the same failure.

Recommended approach

Send a follow-up that includes the specific validation error, asking the model to correct its output based on that error.

Why it works

Retrying without new information yields the same result. Feeding the specific validation error back gives the model the missing context to fix its own output — it now knows exactly what to correct, and this resolves most format failures within a couple of attempts.

Weaker approaches people try — and why they fall short

  • Set the temperature to 0 for more consistency.Makes output more deterministic, but doesn't teach the model that a range isn't a valid number.
  • Reprocess the failures with a larger, more capable model tier.More expensive, and may still stumble on ambiguous source values without any feedback.
  • Pre-process documents to standardize every possible messy format in advance.A huge, brittle undertaking compared to a simple feedback-based retry.
Takeaway — Blind retries just repeat the failure. Retry with the validation error included so the model can actually correct itself.
#38

Few-Shot for Multi-Faceted Consistency Issues

Module 5 · Structured Data ExtractionFix root cause vs. patch symptom

Situation

A skills list field shows three separate problems at once: compound phrases like 'Python and SQL' get split inconsistently, implied skills sneak in that were never actually stated, and the number of extracted items varies wildly between documents.

Recommended approach

Add few-shot examples that demonstrate compound-phrase handling, an explicit-mention-only criterion, and the right level of granularity, all at once.

Why it works

All three problems are really judgment calls about how to extract, not separate bugs. A well-chosen set of few-shot examples can demonstrate how to split compounds, that only explicitly-mentioned skills count, and the right granularity simultaneously — far more precisely than a short instruction could.

Weaker approaches people try — and why they fall short

  • Enrich the schema with confidence and source-quote metadata fields.Metadata fields don't fix the underlying inconsistency in what actually gets extracted.
  • Add hard constraints like a fixed count range and one skill per entry.Hard numeric caps are arbitrary and can truncate legitimately long, valid lists.
  • Do post-extraction normalization against a canonical taxonomy with deduplication.Helps with deduplication but doesn't fix compound-splitting or implied-skill inclusion at the source.
Takeaway — Several related 'how should it decide?' problems are often solved together with one well-chosen set of few-shot examples.

Plain-language notes on the technical terms above

Few-shot examples
A couple of worked examples shown to the model demonstrating exactly how you want a tricky, recurring case handled.
Few-shot examples

Situation

Retry-with-error-feedback is fixing most extraction failures within two or three attempts. Comparing a few failure types — a nested object where a flat array was expected, a comma in a number, a date in the wrong precision, and 'et al.' standing in for a full author list that only exists in an external document — which one will retrying help the least?

Recommended approach

The 'et al.' case, where the full author list simply isn't present anywhere in the input, is the one retries can't fix.

Why it works

The other cases are reformatting problems — the needed data is present, it just needs reshaping, which retries handle well. The author-list case is different: the information was never provided at all, and no amount of retrying can extract information that doesn't exist in the input.

Weaker approaches people try — and why they fall short

  • Assume a nested object being flattened to a list is equally hard to fix.Flattening a nested object into an array is a straightforward, fixable reshape — retries handle this well.
  • Assume a comma-formatted number is equally hard to fix.Stripping a comma to get a plain integer is trivial reformatting — retries handle this well.
  • Assume a full datetime being truncated to a date is equally hard to fix.Truncating a datetime down to a date is straightforward reformatting — retries handle this well.
Takeaway — Retries fix formatting, never missing source data. If it isn't in the input, no retry can conjure it.
#41

Capture Both Computed and Stated Totals, Flag Mismatch

Module 5 · Structured Data ExtractionFix root cause vs. patch symptom

Situation

For invoices, about 18% of extractions show summed line items that don't match the extracted grand total — sometimes from OCR errors, sometimes from extraction mistakes — and downstream systems reject any mismatched total.

Recommended approach

Add a calculated-total field (the model's own sum of the line items) alongside the stated total, and flag the record for human review whenever they differ.

Why it works

Capturing both the computed sum and the stated total turns a hidden inconsistency into a visible, checkable signal. It never silently alters financial data, and it surfaces exactly the ambiguous cases — whether caused by OCR or extraction — for a person to actually resolve.

Weaker approaches people try — and why they fall short

  • Add few-shot examples of correctly-summing invoices.May reduce model errors, but does nothing about mismatches actually caused by OCR misreads.
  • Add post-processing that proportionally adjusts line items to match the stated total.Auto-adjusting financial figures fabricates numbers and can hide real errors — dangerous in an accounting context.
  • Extract independently, then use a separate model call to reconcile which value is correct.A reconciliation model guessing the 'right' value can silently pick wrong — money discrepancies should be adjudicated by a person.
Takeaway — For financial integrity, capture computed vs. stated values and flag mismatches for humans — never silently 'correct' the numbers.

Plain-language notes on the technical terms above

Schema
A precise description of what shape a piece of data must have — which fields exist, what type each one is, and which are required.
Schema
#42

Stratified Sampling to Catch and Measure Hidden Errors

Module 5 · Structured Data ExtractionFix root cause vs. patch symptom

Situation

Extractions at 85%+ confidence are routed away from human review, yet about 12% of those still contain errors — often around comparison tables, appendices, or ambiguous phrasing. A sustainable way is needed both to catch these and to measure improvement over time.

Recommended approach

Use stratified random sampling: review a fixed percentage of high-confidence extractions every week, on an ongoing basis.

Why it works

Both ongoing detection and a measurable error rate over time are needed. Stratified random sampling of high-confidence outputs gives an unbiased, repeatable error-rate estimate, surfaces recurring error patterns, and lets you tell whether changes actually reduce errors — a sustainable audit loop, not a one-off patch.

Weaker approaches people try — and why they fall short

  • Add heuristic rules flagging documents with comparison tables or appendices, regardless of confidence.Heuristics only catch the error types already anticipated, and don't measure the overall error rate.
  • Add a verification pass that re-extracts each high-confidence document and flags disagreements between the two attempts.Doubles extraction cost and only catches disagreements, missing cases where the model is confidently wrong both times.
  • Lower the confidence threshold and review a larger volume.Floods reviewers and defeats the purpose of automation, without actually measuring anything.
Takeaway — To both catch and measure rare high-confidence errors sustainably, use stratified random sampling as an ongoing audit.

Plain-language notes on the technical terms above

Stratified random sampling
Regularly reviewing a fixed, random slice of even the 'trusted' output, so you can catch and measure rare mistakes over time.
Stratified random sampling

Situation

Extracting menu items to JSON (item, description, price, dietary tags), where source menus vary wildly — prices written as '$12' or '$12.00', dietary info shown as icons in one menu and plain text in another.

Recommended approach

Define a strict output schema, and include explicit format-normalization rules directly in the prompt (e.g. always output price as a number with two decimals, map icons to standard tag names).

Why it works

Combining a strict schema (which guarantees structure and types) with normalization rules in the prompt gets consistent, normalized output in a single pass — the model handles the messiness of the input while the schema enforces the shape of the output.

Weaker approaches people try — and why they fall short

  • Make multiple extraction attempts per document and pick the most common format.Costly, and voting across attempts may still agree on a wrongly-formatted value.
  • Extract as-is and normalize everything in post-processing code.Can work, but pushes normalization logic into brittle code and misses the schema's built-in guarantees.
  • Make a separate extraction call per field.Multiplies cost and latency without actually improving normalization.
Takeaway — Strict schema for structure plus explicit normalization rules in the prompt gets consistent output in one pass.

Plain-language notes on the technical terms above

Schema
A precise description of what shape a piece of data must have — which fields exist, what type each one is, and which are required.
Schema

Situation

Extracting calendar-invite data (title, date, time, location, attendees) into strict JSON, where the downstream system rejects any malformed or non-conformant output.

Recommended approach

Define a tool with the target schema as its input parameters, and have the model call it with the extracted data.

Why it works

Just as with resume extraction earlier, a tool whose input parameters are the schema makes the model produce data that conforms to it by construction — types and required fields are enforced by the tool-use mechanism, making this the most reliable path when the downstream system rejects any non-conformance at all.

Weaker approaches people try — and why they fall short

  • Write detailed JSON instructions and a schema description in the prompt, then parse the text response.Prompt-only JSON guidance is advisory and can still emit malformed output.
  • Pre-fill an opening brace to nudge the model toward JSON, then complete and parse it.A prefill nudges the format but doesn't enforce the schema's specific fields or types.
  • Instruct 'output only valid JSON matching the schema' and retry on parse failure.Instruction-plus-retry treats symptoms and is less reliable than schema-enforced tool use.
Takeaway — Guaranteed schema conformance comes from a tool/function schema, not from prompting and parsing.

Plain-language notes on the technical terms above

Tool/function schema (for extraction)
Defining a fake 'tool' whose only job is to hold your desired output shape, so the model's own tool-calling mechanism enforces the structure for you.
Tool/function schema (for extraction)
#45

Route Low-Confidence / Ambiguous Cases for Review

Module 5 · Structured Data ExtractionFix root cause vs. patch symptom

Situation

Reviewers only have capacity to handle about 5% of total extraction volume. What should decide which extractions they actually see?

Recommended approach

Route the extractions where the model itself indicates low confidence, or where the source document is ambiguous or contradictory.

Why it works

With scarce review capacity, it should be spent where errors are most likely — low-confidence outputs and documents with ambiguous or contradictory content. This targets the slice most likely to actually be wrong, maximizing the value of limited human attention.

Weaker approaches people try — and why they fall short

  • Randomly sample 5% of all extractions for review.Wastes most reviews on extractions that were already correct.
  • Route by entity type (like financial figures or dates), regardless of confidence.Ignores confidence entirely — reviewing many correct high-confidence financial fields while missing risky non-financial ones.
  • Only route extractions after downstream systems report a problem.Reacting only after failures means the error has already caused downstream damage.
Takeaway — Scarce review capacity should follow risk: prioritize low-confidence and ambiguous or contradictory cases.