Under Settings → AI/LLM → LLM Extraction Templates you solve a core problem of productive LLM extraction: regression-safe prompt maintenance across very many data points.
The problem
Complex LLM data extractions can involve hundreds of data points per case, each queried via a prompt, returned as JSON, and mapped to placeholders. Users keep improving these prompts over time. The risk: an improvement in one place breaks something else — a classic regression that, without tests, often surfaces only much later.
The solution: tested extraction sets
An extraction template is a JSON-based set in which every data point is stored with its placeholder and its prompt. You also select test cases — real documents for which the expected values for each data point are known.
If you later change a prompt, you can check it with one click against dozens of test cases and see immediately: are the previous results still achieved — and, say, has an edge case now also been improved without breaking anything else? This lets you reach very high quality even for complex extraction tasks.
Structure of the template
| Field | Meaning |
|---|---|
| Template name / Description | Name and purpose of the template. |
| AI connector | the language model the extraction runs through. |
| Pseudonymize | replaces personal data before sending to the model. |
| Document tag filter | uses tags to restrict which documents the template applies to. |
| System prompt / User prompt prefix | the overarching instructions (FEEL); typically “Extract only explicitly stated facts” and “Respond only with valid JSON”. |
Data points (JSON attributes)
Each data point is a JSON attribute with:
| Component | Meaning |
|---|---|
| Key | the JSON key — also the placeholder the value is mapped to (e.g. gerichts_urteil_aktenzeichen). |
| Prompt | the field-specific instruction, precisely stating what to return. |
| Comparison | how the value is checked against the expected value in a test — exact or semantic (see below). |
| Regex validator | an optional pattern for format-checking the extracted value. |
Testing for regressions
On the right side of the editor, you maintain individual test cases under Test case configuration (case + document + known expected value); under Test cases you see the saved collection. Run tests the current case, Test all checks the template against all test cases at once. The statistics (runtime, tokens, tokens/second) also help you keep an eye on cost and speed.
Semantic result comparison (semantic-compare)
For free-text data points, an LLM rarely produces the exact same wording twice — but the content should stay the same. The AI connector with the reserved name semantic-compare compares the output by meaning rather than character-for-character. Selected as the comparison for an attribute, a free-text result is then considered correct even if it’s worded differently but matches the expected value in content.
Using it directly in a workflow
A finished, tested template is also a workflow step: the service task LlmExtractionTemplateRun runs it via its _templateName and returns the result as _extractionJson — an incoming document is thereby read out fully automatically and used in subsequent steps (filling placeholders, gateways, further actions).