Ctrlk
Start your 30-day Pentaho Enterprise Evaluation ..
For the complete documentation index, see llms.txt. This page is also available as Markdown.

LangExtract

Extract structured fields from unstructured text with LangExtract and load them with PDI.

LangExtract lets PDI turn free-form text into structured rows.

Use it when regex rules are too brittle and full model training is too heavy.

Each extraction includes source offsets, so you can trace values back to the original text.

Pipeline architecture

Before you start

Complete LangExtract setup.

This page assumes:

  • the LangExtract API is running on http://localhost:8765

  • Ollama is available locally

  • the API endpoint is POST /extract

Choose an integration pattern

Use one pattern per transformation.

Recommended: REST service

Best for reusable and production-ready pipelines.

Flow:

PDI input → REST Client → JSON Input → transform → output

Optional: Shell step

Best for quick local experiments.

Use it only when you do not need a shared service.

Local Ollama backend

Use this when data must stay on-premises.

This is already covered by the setup pattern on the linked setup page.

This page uses the REST service pattern throughout.

API contract

Request

Response

The endpoint is POST /extract.

Parse response fields as class, text, start, and end.

Support Tickets

Use LangExtract to classify and route free-form helpdesk tickets.

This pattern reduces manual triage and catches urgent cases earlier.

support tickets

Business case

GlobalServ Technologies supports 12,000 end users.

The service desk receives 1,400 free-form tickets per day.

Analysts spend about five minutes reading and routing each ticket.

Misrouting happens on roughly 23% of first passes.

Verify the API

Test the extraction endpoint:

Sample tickets

8KB
helpdesk.sql
Open

Target output

Write one row per ticket to helpdesk.ticket_triage.

Expected columns:

  • ticket_id

  • issue_type

  • affected_system

  • urgency

  • submitter

  • error_code

Optionally route critical rows to helpdesk.escalations.

Transformation design

ticket_triage.ktr
27KB
ticket_triage.ktr
Open

Build ticket_triage.ktr.

  1. Table Input Read unprocessed tickets.

  2. Modified JavaScript Value Build request_json.

  3. REST Client Call POST http://localhost:8765/extract.

  4. JSON Input Parse one row per extraction.

  5. Select values Select the required fields.

  6. Dummy I/O buffer.

  7. Row Denormaliser Pivot extracted fields into one row per ticket.

  8. Filter Rows Route Critical tickets to escalations.

  9. Table Output Write triage rows and escalation rows.

Step 1: Table Input

Use:

Step 2: Build request_json

Use Modified JavaScript Value.

Step 3: REST Client

Set:

  • URL: http://localhost:8765/extract

  • Method: POST

  • Body field: request_json

  • Result field: response_json

  • Content-Type: JSON

  • Connection timeout: 30000

  • Socket timeout: 60000

Step 4: Parse the response

Use JSON Input with source field response_json.

Parse these paths:

  • $.extractions[*].class → class

  • $.extractions[*].text → text

  • $.extractions[*].start → start

  • $.extractions[*].end → end

The response field names are generic by design.

Use the class field to map values into ticket-specific columns.

Step 5: Select required fields

Use Select values.

Under Select & Alter tab:

Fieldname:

  • ticket_id

  • class

  • text

Step 6: Buffer

Use Dummy.

Nothing to configure. If ticket_id is already ordered ascending from Table Input - ORDER BY created_at ASC), and the REST Client processes rows sequentially (single copy, no parallelism), then the extractions coming out of JSON Input will already be grouped by ticket_id.

In that case you can replace Sort Rows with a Dummy step and the Row Denormaliser will work correctly because consecutive rows for the same ticket_id are already contiguous.

Step 7: Pivot extracted fields

Use Row Denormaliser.

Set:

  • Key field: class

  • Value field: text

  • Group field: ticket_id

Map these values:

  • issue_type → issue_type

  • system → affected_system

  • urgency → urgency

  • user → submitter

  • error_code → error_code

Step 6: Route critical tickets

Use Filter Rows.

Condition:

Send:

  • true → Insert Escalations

  • false → Insert Ticket Triage

Step 7: Write results

Write standard rows to helpdesk.ticket_triage.

Write critical rows to helpdesk.escalations.

CRITICAL

Quick validation

Check results after the run:

Critical escalations:

Optional alternative: call a local Python wrapper with a Shell step

Use this only for local prototyping.

Pass request_json into a wrapper script on stdin.

Capture stdout into response_json.

Keep the downstream JSON Input and pivot steps unchanged.

Clinical Notes

Use LangExtract to turn narrative clinical notes into structured medical facts.

This helps with coding, reconciliation, and downstream review workflows.

clinical notes

Business case

Northbridge NHS Foundation Trust runs eight hospitals and 34 outpatient clinics.

Clinical staff produce about 3,200 encounter notes per day.

Key facts such as medications, diagnoses, symptoms, and allergies are buried in narrative text.

Sample clinical note

Target output

Write one row per extraction to staging.patient_extractions.

Expected columns:

  • patient_id

  • class

  • text

  • start

  • end

  • extracted_at

Common classes:

  • medication

  • symptom

  • diagnosis

  • allergy

Transformation design

Build clinical_notes.ktr.

  1. Get File Names Read all note files from /data/clinical_notes/.

  2. Text File Input Read each file into note_text.

  3. Modified JavaScript Value Build request_json.

  4. REST Client Call LangExtract.

  5. JSON Input Parse extraction rows.

  6. Modified JavaScript Value Add patient_id and validation flags.

  7. Filter Rows Drop unknown classes.

  8. Table Output Write to staging.patient_extractions.

Step 1: Get file names

Step 2: Text file input

Step 3: Build request_json

Step 4: REST Client settings

Set:

  • URL: http://localhost:8765/extract

  • Method: POST

  • Body field: request_json

  • Result field: response_json

  • Content-Type: application/json

  • Connection timeout: 30000

  • Socket timeout: 120000

Step 5: JSON Input paths

Parse from response_json:

  • $.extractions[*].class → class

  • $.extractions[*].text → text

  • $.extractions[*].start → start

  • $.extractions[*].end → end

Step 6:

Step 7: Validate classes

After parsing, keep only expected classes:

  • medication

  • symptom

  • diagnosis

  • allergy

Use a Filter Rows step to discard anything else or route it to review.

Example load table

Use a staging table like this:

Quick validation

Clinical notes often contain repeated facts.

If you see duplicates, deduplicate on patient_id + class + text + start + end.

Contract Documents

Use LangExtract to pull critical legal and commercial terms from long contracts.

This is useful for due diligence, review queues, and clause indexing.

contract documents

Business case

Meridian Capital Partners reviews about 220 contracts per quarter.

Documents range from 8 to 140 pages.

Key terms such as liability caps and governing law appear in inconsistent locations.

Sample contract text

Target outputs

Use two outputs:

Branch A: clause details

Write one row per extraction to staging.clause_details.

Columns:

  • contract_id

  • class

  • text

  • start

  • end

  • extracted_at

Branch B: contract master

Write one wide row per contract to staging.contract_master.

Columns:

  • contract_id

  • party_a

  • party_b

  • effective_date

  • termination

  • payment_terms

  • liability_cap

  • governing_law

  • validation_status

  • extracted_at

Transformation design

Build langextract_contracts.ktr.

  1. Get File Names Read all *.txt contracts from /data/contracts/.

  2. Text File Input Load each file into contract_text.

  3. Modified JavaScript Value Build request_json.

  4. REST Client Call LangExtract.

  5. JSON Input Parse clause rows.

  6. Add Constants Stamp extracted_at.

  7. Table Output Write normalized clause rows.

  8. Row Denormaliser Pivot target classes into a wide contract row.

  9. Modified JavaScript Value Validate required fields.

  10. Filter Rows Route valid rows to master and invalid rows to review.

Build request_json

REST Client settings

Set:

  • URL: http://localhost:8765/extract

  • Method: POST

  • Body field: request_json

  • Result field: response_json

  • Content-Type: application/json

  • Connection timeout: 30000

  • Socket timeout: 180000

JSON Input paths

Parse from response_json:

  • $.extractions[*].class → class

  • $.extractions[*].text → text

  • $.extractions[*].start → start

  • $.extractions[*].end → end

Pivot classes into contract columns

Use Row Denormaliser with:

  • Key field: class

  • Value field: text

  • Group field: contract_id

Map:

  • party_a → party_a

  • party_b → party_b

  • effective_date → effective_date

  • termination → termination

  • payment_terms → payment_terms

  • liability_cap → liability_cap

  • governing_law → governing_law

Validate required fields

Set validation_status = 'Y' only when these fields exist:

  • party_a

  • party_b

  • effective_date

  • governing_law

Route everything else to staging.contract_review_queue.

Example staging tables

Quick validation

Normalized clause rows:

Wide contract rows:

Long contracts need chunk overlap.

If clauses are split at boundaries, increase overlap before raising model size.

Troubleshooting

Common issues:

  • Connection refused on 8765 Start the LangExtract API and verify curl http://localhost:8765/docs.

  • Empty extractions Tighten the prompt and improve few-shot examples.

  • Wrong JSON paths Parse class, text, start, and end.

  • Weak long-document recall Increase extraction_passes or add overlap.

  • Too many duplicates Deduplicate before final load.

PreviousLLM IntegrationNextAgent-as-a-Service

Last updated

Was this helpful?