> For the complete documentation index, see [llms.txt](https://academy.pentaho.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.pentaho.com/pdi-developer-path-sandbox/speciality-courses/ai-pipelines.md).

# AI Pipelines

AI-augmented data pipelines with Pentaho Data Integration. GenAI fundamentals, LLM integration patterns, structured extraction, and agent-as-a-service.

{% hint style="warning" icon="eye" %}
**SANDBOX — draft page.** For review only. Items marked REPLACE-ME need an owner's confirmation.
{% endhint %}

Use Pentaho Data Integration as the orchestration layer for AI work. Start with prompts, tokenisation and embeddings, then put a locally hosted LLM to work on sentiment, data quality, enrichment, entity recognition and summarisation — and finish by deploying an agent that reasons over intermediate output rather than answering one call at a time.

{% columns %}
{% column %}

<p align="center"><i class="fa-file-lines" style="color:green;">:file-lines:</i><br><strong>19 lessons</strong><br><em>across 6 modules</em></p>
{% endcolumn %}

{% column %}

<p align="center"><i class="fa-flask" style="color:orange;">:flask:</i><br><strong>13 workshops</strong><br><em>hands-on labs</em></p>
{% endcolumn %}

{% column %}

<p align="center"><i class="fa-clock" style="color:cyan;">:clock:</i><br><strong>~8h 15m</strong><br><em>8h 10m of workshops</em></p>
{% endcolumn %}
{% endcolumns %}

{% hint style="success" icon="cloud-arrow-up" %}
Every workshop is hands-on and runs on Pentaho's hosting platform. The Lab Guide is embedded in each lab and loads automatically when you open it — nothing to install.
{% endhint %}

***

<h3 align="center">Before you start</h3>

{% columns %}
{% column %}
{% hint style="success" icon="circle-check" %}
**What you'll need**

Comfortable building transformations and jobs in Pentaho Data Integration. The labs provide a local Ollama model, Python and the two lab services the later modules depend on — no AI experience assumed.
{% endhint %}
{% endcolumn %}

{% column %}
{% hint style="info" icon="tag" %}
**Built against**

Pentaho Enterprise Edition **11.0.0.0-237**, the build every course on the Academy runs, with the GenAI EE plugin and a locally hosted LLM via Ollama. Models run on the lab machine, so nothing leaves the environment.
{% endhint %}
{% endcolumn %}
{% endcolumns %}

***

{% hint style="info" icon="trophy" %}

#### What you'll build

A progression from single LLM calls to a deployed agent. Along the way: pipelines that classify sentiment, flag data-quality problems, enrich records, pull named entities and summarise text; LangExtract services that turn support tickets, clinical notes and contract documents into structured rows; and finally a Python agent that PDI orchestrates for cross-text pattern recognition no ETL step can replicate.
{% endhint %}

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FCotDp4c58VEub6ncnXym%2Fthree-approaches.svg?alt=media" alt="Direct LLM call: one row in, Ollama&#x27;s /api/generate once per row or stage, one enriched row out. LangExtract: one text with a prompt and examples in, one call per extraction pass, N extraction rows out. Agent as a service: one row and its history in, one call over every text, one assessment row out; only it needs other records. Pentaho Data Integration owns retrieval, routing and persistence"><figcaption><p>What goes in, what does the work and what comes out, for each approach</p></figcaption></figure>

***

<h3 align="center">Skills you'll gain</h3>

<table data-view="cards"><thead><tr><th></th><th></th></tr></thead><tbody><tr><td><i class="fa-lightbulb" style="color:blue;">:lightbulb:</i> <strong>LLM concepts</strong></td><td>Prompts · tokenisation · embeddings · semantic search · transformer architecture</td></tr><tr><td><i class="fa-sparkles" style="color:blue;">:sparkles:</i> <strong>GenAI fundamentals</strong></td><td>HTML parser · Apache Tika for unstructured documents · prompt engineering</td></tr><tr><td><i class="fa-comments" style="color:blue;">:comments:</i> <strong>LLM integration</strong></td><td>A local LLM via Ollama and the GenAI EE plugin · sentiment analysis · data quality · enrichment · named entity recognition · summarisation · multi-staged pipelines</td></tr><tr><td><i class="fa-file-magnifying-glass" style="color:blue;">:file-magnifying-glass:</i> <strong>LangExtract</strong></td><td>Few-shot extraction prompts · a REST service backed by Ollama · support tickets, clinical notes and contracts into structured rows</td></tr><tr><td><i class="fa-robot" style="color:blue;">:robot:</i> <strong>Agent-as-a-Service</strong></td><td>Deploy a Python agent · orchestrate it from PDI · reasoning over intermediate output · cross-text pattern recognition</td></tr></tbody></table>

***

<h3 align="center">Course outline</h3>

<p align="center"><i class="fa-file-lines" style="color:green;">:file-lines:</i> reading · <i class="fa-flask" style="color:orange;">:flask:</i> hands-on workshop</p>

<table data-search="false"><thead><tr><th>Module</th><th width="100" align="center">Reading</th><th width="110" align="center">Workshops</th><th width="100" align="right">Time</th></tr></thead><tbody><tr><td>Getting Started</td><td align="center">1</td><td align="center">—</td><td align="right">~5m</td></tr><tr><td>PDI AI Pipelines</td><td align="center">1</td><td align="center">1</td><td align="right">~45m</td></tr><tr><td>GenAI</td><td align="center">1</td><td align="center">2</td><td align="right">~30m</td></tr><tr><td>LLM Integration</td><td align="center">1</td><td align="center">6</td><td align="right">~3h 35m</td></tr><tr><td>LangExtract</td><td align="center">1</td><td align="center">3</td><td align="right">~1h 50m</td></tr><tr><td>Agent-as-a-Service</td><td align="center">1</td><td align="center">1</td><td align="right">~1h 30m</td></tr><tr><td><strong>Total</strong></td><td align="center"><strong>6</strong></td><td align="center"><strong>13</strong></td><td align="right"><strong>~8h 15m</strong></td></tr></tbody></table>

{% stepper %}
{% step %}

### <i class="fa-rocket" style="color:blue;">:rocket:</i> Getting Started

`1 reading · ~5m`

How to get the most out of this guide, and how to set your machine up — Pentaho Data Integration, a local Ollama model, Python, and the two lab services you will start later in the course.

<details>

<summary>See the reading</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Before You Start</td><td>~5m</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-lightbulb" style="color:blue;">:lightbulb:</i> PDI AI Pipelines

`1 reading · 1 workshop · ~45m`

Key concepts behind LLM-powered pipelines — prompts, tokenisation, embedding, semantic search and transformer architecture. Foundations: the ideas and a first hands-on workshop.

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FWJmMpXjlLHRP3QPsGLBu%2Fsemantic-search.svg?alt=media" alt="Two phases. Before any query, documents are split into chunks, each embedded by nomic-embed-text into 768 numbers and kept in a vector store. For each query, the query is embedded by the same model and compared with the chunk vectors by cosine similarity; the nearest chunks and the query go into a prompt, within the token limit, and an LLM answers from them"><figcaption><p>Semantic search: the query's embedding finds the relevant chunks for the LLM</p></figcaption></figure>

<details>

<summary>See all 2 lessons</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview of a Prompt</td><td>—</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Key Concepts Workshop</td><td>~45m</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-sparkles" style="color:blue;">:sparkles:</i> GenAI

`1 reading · 2 workshops · ~30m`

GenAI fundamentals for data engineers — parsing HTML with the HTML Parser, and reading unstructured documents with Apache Tika.

<details>

<summary>See all 3 lessons</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview</td><td>—</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>HTML Parser</td><td>~15m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Apache Tika — read unstructured documents</td><td>~15m</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-comments" style="color:blue;">:comments:</i> LLM Integration

`1 reading · 6 workshops · ~3h 35m`

Pentaho Data Integration as the orchestration layer with a locally hosted LLM via Ollama. Sentiment analysis, data quality, enrichment, named entity recognition, summarisation, and multi-staged pipelines.

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FvOSZeVgImGE6kYLBS9Ir%2Fllm-row-skeleton.svg?alt=media" alt="One row&#x27;s path: CSV file input, Modified JavaScript value building json_payload with getVariable and JSON.stringify, REST client posting to Ollama&#x27;s /api/generate on localhost:11434 with stream false and format json, then Parse 1 reading $.response and Parse 2 reading the model&#x27;s own fields, and Text file output. A callout shows the model&#x27;s JSON held as a string inside Ollama&#x27;s reply"><figcaption><p>The skeleton every workshop shares: build, call, parse twice</p></figcaption></figure>

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2Fc9s1WxrBSxZTpA83pq6h%2Fmulti-stage-fields.svg?alt=media" alt="Read Documents feeds five stages. Each reads the document and earlier fields and adds its own: Stage 1 doc_type, confidence and reasoning; Stage 2 urgency, priority and priority_reasoning; Stage 3 extracted_details, by doc_type; Stage 4 actions_json, action_count and requires_escalation; Stage 5 primary_dept, escalate_to, cc_depts and sla_hours. Write Processed Documents writes every field"><figcaption><p>The five stages and what each adds to the row</p></figcaption></figure>

<details>

<summary>See all 7 lessons</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview</td><td>—</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Sentiment Analysis</td><td>~45m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Data Quality</td><td>~35m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Data Enrichment</td><td>~30m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Named Entity Recognition</td><td>~30m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Text Summarisation</td><td>~30m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Multi-Stage Pipelines</td><td>~45m</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-file-magnifying-glass" style="color:blue;">:file-magnifying-glass:</i> LangExtract

`1 reading · 3 workshops · ~1h 50m`

Turn free-form text into structured rows when regex is too brittle. Support tickets, clinical notes and contract documents — extracted via a lightweight REST service backed by Ollama.

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FrKm8aubS2PwLRewvfGJe%2Flangextract-architecture.svg?alt=media" alt="Pentaho Data Integration: Table input, Modified JavaScript value building request_json, REST client, JSON input on $.extractions[*] and Row denormaliser. The LangExtract service, localhost:8765, has POST /extract and lx.extract with chunking, overlap and passes. Ollama, localhost:11434, runs the model. Requests send extraction_class and extraction_text; responses return class, text, start and end"><figcaption><p>Pentaho Data Integration calls the LangExtract service, which calls the local Ollama</p></figcaption></figure>

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2F7VTnI92a9OGkEDLCpom9%2Fsupport-tickets-flow.svg?alt=media" alt="Production design. A Pentaho Data Integration column for ticket_triage.ktr: Table input on helpdesk.tickets, Modified JavaScript value building request_json, REST client to the LangExtract service and Ollama, JSON input, Select values, a Dummy buffer, Row denormaliser and Filter rows on urgency = CRITICAL. TRUE rows load helpdesk.escalations and FALSE rows helpdesk.ticket_triage"><figcaption><p>The production design</p></figcaption></figure>

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2Fkg5C2TiTVfVB9IyrNj3p%2Fclinical-notes-flow.svg?alt=media" alt="A Pentaho Data Integration column for clinical_notes.ktr: Get file names, Text file input reading note_text, Modified JavaScript value building patient_id and request_json, REST client to the LangExtract service and Ollama, JSON input with a row per extraction, and Filter rows on class IN LIST medication, symptom, diagnosis, allergy. TRUE rows load staging.patient_extractions; others are dropped"><figcaption><p>What you build: the steps of clinical_notes.ktr, the service they call, and where each fact goes</p></figcaption></figure>

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FI7obWsnMtvxOMQzKkiJD%2Fcontract-documents-flow.svg?alt=media" alt="langextract_contracts.ktr in Pentaho Data Integration: Get file names, Text file input, Modified JavaScript value building contract_id and request_json, REST client to the LangExtract service, JSON input, Get system info, Row denormaliser, a validation script and Filter rows on validation_status = Y. Branch A loads clause_details; TRUE rows go to contract_master, FALSE to contract_review_queue"><figcaption><p>One stream, two branches: a row per clause, and a wide row per contract that is validated before it loads</p></figcaption></figure>

<details>

<summary>See all 4 lessons</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview</td><td>—</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Support Tickets</td><td>~40m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Clinical Notes</td><td>~35m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Contract Documents</td><td>~35m</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-robot" style="color:blue;">:robot:</i> Agent-as-a-Service

`1 reading · 1 workshop · ~1h 30m`

Move beyond direct LLM calls to genuine AI agents that reason over intermediate output. Deploy a Python agent that Pentaho Data Integration orchestrates for cross-text pattern recognition no ETL step can replicate.

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FWMFa22NoBSPUdlfw7Y73%2Fagent-assess-sequence.svg?alt=media" alt="Four lanes: SQLite, the transformation, the agent on localhost:8000 and Ollama. Table input and Database join read the entries and their history; Group by leaves one row per entry; REST client posts to /assess; the agent makes one Ollama call and answers 200 with seven keys; Filter rows sends other codes to Write HTTP Errors; Switch / case routes the rest to four Table outputs into assessed_log"><figcaption><p>One request per log entry, and one LLM call per request</p></figcaption></figure>

<details>

<summary>See all 2 lessons</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview</td><td>—</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Deploy &#x26; Run the Agent</td><td>~1h 30m</td></tr></tbody></table>

</details>
{% endstep %}
{% endstepper %}

***

{% columns %}
{% column %}
{% hint style="warning" icon="file-certificate" %}
**Course accreditation**

REPLACE-ME — what a learner must finish to complete the course, and what they receive for it.
{% endhint %}
{% endcolumn %}

{% column %}
{% hint style="warning" icon="award" %}
**Level 3 credit**

REPLACE-ME — confirm whether this speciality counts towards the Level 3 — Certified credential.
{% endhint %}
{% endcolumn %}
{% endcolumns %}

***

<h3 align="center">How to get access</h3>

{% stepper %}
{% step %}
**Read the course outlines**

Open the level or speciality you're considering and check the modules cover the work you actually do.
{% endstep %}

{% step %}
**Talk to your Pentaho Account Executive**

Courses and subscriptions are purchased through your Pentaho Account Executive — a single seat or the whole team. They'll size it with you and get the order placed. No Account Executive yet? Email <academy@pentaho.com> and we'll connect you.
{% endstep %}

{% step %}
**Receive your voucher code**

You'll be sent a voucher code giving access to the course or courses you bought.
{% endstep %}

{% step %}
**Redeem your code and start your lab**

Redeem the code on the Pentaho training portal and your hosted lab deploys with the Lab Guide built in. Every screen is shown step by step in the voucher walkthrough.

{% embed url="<https://pentaho.selfpaced.training/access/login>" %}

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FPEKFCHIQlCddWOeTkw1V%2Fportal.png?alt=media&#x26;token=1552611a-6317-4746-ab5a-acd61397acca" alt="The Pentaho training portal sign-in page"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

***

<p align="center"><a href="https://academy.pentaho.com/pdi-developer-path-sandbox/speciality-courses.md" class="button secondary" data-icon="arrow-left">Back to Speciality Courses</a></p>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://academy.pentaho.com/pdi-developer-path-sandbox/speciality-courses/ai-pipelines.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
