> For the complete documentation index, see [llms.txt](https://academy.pentaho.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.pentaho.com/pdi-developer-path-sandbox/speciality-courses/streaming-data.md).

# Streaming Data

Real-time streaming data integration with Pentaho Data Integration. Consume and process unbounded event streams over MQTT, AMQP and Apache Kafka.

{% hint style="warning" icon="eye" %}
**SANDBOX — draft page.** For review only. Items marked REPLACE-ME need an owner's confirmation.
{% endhint %}

Process unbounded, real-time event streams with Pentaho Data Integration. Three protocols, four brokers, four use cases — industrial robot telemetry and delivery-truck predictive maintenance over MQTT, room-sensor data over AMQP, and a live user-activity stream over Kafka — all built on PDI's parent/child streaming pattern.

{% columns %}
{% column %}

<p align="center"><i class="fa-file-lines" style="color:green;">:file-lines:</i><br><strong>9 lessons</strong><br><em>across 5 modules</em></p>
{% endcolumn %}

{% column %}

<p align="center"><i class="fa-flask" style="color:orange;">:flask:</i><br><strong>4 workshops</strong><br><em>hands-on labs</em></p>
{% endcolumn %}

{% column %}

<p align="center"><i class="fa-clock" style="color:cyan;">:clock:</i><br><strong>~2h 35m</strong><br><em>2h 30m of workshops</em></p>
{% endcolumn %}
{% endcolumns %}

{% hint style="success" icon="cloud-arrow-up" %}
Every workshop is hands-on and runs on Pentaho's hosting platform. The Lab Guide is embedded in each lab and loads automatically when you open it — nothing to install.
{% endhint %}

***

<h3 align="center">Before you start</h3>

{% columns %}
{% column %}
{% hint style="success" icon="circle-check" %}
**What you'll need**

Comfortable building transformations and jobs in Pentaho Data Integration. The labs provide Podman Desktop, all four brokers and Python for the publisher scripts — one command brings HiveMQ, EMQX, RabbitMQ and Kafka up together.
{% endhint %}
{% endcolumn %}

{% column %}
{% hint style="info" icon="tag" %}
**Built against**

Pentaho Enterprise Edition **11.0.0.0-232**, the build every course on the Academy runs, with the Kafka Consumer EE step. Brokers run in containers alongside the lab, so you see real message flow rather than a simulation.
{% endhint %}
{% endcolumn %}
{% endcolumns %}

***

{% hint style="info" icon="trophy" %}

#### What you'll build

Four working streaming pipelines. Industrial robot-sensor telemetry consumed from a HiveMQ broker; a delivery-truck predictive-maintenance pipeline on EMQX; room-sensor readings pulled from RabbitMQ and appended to output files; and a real-time user-activity stream consumed from Kafka and loaded into a MySQL warehouse.
{% endhint %}

```mermaid
%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, Helvetica, sans-serif","fontSize":"13px","lineColor":"#64748b","clusterBkg":"#fafaf9","clusterBorder":"#d6d3d1","edgeLabelBackground":"#ffffff"},"flowchart":{"curve":"basis","nodeSpacing":28,"rankSpacing":44,"padding":22,"wrappingWidth":600,"htmlLabels":true}}}%%
flowchart TB
  classDef input fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a,text-align:left,rx:6,ry:6
  classDef enrich fill:#ede9fe,stroke:#6d28d9,color:#3b0764,text-align:left,rx:6,ry:6
  classDef engine fill:#ede9fe,stroke:#6d28d9,color:#3b0764,text-align:left,rx:6,ry:6
  classDef output fill:#dcfce7,stroke:#15803d,color:#14532d,text-align:left,rx:6,ry:6
  classDef support fill:#f1f5f9,stroke:#475569,color:#0f172a,text-align:left,rx:6,ry:6
  PUB["<b>Publishers</b>&nbsp;&nbsp;&nbsp;<br/>Python scripts simulating&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br/>sensors and users&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]:::input
  MQ["<b>MQTT</b>&nbsp;&nbsp;&nbsp;<br/>HiveMQ, EMQX&nbsp;&nbsp;&nbsp;<br/>robot telemetry, truck maintenance&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]:::support
  AM["<b>AMQP</b>&nbsp;&nbsp;&nbsp;<br/>RabbitMQ&nbsp;&nbsp;&nbsp;<br/>room sensors&nbsp;&nbsp;&nbsp;"]:::support
  KA["<b>Kafka</b>&nbsp;&nbsp;&nbsp;<br/>KRaft mode&nbsp;&nbsp;&nbsp;<br/>user activity&nbsp;&nbsp;&nbsp;&nbsp;"]:::support
  PAR["<b>Parent transformation</b>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br/>MQTT, AMQP or Kafka Consumer step&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br/>batch size and duration shape&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br/>the stream into windows&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]:::engine
  CH["<b>Child transformation</b>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br/>Get records from stream,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br/>then ordinary steps&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br/>runs once per window&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]:::enrich
  FO["<b>Output files</b>&nbsp;&nbsp;&nbsp;<br/>room-sensor readings appended&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]:::output
  DB[("<b>MySQL warehouse</b>&nbsp;&nbsp;&nbsp;&nbsp;<br/>user-activity stream loaded&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;")]:::output
  PUB --> MQ
  PUB --> AM
  PUB --> KA
  MQ --> PAR
  AM --> PAR
  KA --> PAR
  PAR -->|"each window&nbsp;&nbsp;&nbsp;"| CH
  CH --> FO
  CH --> DB
```

<p align="center"><em>The parent/child pattern behind every broker: the parent consumes and windows the stream, the child processes each window.</em></p>

***

<h3 align="center">Skills you'll gain</h3>

<table data-view="cards"><thead><tr><th></th><th></th></tr></thead><tbody><tr><td><i class="fa-wave-pulse" style="color:blue;">:wave-pulse:</i> <strong>Streaming fundamentals</strong></td><td>Batch versus streaming · the parent/child transformation pattern · Get records from stream · batch size and duration to window a stream</td></tr><tr><td><i class="fa-tower-broadcast" style="color:blue;">:tower-broadcast:</i> <strong>MQTT</strong></td><td>HiveMQ and EMQX brokers · MQTT Consumer step · robot-sensor telemetry · delivery-truck predictive maintenance</td></tr><tr><td><i class="fa-inbox" style="color:blue;">:inbox:</i> <strong>AMQP</strong></td><td>RabbitMQ · exchanges, bindings and queues · AMQP Consumer step · room-sensor readings appended to output files</td></tr><tr><td><i class="fa-bolt" style="color:blue;">:bolt:</i> <strong>Kafka</strong></td><td>Apache Kafka in KRaft mode · Kafka Consumer EE step · a live user-activity stream · loading into a MySQL warehouse</td></tr></tbody></table>

***

<h3 align="center">Course outline</h3>

<p align="center"><i class="fa-file-lines" style="color:green;">:file-lines:</i> reading · <i class="fa-flask" style="color:orange;">:flask:</i> hands-on workshop</p>

<table data-search="false"><thead><tr><th>Module</th><th width="100" align="center">Reading</th><th width="110" align="center">Workshops</th><th width="100" align="right">Time</th></tr></thead><tbody><tr><td>Getting Started</td><td align="center">1</td><td align="center">—</td><td align="right">~5m</td></tr><tr><td>Streaming Data</td><td align="center">1</td><td align="center">—</td><td align="right">—</td></tr><tr><td>MQTT</td><td align="center">1</td><td align="center">2</td><td align="right">~1h 15m</td></tr><tr><td>AMQP</td><td align="center">1</td><td align="center">1</td><td align="right">~30m</td></tr><tr><td>Kafka</td><td align="center">1</td><td align="center">1</td><td align="right">~45m</td></tr><tr><td><strong>Total</strong></td><td align="center"><strong>5</strong></td><td align="center"><strong>4</strong></td><td align="right"><strong>~2h 35m</strong></td></tr></tbody></table>

{% stepper %}
{% step %}

### <i class="fa-rocket" style="color:blue;">:rocket:</i> Getting Started

`1 reading · ~5m`

How to get the most out of this guide, and how to set your machine up — Podman Desktop, one command for all four brokers (HiveMQ, EMQX, RabbitMQ, Kafka), and Python for the publisher scripts.

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FwGbTOJiO4gapfRz3W5JW%2Fsd-lab-setup.svg?alt=media" alt="Four rows, one per workshop: sensor.py to HiveMQ on 1883 to an MQTT consumer writing robot_sensor.txt; the MQTT producer to EMQX on 1884 to an MQTT consumer writing the alert files; sensor_tv_room.py to RabbitMQ on 5672 to an AMQP consumer writing tv_room_sensor.txt; produce_users.py to Kafka on 9092 to a Kafka consumer loading MySQL user_events on 3306."><figcaption><p>The lab map: publishers, broker containers, consumers and outputs</p></figcaption></figure>

<details>

<summary>See the reading</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Before You Start</td><td>~5m</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-wave-pulse" style="color:blue;">:wave-pulse:</i> Streaming Data

`1 reading`

Process unbounded, real-time event streams with Pentaho Data Integration's streaming steps. Batch versus streaming, the parent/child transformation pattern, and the **Get records from stream** step that underpins every streaming source.

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FttF89QLoAaLnqHidW1QZ%2Fparent-child-streaming.svg?alt=media" alt="A broker, HiveMQ, EMQX, RabbitMQ or Kafka, feeds the consumer step in the parent transformation, which closes a batch at Duration (ms) or Number of records, whichever comes first. Each batch runs the child: Get records from stream, JSON input, Select values and Formula, then Table output."><figcaption><p>One pattern for every protocol: the parent batches, the child runs once per batch</p></figcaption></figure>

<details>

<summary>See the reading</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview</td><td>—</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-tower-broadcast" style="color:blue;">:tower-broadcast:</i> MQTT

`1 reading · 2 workshops · ~1h 15m`

Message Queuing Telemetry Transport — the lightweight publish/subscribe protocol for IoT. Two workshops: industrial robot-sensor telemetry via a HiveMQ broker, and a delivery-truck predictive-maintenance use case via an EMQX broker.

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FB04M1ypAWHUUBOduRSiD%2Fmqtt-qos-levels.svg?alt=media" alt="Three sequences between the publisher and the broker. QoS 0: one PUBLISH and no reply; a message may be lost. QoS 1: PUBLISH, a re-send while no PUBACK has come back, which gives the broker a duplicate, then PUBACK. QoS 2: PUBLISH, PUBREC, PUBREL and PUBCOMP; delivered once, at the most overhead."><figcaption><p>The three QoS levels, packet by packet</p></figcaption></figure>

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FhrOZm0XnCVybRRrXuNKA%2Fmqtt-lwt-retained.svg?alt=media" alt="sensor.py, HiveMQ on localhost:1883, a subscriber and a later subscriber. sensor.py connects with a will (offline, QoS 1, retained) and publishes a retained online. The window is closed without DISCONNECT, so HiveMQ publishes the will: the subscriber gets offline, and a later subscriber gets the retained offline. A clean disconnect() would discard the will."><figcaption><p>The will and the retained status topic, from connect to crash</p></figcaption></figure>

<details>

<summary>See all 3 lessons</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview</td><td>—</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>HiveMQ — industrial robot-sensor telemetry</td><td>~40m</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>EMQX — delivery-truck predictive maintenance</td><td>~35m</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-inbox" style="color:blue;">:inbox:</i> AMQP

`1 reading · 1 workshop · ~30m`

Advanced Message Queuing Protocol — reliable, decoupled messaging via exchanges, bindings and queues. Consume room-sensor data from a RabbitMQ broker with the **AMQP Consumer** step and append it to output files.

<details>

<summary>See all 2 lessons</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview</td><td>—</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>RabbitMQ — room-sensor data to output files</td><td>~30m</td></tr></tbody></table>

</details>
{% endstep %}

{% step %}

### <i class="fa-bolt" style="color:blue;">:bolt:</i> Kafka

`1 reading · 1 workshop · ~45m`

Apache Kafka — a distributed event-streaming platform running in KRaft mode. Consume a real-time user-activity stream with the **Kafka Consumer EE** step and load it into a MySQL warehouse.

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2F9kGSUDZ04dUIJFNWsd8T%2Fkafka-users-to-db.svg?alt=media" alt="produce_users.py feeds the Kafka topic pdi-users on 127.0.0.1:9092. In users-to-db-parent, Kafka consumer reads it as group pdi-warehouse-users, a batch every 500 ms or 100 records. Each batch runs users-to-db-child: Get records from stream, JSON input, Select values, Formula and Table output, which inserts into the MySQL table user_events."><figcaption><p>The pipeline: users-to-db-parent hands each batch to users-to-db-child, which loads user_events</p></figcaption></figure>

<details>

<summary>See all 2 lessons</summary>

<table data-search="false"><thead><tr><th width="79"></th><th>Lesson</th><th width="90">Time</th></tr></thead><tbody><tr><td><i class="fa-file-lines" style="color:green;">:file-lines:</i></td><td>Overview</td><td>—</td></tr><tr><td><i class="fa-flask" style="color:orange;">:flask:</i></td><td>Use Cases — user-activity stream into MySQL</td><td>~45m</td></tr></tbody></table>

</details>
{% endstep %}
{% endstepper %}

***

{% columns %}
{% column %}
{% hint style="warning" icon="file-certificate" %}
**Course accreditation**

REPLACE-ME — what a learner must finish to complete the course, and what they receive for it.
{% endhint %}
{% endcolumn %}

{% column %}
{% hint style="warning" icon="award" %}
**Level 3 credit**

REPLACE-ME — confirm whether this speciality counts towards the Level 3 — Certified credential.
{% endhint %}
{% endcolumn %}
{% endcolumns %}

***

<h3 align="center">How to get access</h3>

{% stepper %}
{% step %}
**Read the course outlines**

Open the level or speciality you're considering and check the modules cover the work you actually do.
{% endstep %}

{% step %}
**Talk to your Pentaho Account Executive**

Courses and subscriptions are purchased through your Pentaho Account Executive — a single seat or the whole team. They'll size it with you and get the order placed. No Account Executive yet? Email <academy@pentaho.com> and we'll connect you.
{% endstep %}

{% step %}
**Receive your voucher code**

You'll be sent a voucher code giving access to the course or courses you bought.
{% endstep %}

{% step %}
**Redeem your code and start your lab**

Redeem the code on the Pentaho training portal and your hosted lab deploys with the Lab Guide built in. Every screen is shown step by step in the voucher walkthrough.

{% embed url="<https://pentaho.selfpaced.training/access/login>" %}

<figure><img src="https://1407627559-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3zO083cdmEBCaZG8U9P2%2Fuploads%2FPEKFCHIQlCddWOeTkw1V%2Fportal.png?alt=media&#x26;token=1552611a-6317-4746-ab5a-acd61397acca" alt="The Pentaho training portal sign-in page"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

***

<p align="center"><a href="https://academy.pentaho.com/pdi-developer-path-sandbox/speciality-courses.md" class="button secondary" data-icon="arrow-left">Back to Speciality Courses</a></p>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://academy.pentaho.com/pdi-developer-path-sandbox/speciality-courses/streaming-data.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
