> For the complete documentation index, see [llms.txt](https://academy.pentaho.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.pentaho.com/pentaho-data-integration/data-integration/data-sources/big-data.md).

# Big Data

{% hint style="info" %}
Big Data refers to extremely large and complex datasets that traditional data processing tools can't handle effectively. It's characterized by the "six Vs":

Think of social media posts, sensor readings, transaction records, and video files all being created simultaneously across millions of devices.

The challenge with Big Data isn't just storing these massive datasets, but extracting meaningful insights from them quickly enough to be useful. Organizations use specialized technologies like distributed computing systems (such as Hadoop and Spark) and cloud platforms to process and analyze this information. Machine learning algorithms help identify patterns that would be impossible for humans to spot manually.

Big Data has transformed how businesses operate and make decisions. Companies use it for everything from predicting customer behavior and optimizing supply chains to detecting fraud and personalizing recommendations. In healthcare, it helps analyze patient records and research data to improve treatments. The key value lies not in having lots of data, but in using advanced analytics to turn that data into actionable intelligence that drives better outcomes.
{% endhint %}

<figure><img src="/files/9OhCXPEL0LMOzyK0iE0j" alt=""><figcaption></figcaption></figure>

***

{% hint style="info" %}
**Workshops**
{% endhint %}

{% tabs %}
{% tab title="Apache Hadoop" %}
x

x

{% content-ref url="/pages/MeTNsXZW0WrnyqieOUKE" %}
[Apache Hadoop](/pentaho-data-integration/data-integration/data-sources/big-data/apache-hadoop.md)
{% endcontent-ref %}
{% endtab %}

{% tab title="Snowflake" %}

{% endtab %}
{% endtabs %}

x


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://academy.pentaho.com/pentaho-data-integration/data-integration/data-sources/big-data.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.