# Observability

{% hint style="info" %}

### Observability Stack Overview

Our observability infrastructure comprises various tools designed to capture and analyze telemetry data, ensuring the health and performance of our systems. The stack is structured around three primary data types: traces, logs, and metrics.

#### Metrics

Metrics collection utilizes the Otel Collector along with a suite of exporters included within our Platform Data Collection (PDC) framework. This includes tools like the Node Exporter, MongoDB Exporter, and cAdvisor, among others. Together, they provide a comprehensive view of our system's performance and usage statistics.

#### Traces

For tracing, we employ the OpenTelemetry (Otel) Collector, which facilitates capturing and managing trace data across our distributed systems. This component is key for understanding request lifecycles and inter-service dependencies.

#### Logs

Log aggregation and management are handled by Fluent Bit. This lightweight data processor is part of our future work plans to enhance log analysis and storage capabilities. Stay tuned for updates in this area.
{% endhint %}

<figure><img src="/files/QAZQyeWyC7ROIVFUSFOF" alt=""><figcaption><p>Observability</p></figcaption></figure>

{% tabs %}
{% tab title=" cAdvisor" %}
{% hint style="info" %}
cAdvisor (short for **c**ontainer **Advisor**) analyzes and exposes resource usage and performance data from running containers. cAdvisor exposes Prometheus metrics out of the box.
{% endhint %}

{% tabs %}
{% tab title="Enable cAdvisor" %}
{% hint style="info" %}
cAdvisor is not enabled by default.
{% endhint %}

To enable cAdvisor:

1. Navigate to deployment folder:

```bash
cd
cd /opt/pentaho/pdc-docker-deployment/vendor
sudo nano docker-compose.mom.yml
```

2. Scroll down to the mon-cadvisor: section
3. Uncomment all the lines:

<figure><img src="/files/yIEGyqV7RxKcGkkpU0NY" alt="" width="338"><figcaption><p>Uncomment cAdvisor</p></figcaption></figure>

4. Save.

```
CTRL + o
ENTER
CTRL + x
```

{% hint style="info" %}
Note the profile: mon\_enhanced
{% endhint %}

***

#### Enable the OTEL collector to scrap the logs.

1. Navigate to deployment folder:

```bash
cd
cd /opt/pentaho/pdc-docker-deployment/vendor/mon/otel_col
sudo nano otelcol-config.yml
```

2. Uncomment cadvisor.&#x20;

<figure><img src="/files/Dq8b0OYJuUx7NkzSn3OS" alt=""><figcaption><p>Enable Prometheus to scrap</p></figcaption></figure>

3. Save.

```
CTRL + o
ENTER
CTRL + x
```

***

#### COMPOSE\_PROFILES

Ensure the profile has been enabled

1. Navigate to deployment folder:

```bash
cd
cd /opt/pentaho/pdc-docker-deployment/vendor
sudo nano .env.default
```

2. Add mon\_enhanced

<figure><img src="/files/XrD8llSAcQJWrfVl0kTy" alt=""><figcaption><p>add mon_enhanced</p></figcaption></figure>

3. Save.

```
CTRL + o
ENTER
CTRL + x
```

4. You will need to restart PDC to deploy the cAdvisor container.

```bash
cd
cd /opt/pentaho/pdc-docker-deployment
./pdc.sh restart
```

5. Check cAdvisor container is up and running.

```bash
docker ps -n 1
```

<figure><img src="/files/9fOgr3qgtDYtzSykHaJX" alt=""><figcaption><p>Restart PDC</p></figcaption></figure>
{% endtab %}

{% tab title="cAdvisor" %}
{% hint style="info" %}

{% endhint %}

1. Log into Portainer either by clicking on the bookmark or&#x20;

   [**https://localhost:9443/#!/auth**](https://localhost:9443/#!/auth)
2. Enter credentials.

<table data-header-hidden><thead><tr><th width="148"></th><th></th></tr></thead><tbody><tr><td>Username</td><td>admin</td></tr><tr><td>Password</td><td>Portainer123</td></tr></tbody></table>

3. Click on 'Live Connect' option.
4. Make a note of the mon\_cadvisor-1 container IP address & port.

<figure><img src="/files/agMqJAJWHZhHLKbuRPzZ" alt=""><figcaption><p>mon_advisor-1 container IP &#x26; port</p></figcaption></figure>

5. In your brower enter: http\://\[mon\_cadvisor IP:8080]

{% hint style="warning" %}
Be aware that the IP & port is exposed.
{% endhint %}

<figure><img src="/files/iEpqOQayJqgZ59lp6Jdx" alt=""><figcaption><p>cAdvisor UI</p></figcaption></figure>
{% endtab %}
{% endtabs %}
{% endtab %}

{% tab title="OpenObserve" %}
{% hint style="info" %}
OpenObserve is a comprehensive observability platform designed to provide insights into the health and performance of IT systems. At its core, OpenObserve integrates seamlessly with a suite of tools for collecting metrics, traces, and logs— the three primary pillars of observability.&#x20;
{% endhint %}

{% embed url="<https://openobserve.ai/>" %}
Link to OpenObserve
{% endembed %}

{% hint style="success" %}
This section is for reference only.&#x20;

OpenObserve has been enabled by default.
{% endhint %}

1. Edit the .env.default

```bash
cd
cd /opt/pentaho/pdc-docker-deployment/vendor
sudo nano .env.default
```

2. Check the following parameter has been added.

```
COMPOSE_PROFILES=mongodb,collab,pdso,mon_enhanced
```

3. Save.

```
CTRL + o
ENTER
CTRL + x
```

4. You will need to restart PDC to deploy the container.

```bash
cd
cd /opt/pentaho/pdc-docker-deployment
./pdc.sh up
```

***

#### **OpenObserve**

1. Open your web browser and navigate to the following URL: [**http://localhost/internal/openobserve/web**](http://localhost/internal/openobserve/web)
2. Log in using the credentials provided below:

<figure><img src="/files/WhMsiY3wKLUwbyEUmBs2" alt=""><figcaption><p>OpenObserve - Login</p></figcaption></figure>

<table data-header-hidden><thead><tr><th width="201"></th><th></th></tr></thead><tbody><tr><td>Username</td><td>root@example.com</td></tr><tr><td>Password</td><td>Complexpass123</td></tr></tbody></table>

3. Ensure that the organisation on the top right of the home page has been set to: **pdc**

<figure><img src="/files/i1KXplmkuMnm0dGrd2LU" alt=""><figcaption><p>OpenObserve</p></figcaption></figure>

{% tabs %}
{% tab title="Logs" %}
{% hint style="info" %}
**Logs**

OpenObserve provides a centralized log management interface that allows users to easily search, filter, and analyze log data from various sources. This aids in troubleshooting issues and understanding the system's behavior over time.
{% endhint %}

1. Select: Logs from the left-hand menu.

{% hint style="info" %}
You can filter logs by specific fields or keywords and visualize the log events in chronological order. This enables quick identification of patterns and anomalies in the log data.&#x20;

As you type the SELECT statement, you will be prompted ..!
{% endhint %}

<figure><img src="/files/66lhFWGirdywe1pZrER3" alt=""><figcaption><p>SQL Mode</p></figcaption></figure>

2. Ensure your in SQL mode and type in the following Query.

```sql
SELECT * FROM "default" WHERE container_name='/pdc-mongodb-1' AND attr_principalname='root'
```

{% hint style="info" %}
A simplier method is to select the field/value from the list
{% endhint %}

3. Delete the query and select the required fields.

<figure><img src="/files/cu8RfpDeQS9Rj0GqJ9go" alt=""><figcaption></figcaption></figure>

This will also give us an idea of the number of records.

<figure><img src="/files/xuUvcunDGWYhfLnVuHdo" alt=""><figcaption><p>search for field</p></figcaption></figure>
{% endtab %}

{% tab title="Metrics" %}
{% hint style="info" %}
**Metrics**

Each exporter utilizes a specific naming convention for its metrics, facilitating the identification of their sources.&#x20;

For instance, metrics from the node exporter, responsible for server-specific data, start with `node_*`, while metrics collected by cAdvisor, which targets metrics from all Docker containers, begin with container\_\*
{% endhint %}

1. Select: Metrics from the left-hand menu.

In this example: node\_memory\_MemAvailable\_bytes

The screenshot displays the total memory (bytes) available for the last 4 days.

<figure><img src="/files/DcY35s3f1EtLduUObgWA" alt=""><figcaption><p>node_memory_MemAvailable_bytes</p></figcaption></figure>

Choose any metric and confirm that:

* A time series graph can be produced for any metric stored in Prometheus
* Metric values are being recorded in real time
* The PromQL editor can be used to drill down on metrics values with a specific parameter.
  {% endtab %}

{% tab title="Traces" %}
{% hint style="info" %}
Traces are one of its key components of OpenObserve.&#x20;

Traces help you understand the flow of requests across multiple services in a distributed system. This is crucial for identifying bottlenecks and optimizing performance.

By linking errors to specific traces, you can quickly identify the root cause of issues and the context in which they occurred.

Traces reveal how different services interact, helping you understand and manage dependencies in your system.

With trace data, you can identify areas for potential optimization, such as reducing unnecessary API calls or improving database queries.
{% endhint %}

#### Enable Tracing

1. Log into OpenObserve and select Ingestion -> Traces (Open Telementry)

<figure><img src="/files/QOO3sBGLsHjA0DpZuc3V" alt=""><figcaption><p>Ingestion</p></figcaption></figure>

2. Make a note of the OLTP gRPC settings (you will have a different Auth token)

```
endpoint: localhost
  Authorization: "Basic cm9vdEBleGFtcGxlLmNvbTpKeFJHYTh6UEFSZUZUSUlt"
  organization: pdc
  stream-name: default
tls:
  insecure: false
```

3. Edit the following OTEL configuration file.

```bash
cd
sudo nano /opt/pentaho/pdc-docker-deployment/vendor/mon/otelcol.config.yml
```

<figure><img src="/files/p2iO1Hq2wUVHAM0DO2NS" alt=""><figcaption><p>Configure endpoints</p></figcaption></figure>

4. Restart PDC

```bash
cd
cd /opt/pentaho/pdc-docker-deployment
./pdc.sh restart
```

***

#### Searching for a Specific Trace

To locate a specific trace by its `traceID`, you can refine your search by editing the query in the query editor. Use the field name `trace_id` to direct your search to a particular trace.

On the left side of the page, you'll find a list of field names that assist in filtering traces. For instance, to explore traces originating from the front end of the PDC, you can input the following query into the editor:

```plaintext
str_match(service_name, 'pdc-web-client')
```

This query retrieves all traces associated with the `pdc-web-client` service, allowing for a focused analysis of front-end activities.

<figure><img src="/files/gougGcohJFo5vEAmfzCz" alt=""><figcaption><p>pdc-web-client</p></figcaption></figure>
{% endtab %}

{% tab title="Functions" %}
{% hint style="info" %}

{% endhint %}

x

x

{% tabs %}
{% tab title="Query" %}
{% hint style="info" %}
**Query Functions**

OpenObserve supports a variety of functions to manipulate and analyze data effectively. These functions can be used within queries to perform operations like aggregations, calculations, and transformations on collected metrics, traces, and logs.

**Aggregation Functions**: Functions such as `SUM()`, `AVG()`, and `COUNT()` allow for the aggregation of data points over a specified interval.

**Transformation Functions**: Functions like `TOPK()`, `PERCENTILE()`, and `RATE()` help in transforming raw data into useful insights.

**Math Functions**: Basic arithmetic functions (`+`, `-`, `*`, `/`) can be applied to metrics for custom calculations.

**String Functions**: Functions such as `str_match()` and `str_replace()` aid in manipulating text-based log and trace data.

**Aggregation Functions**: Functions such as `SUM()`, `AVG()`, and `COUNT()` allow for the aggregation of data points over a specified interval.

**Transformation Functions**: Functions like `TOPK()`, `PERCENTILE()`, and `RATE()` help in transforming raw data into useful insights.

**Math Functions**: Basic arithmetic functions (`+`, `-`, `*`, `/`) can be applied to metrics for custom calculations.

**String Functions**: Functions such as `str_match()` and `str_replace()` aid in manipulating text-based log and trace data.

Combining these functions within queries can help in deriving meaningful and actionable insights from your telemetry data.
{% endhint %}

x

x

x

x
{% endtab %}

{% tab title="VRL" %}
{% hint style="info" %}

#### VRL Functions

OpenObserve integrates VRL (Vector Remap Language) for complex data transformations and log manipulations. VRL functions provide a powerful way to convert, enrich, and process logs, metrics, and traces with ease.

**Basic Functions**: Use functions like `parse_json()`, `to_string()`, and `to_int()` for basic data type conversions.

**Conditional Functions**: Implement conditional logic with `if`, `else`, and `case` statements.

**String Functions**: Manipulate text data using `upcase()`, `downcase()`, `trim()`, and `substring()` functions.

**Log Enrichment**: Enrich logs with metadata or additional context using functions like `add_field()` and `merge()`.

As a simple example we're going to add a timestamp to the logs.
{% endhint %}

{% embed url="<https://vector.dev/docs/reference/vrl/functions/>" %}
Link to VRL Reference
{% endembed %}

Let's take a look at the different fields in the default log stream.

1. Click on a row and select table to view the fields.

<figure><img src="/files/C00XwzMPJgSxQ3WrYg9g" alt=""><figcaption><p>Fields in Log File - pdc default stream</p></figcaption></figure>

{% hint style="info" %}
We're going to add a formatted\_date field to the logs to help with selecting range - dates in our dashboard.
{% endhint %}

2. The following VRL function will add a field: formatted\_date to the logs.

```
# Create a timestamp (current time)
current_timestamp = now()

# Format the current timestamp into a standard date format
.formatted_date, err = format_timestamp(current_timestamp, "%Y-%m-%d %H:%M:%S")

# Output the formatted date
.
```

3. Copy and paste the following into the VRL Function Editor.

```bash
current_timestamp = now()
.formatted_date, err = format_timestamp(current_timestamp, "%Y-%m-%d %H:%M:%S")
.
```

4. Execute the VRL function & check the log file.

<figure><img src="/files/a0K7pTQlBERaKH4frqJs" alt=""><figcaption><p>formatted_date field</p></figcaption></figure>

5. Save the function so that you can apply the function to all incoming pdc logs.

<figure><img src="/files/BIyHRwpd5Gd9j24CFIJy" alt=""><figcaption><p>save function</p></figcaption></figure>

6. Click on the function option in the sidebar & select: Stream Association.
7. Highlight the default logs stream & associate the formatted\_date function.&#x20;

<figure><img src="/files/IFnpTkQbbSIBxBHvN1ZK" alt=""><figcaption><p>Associate function with stream</p></figcaption></figure>

{% hint style="info" %}
The function will now be applied to all incoming pdc logs.
{% endhint %}
{% endtab %}
{% endtabs %}
{% endtab %}

{% tab title="Alerts" %}
{% hint style="info" %}

#### Alerts in OpenObserve

Alerts in OpenObserve enable proactive monitoring by notifying users of anomalies or performance issues in real-time.
{% endhint %}

x

1. Click on the "Alerts" option from the left-hand menu.
2. Click on 'Create Template'.

<figure><img src="/files/SLVbI8yz5aGILB0SzEhi" alt=""><figcaption><p>Create Template</p></figcaption></figure>

3.

x

x
{% endtab %}

{% tab title="Dashboards" %}
{% hint style="info" %}

#### Dashboards in OpenObserve

Dashboards in OpenObserve offer a consolidated view of metrics, logs, and traces, enabling a holistic perspective on system health and performance.

Let's create a simple dashboard that monitors OS resources.
{% endhint %}

1. Click on the "Dashboards" option from the left-hand menu.
2. Click on 'New Folder' and enter the following details:

<figure><img src="/files/DwvG4CQskFHROTv2KsbY" alt=""><figcaption><p>Create a folder</p></figcaption></figure>

3. Click 'Save'.
4. Click on 'New Dasboard' and enter the following details:

<figure><img src="/files/2CmjVPhZEuU0y2Gjk5Zh" alt=""><figcaption><p>New Dashboard</p></figcaption></figure>

3. Click 'Save'.
4. Start by clicking on 'ADD PANEL'

<figure><img src="/files/RMGr366CSyz01FsdfQ45" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}

{% endhint %}

5.
6. Use the editor to add widgets for visualizing telemetry data.

x

x

x

* **Graphs:** Plot time-series data for real-time monitoring.
* **Tables:** Display logs or metrics in tabular form.
* **Heatmaps:** Identify patterns and anomalies.

x

***

#### Community Dashboards

{% hint style="info" %}
Explore and import pre-built dashboards shared by the OpenObserve community. These can serve as a quick start for common monitoring scenarios.
{% endhint %}

1. To import a dashboard, browse to:

&#x20;        \~/Workshop--Pentaho-Data-Catalog/Dashboards

x

x
{% endtab %}
{% endtabs %}
{% endtab %}
{% endtabs %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://academy.pentaho.com/pentaho-data-catalog-en/setup/components/observability.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.