MLflow

Docker & Docker-Compose

Create MLflow folder and copy over scripts:

# Create MLflow folder and subfolder data
mkdir -p ~/MLflow/{data,mlflow}

# Copy all the MLflow files from Workshop--Data-Catalog/MLflow
cp -r ~/Workshop--Data-Catalog/MLflow/* ~/MLflow/
mv ./Dockerfile ~MLflow/mlflow/

Here's the docker-compose.yml file:

services:
  db:
    restart: always
    image: postgres:15  # Added specific version for consistency
    container_name: mlflow_db
    expose:
      - "${PG_PORT}"
    networks:
      - backend
    environment:
      - POSTGRES_USER=${PG_USER}
      - POSTGRES_PASSWORD=${PG_PASSWORD}
      - POSTGRES_DB=${PG_DATABASE}
    volumes:
      - db_data:/var/lib/postgresql/data/
    healthcheck:
      test: ["CMD", "pg_isready", "-p", "5432", "-U", "${PG_USER}"]
      interval: 5s
      timeout: 5s
      retries: 3

  s3:
    restart: always
    image: minio/minio:RELEASE.2025-04-22T22-12-26Z  # Last version with full admin UI
    container_name: mlflow_minio
    volumes:
      - minio_data:/data
    ports:
      - "${MINIO_PORT}:9000"
      - "${MINIO_CONSOLE_PORT}:9001"
    networks:
      - frontend
      - backend
    environment:
      - MINIO_ROOT_USER=${MINIO_ROOT_USER}
      - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}
      - MINIO_ADDRESS=${MINIO_ADDRESS}
      - MINIO_PORT=${MINIO_PORT}
      - MINIO_STORAGE_USE_HTTPS=${MINIO_STORAGE_USE_HTTPS}
      - MINIO_CONSOLE_ADDRESS=${MINIO_CONSOLE_ADDRESS}
    command: server /data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  tracking_server:
    restart: always
    build: ./mlflow
    image: mlflow_server
    container_name: mlflow_server
    depends_on:
      - db
      - s3
    ports:
      - "${MLFLOW_PORT}:5000"
    networks:
      - frontend
      - backend
    environment:
      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_ACCESS_KEY}
      - MLFLOW_S3_ENDPOINT_URL=http://s3:${MINIO_PORT}
      - MLFLOW_S3_IGNORE_TLS=true
    command: >
      mlflow server
      --backend-store-uri postgresql://${PG_USER}:${PG_PASSWORD}@db:${PG_PORT}/${PG_DATABASE}
      --host 0.0.0.0
      --serve-artifacts
      --artifacts-destination s3://${MLFLOW_BUCKET_NAME}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5000/"]
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  db_data:
  minio_data:

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge

both MinIO and PostgreSQL are using the local file system to store data.

.env

Docker Compose automatically reads a file named .env in the same directory:

Here's the .env file:

# Postgres configuration
PG_USER=mlflow
PG_PASSWORD=mlflow
PG_DATABASE=mlflow
PG_PORT=5432

# MLflow configuration
MLFLOW_PORT=5000
MLFLOW_BUCKET_NAME=mlflow-artefacts

# MinIO access keys - these are needed by MLflow
MINIO_ACCESS_KEY=YuA9IVoTrkIAgiLb8Mfv
MINIO_SECRET_ACCESS_KEY=WlR5eRpiIhZGLVSTLPtyzJakMWRUNf1KvqMhvLAA

# MinIO configuration
MINIO_ROOT_USER=minio_user
MINIO_ROOT_PASSWORD=minio_pwd
MINIO_ADDRESS=:9000
MINIO_STORAGE_USE_HTTPS=false
MINIO_CONSOLE_ADDRESS=:9001
MINIO_PORT=9000
MINIO_CONSOLE_PORT=9001

Get the show on the road loading python libraries

here's the Dockerfile:

FROM python:3.10-slim

RUN apt-get update && apt-get install -y curl
RUN pip install mlflow[extras] scikit-learn pandas numpy matplotlib psycopg2-binary boto3 cryptography pymysql

EXPOSE 5000

minIO

x

Start the MLflow services:

cd
cd ~/MLflow
docker compose --env-file config.env up -d --build

Log into minIO:

https://pdc.pentaho.lab:9001pdc.pentaho.lab

Username: minio_user

Password: minio_pwd

“The community version is now limited to being an object browser only, with deprecated support for accounts & policies management, bucket management, configuration management, lifecycle & tiers management, and site replication.

minio:RELEASE.2025-04-22T22-12-26Z  # Last version with full admin UI

Use a community fork: OpenMaxIO is a community-maintained fork created to preserve the full functionality that was removed from MinIO CE

Create an access key:

Your access key and secret key are not saved until you click the Create button.

Download for import & update the config.env file - save

# MinIO access keys - these are needed by MLflow
MINIO_ACCESS_KEY=YuA9IVoTrkIAgiLb8Mfv
MINIO_SECRET_ACCESS_KEY=WlR5eRpiIhZGLVSTLPtyzJakMWRUNf1KvqMhvLAA

Create a bucket: mlflow-artefacts

MLflow Tracking

The first time you fired up these services, the MLflow Tracking services will not work as you'll need to login into MinIO UI, create your keys and mlflow bucket that the tracking service will use to store artifacts.

Stop & remove your containers

cd
cd ~/MLflow
docker compose down

Restart the containers:

cd
cd ~/MLflow
docker compose up -d --build

Browse to:

https://pdc.pentaho.lab:5000pdc.pentaho.lab

Install the MLflow Python Package.

The MLflow Python package is a simple `pip` install. I recommend installing it in a Python virtual environment.

pip install mlflow

Double check the installation by listing the MLflow library.

pip list | grep mlflow

You should see the library below.

mlflow 2.5.0

Install the MinIO Python Package

You do not need to access MinIO directly to take advantage of MLflow features - the MLflow SDK will interface with the instance of MinIO we set up above. However, you may want to interface with this instance of MinIO directly to manage data before it is given to MLflow. MinIO is a great way to store all sorts of unstructured data. MinIO makes full use of underlying hardware, so you can save all the raw data you need without worrying about scale or performance. MinIO includes bucket replication features to keep data in multiple locations synchronized. Also, someday AI will be regulated; when this day comes, you will need MinIO’s enterprise features (object locking, versioning, encryption, and legal locks) to secure your data at rest and to make sure you do not accidentally delete something that a regulatory agency may request.

If you installed the MLflow Python package in a virtual environment, install MinIO in the same environment.

pip install minio

Double check the installation.

pip list | grep minio

This will confirm that the Minio library was installed and display the version you are using.

minio 7.1.15

x

xx

You can connect a machine learning (ML) server to Data Catalog and import ML model server components, including ML models, experiments, versions, and runs into the ML Models hierarchy within Data Catalog.

Edit the file external-data-source-config.yml :

cd
sudo nano /opt/pentaho/pdc-docker-deployment/conf/external-datasource/external-data-source-config.yml

Add the MLflow server configuration:

servers:
   -    id: {SERVER_ID}
	name: {SERVER_NAME}
	type: {SERVER_TYPE}
	url: {SERVER_URL}
	config:
   	username: {username}
   	password: {password}
   	access_token: {access_token}

Parameter

Description

Value

id

Unique identifier (UUID) for the ML server.

name

Display name for the server. This name appears in the UI.

MLflowServer

type

Type of server (enum value). For ML server, use ‘MlFlow’.

URL

The base URL of the ML server.

http://pdc.pentaho.lab:5000

username

password

access_token

After configuring the ML server in the YAML file, restart the PDC services to apply the changes.

You have successfully configured the ML server in Data Catalog as an external data source. It appears under the Synchronize card in the Management section of Data Catalog.