Pipeline Designer

Pipeline Designer

What is Pipeline Designer?

Pipeline Designer is a web-based interface that lets you design, execute, and manage data integration workflows directly in your browser. It supports a wide range of database connections, advanced transformation steps, and robust execution monitoring capabilities.

Installation and Updates

Pipeline Designer is installed by default when you install Pentaho Server. When updates are available, you can use the Plugin Manager to update the plugin. For details, see Update plugins.

Note on Projects

Projects appear on the Pipeline Designer main page but are not currently supported in Pipeline Designer. You can use Projects in the Pentaho Data Integration client instead. For details, see Organizing data integration with projects.

Design and XML Views

Pipeline Designer offers two ways to work with transformations and jobs. You can build workflows in the Design View using a visual interface, or review and edit the raw XML code in the XML View.

ETL Workflow Concepts

Pipeline Designer uses a workflow approach as the foundation for transforming your data. Workflows are built using individual steps that you connect together to create transformations and jobs. Each step is joined by a "hop" which passes data flow from one step to the next.

Pip[eline Designer

Managing Your Work

The Pipeline Designer main page provides tools to manage your transformations and jobs. You can mark items as favorites, download them as files, create duplicates, move them between folders or to the trash, rename them, and view their details.

Working with Transformations

Transformations perform the actual ETL work. You create and configure transformations to define specific data processing tasks, then run them as part of your workflow. After execution, you can analyze the results to explore the data or identify improvements and problems.

Working with Jobs

Jobs orchestrate your ETL activities by coordinating multiple transformations and other tasks. You create and configure jobs to control the execution sequence, then run them to automate your data integration processes. Results analysis helps identify areas for improvement or troubleshooting.

Stopping Execution

Pipeline Designer offers two different methods for stopping running transformations or jobs. The method you choose depends on the specific processing requirements of your ETL task.

Available Steps

Pipeline Designer includes extensive libraries of both transformation steps and job steps. These steps extend and expand the functionality available for building your data integration workflows.

---