Merge Streams
Workshop - Merge data streams
The transformation underlines the ‘rules’ for manipulating data streams. Each data stream must have the same data stream fields / order / data type, before they can be merged.
In this workshop, you will need to add a 'description' to the data stream:
Add constant step
Create a new Transformation
Any one of these actions opens a new Transformation tab for you to begin designing your transformation.
By clicking File > New > Transformation
By using the CTRL-N hot key
Text File input
The Text File Input step is used to read data from a variety of different text-file types. The most commonly used formats include Comma Separated Values (CSV files) generated by spreadsheets and fixed width flat files.
The Text File Input step provides you with the ability to specify a list of files to read, or a list of directories with wild cards in the form of regular expressions. In addition, you can accept filenames from a previous step making filename handling more even more generic.
Examine both the orders.txt and description.txt.
Configure the Text file input steps to point to, and retrieve the data from each of the files.
Add Constant
The Add constant values step is a simple and high performance way to add constant values to the stream.
Why - in order to merge the streams each stream has to have the same layout.
The PRODUCTDESCRIPTION field is added to the ‘orders stream’ to ensure the data stream matches the ‘description’ data stream.
Select Values
The Select Values step is useful for selecting, removing, renaming, changing data types and configuring the length and precision of the fields on the stream. These operations are organized into different categories:
Select and Alter - Specify the exact order and name in which the fields could be placed in the output rows
Remove - Specify the fields that could be removed from the output rows
Meta-data - Change the name, type, length and precision (the metadata) of one or more fields
Each of the Select values ensures that each data stream is consistent in layout before merging. Each field must be in the correct order within the data stream so that mappings are successful.
RUN
Run the Transformation.
Click on the Dummy step and ‘Preview’.
As you can see we have 2 merged streams ..
Last updated
Was this helpful?