Read data from multiple files

Reading data from multiple dynamically specified files is fairly straight forward to do and is covered in the getting started samples included with the product (have a look at 11 - Dynamic Data Sources.dm).
 
Note you can dynamically control the file the readers and writers operate on using the Filename connector, but you will need to toggle the "Simple View" on the right click context menu, to see them.
 
Once you have a list of filenames you can feed them into the readers Filename connector and it will load a new file for each one its presented with.
 
Their are a number of options for producing the list of filenames, you can either use the ListFiles component that gets all the files in a folder matching a given filter (i.e *.csv). If that's not the way you want to go then you could create an input parameter which is a list of filenames, you can then split this (split when it sees a separator char like ',') then feed these values into your reader Filename connector. You can then run it from the command line with syntax something like

Transform.exe SourceXmlFiles:"I:\sourceA.xml,I:\sourceB.xml" OutputCsvFile:"c:\Output.csv"
 
 
 

Add Feedback