ScalaPipeline Framework

ScalaPipeline is a data pipeline framework I want to build in Scala. The idea is to read data from a source, apply some transformations and checks, then write the result somewhere else, with types so that the compiler catches the mistakes.

The work is on the dev branch. I added sources and outputs for CSV, JSON and XML, and transformations to filter data, change fields and group records. The tests show how to use them, for example by grouping people by name and city, counting their emails and calculating their average age. It is still in development. Monitoring, logs and parallel processing are among the things I want to add next.

I also added a test that reads a CSV, filters out people under 18, cleans and renames fields, then writes JSON. It checks the saved file against the exact records I expect. Another test checks that a transformation error stops the pipeline before it writes anything. On 8 October 2026, all 24 tests passed with sbt test. You can read the integration test and the test output.