Most small and medium data pipelines I have seen end up in one of two places. Some get built in a visual tool such as KNIME or Alteryx. The rest get written as Python: a notebook, a script on a cron job, or a task in Airflow, Dagster, or Prefect.
Visual tools are quick to start with. You drop a file reader on a canvas, connect a filter and a join, and look at the rows after each step. The cost arrives later. The workflow is saved in the tool's own format and runs on the tool's own engine, so the logic you put into it (the join keys, the filter that drops test accounts, the fix for a date column that arrives in two formats) can only be executed by that product. Leaving means rebuilding it by hand.
Code has the opposite profile. You own every line, it goes through review, and it runs anywhere Python runs. For routine cleaning and joining it is slower to write, and checking intermediate results means re-running cells and printing df.head(). Colleagues who read SQL comfortably but don't write pandas have a hard time reviewing it.
I wanted the canvas for building and checking a pipeline, and a Python file as the thing I keep. Ciaren is my attempt at that.
What Ciaren does
Ciaren is an open-source visual ETL tool that runs on your own machine. You build a flow on a canvas from 80 built-in nodes: file, SQL, and cloud storage inputs and outputs, cleaning and reshaping steps, joins, window functions, data quality assertions, charts, and scikit-learn based ML nodes. Clicking a node shows the rows it produces. When the flow does what you want, you can run it, put it on a cron schedule, or export it.
The export shaped most of the other design decisions, so it is worth showing. Here is what Ciaren generates for a flow that reads orders and customers, keeps completed orders, joins the two, and totals revenue by region. This is the Polars version, unedited:
import polars as pl
df_orders = pl.read_csv('orders.csv')
df_customers = pl.read_csv('customers.csv')
df_orders = df_orders.filter(pl.col('status') == 'completed')
df_1 = (
df_orders.join(df_customers, left_on='customer_id', right_on='customer_id', how='left', suffix='_y')
.group_by('region')
.agg(pl.col('amount').sum())
.rename({'amount': 'revenue'})
.sort('revenue', descending=True, nulls_last=True)
)
df_1.write_csv('revenue_by_region.csv')The same flow also exports as pandas and as lazy Polars. Nothing in the script imports Ciaren. Input frames are named after their files, so orders.csv becomes df_orders, and a straight run of steps becomes one method chain. If you uninstalled Ciaren tomorrow, this file would still run with pip install polars.
Questions to ask any visual pipeline tool
Whether or not you ever try Ciaren, these are the questions I would ask before putting months of logic into a visual tool:
- Can you export a pipeline as code in a language your team already uses?
- Does that code run without the vendor's runtime, SDK, or license check?
- Could someone maintain the exported code by hand if they had to?
- Is the saved workflow a documented format you can diff in git?
- Can a workflow run headlessly, from a scheduler or CI job, with no GUI open?
My answers for Ciaren, in order: yes, pandas or Polars; yes, the script only needs the dataframe library plus whatever a specific node uses (SQLAlchemy for a database read, scikit-learn for a model); readability is the reason for the naming and chaining rules above; flows export as a JSON .flow document with a published draft spec; and a flow can be started through the REST API or replaced entirely by its exported script.
I would treat a "no" on the second question as the most serious. An export that calls back into the vendor's library is a launcher, and the logic still lives in their product.
Why it is open source and runs locally
A tool that makes it easy to leave has to be useful enough that people stay. I think that is healthy pressure for a project like this, and the license backs it up. The core is AGPL-3.0 and includes the whole product: both engines, all the nodes, code export, scheduling, and ML with MLflow tracking. The plugin API is a separate package under Apache-2.0, so plugin authors can pick their own license for what they build.
It runs locally for a related reason. Most of the data I care about already sits in files and databases I can reach from my laptop, and I did not want a tool that needed it uploaded first. pip install ciaren followed by ciaren serve starts the editor and API on 127.0.0.1:8055, with a single SQLite file for app state. There is no account and no telemetry. The local-first post goes through what that means setting by setting.
Where it does not fit
Ciaren is for batch work on data that fits on one machine. The FAQ puts the practical ceiling at a few gigabytes, depending on your RAM. It does not do streaming, distributed execution, or 100 GB datasets, and its scheduler runs single flows on cron expressions rather than orchestrating hundreds of interdependent DAGs. For those jobs, Spark or Airflow is the better home, and an exported Ciaren script can run inside an Airflow task like any other Python.
It is also alpha software. APIs and formats can still change before 1.0, and there has been no third-party security audit. I would use it today for prototypes and controlled internal workflows. The export helps here too: if the project changed direction, the script would already be sitting in your repository.
The code is on GitHub and the quick start takes about five minutes. If a node you need is missing, or the exported code reads badly for your case, an issue with the flow attached is the most useful thing you can send me.