Ciaren runs the same visual flow on either pandas or Polars, and exports that flow as a pandas script, an eager Polars script, or a lazy Polars script. This post explains how that is put together. It is written for people who might contribute, and for anyone building a tool that has to keep two dataframe libraries and a code generator in agreement.
One process
Ciaren installs as one Python package. The backend is FastAPI with async SQLAlchemy 2.0 and Alembic migrations. The frontend is React 19 with React Flow for the canvas, built with Vite and bundled inside the wheel, so ciaren serve serves both the UI and the API on port 8055 and Node.js is not needed at runtime. App state lives in SQLite by default. The cron scheduler runs inside the same process.
The rest of this post is about the engine, which is where the interesting constraints are.
A node is one class with three jobs
Every transformation node is a subclass of BaseTransformation with three abstract methods: execute, which runs the node through whichever engine is active; to_python_code, which returns pandas source; and to_polars_code, which returns Polars source. Keeping all three in one class means a change to a node's behavior and a change to its generated code show up in the same diff.
Here is the row filter, abridged (type hints and most operators removed):
class FilterRowsTransformation(BaseTransformation):
type = "filterRows"
def execute(self, engine, inputs, config):
col, op = config["column"], config["operator"]
return {"out": engine.filter_rows(inputs["in"], col, op, self._values(config))}
def to_python_code(self, input_vars, output_vars, config):
src, dst = input_vars["in"], output_vars["out"]
col, op, val = config["column"], config["operator"], config.get("value")
if op in _SIMPLE_OPS:
return f"{dst} = {src}.loc[lambda _d: _d[{col!r}] {op} {val!r}]"
...
def to_polars_code(self, input_vars, output_vars, config):
src, dst = input_vars["in"], output_vars["out"]
col, op, val = config["column"], config["operator"], config.get("value")
if op in _SIMPLE_OPS:
return f"{dst} = {src}.filter(pl.col({col!r}) {op} {val!r})"
...Notice that execute never touches pandas or Polars. It calls engine.filter_rows. The engine is an object that satisfies the EngineBackend protocol: read, write, filter_rows, join, groupby_agg, sort_rows, and the other operations nodes need. There is one implementation per library, registered by name, and the executor looks one up at the start of a run with get_engine(engine_name). The two backend modules are not allowed to import each other's library.
Where pandas and Polars disagree
Most of the work in keeping two engines consistent is in small defaults. A text filter is a good example. In pandas, Series.str.contains treats its argument as a regular expression unless you pass regex=False, so a filter for a.b would also match axb. Polars has the same default: str.contains reads the pattern as a regex unless you pass literal=True. Ciaren's engines and its generated code both turn regex matching off for the "contains" operator, so a filter for a.b matches that literal text on either engine and in either export.
Other differences cannot be removed, only made visible. After a groupby, pandas moves the group key into the index while Polars keeps it as a column, so the pandas export adds .reset_index() and the Polars export does not. Null handling, dtype inference, and row order can still differ in edge cases, and the engines guide says so.
Two sets of tests hold this together. A parity test runs representative flows through both engines and compares the normalized output. An equivalence test runs each node configuration through an engine, executes that node's generated code on the same input, and compares the frames. A guard test fails if a codegen test case is added without equivalence input data, so new generated code cannot skip the comparison.
From graph to script
The code generators walk the graph in topological order and ask each node for its line. Several passes then make the output read like something a person wrote:
- Variable reuse. A node that is the last consumer of a single input writes back into that input's variable, so a linear flow uses one name throughout. A join or a fan-out gets a fresh
df_N, because more than one consumer needs the earlier frame. - Naming. Input frames are named after the dataset file:
Sales 2024.csvbecomesdf_sales_2024. Stems that are empty, all digits, or over 30 characters fall back to numbered names, and duplicates get_2,_3. - Chain fusion. Consecutive
df = df.method(...)statements merge into one method chain, parenthesized with one call per line when it gets long.
The lazy Polars dialect reads with scan_csv and similar functions and collects once at the write. Some nodes have no lazy equivalent, including pivot, sampling, and the data quality assertions, so the lazy export wraps just those steps in .collect() and .lazy() and keeps the rest of the plan lazy.
What a run records
Before a run starts, SQL and cloud storage inputs are read in the async layer and saved as local Parquet snapshots. The executor then reads those snapshots like any other file, so node execution only ever works on local data. File inputs point at a dataset version. Datasets are immutable: uploading a file with the same name adds a new version.
Each run then records its status, timestamps, logs, the engine it used, and the resolved dataset versions, so you can tell which data and which library produced an output. Per node, it keeps the status, row and column counts, a small sample, and duration_ms.
Compute runs off the event loop. The default is a worker thread, which supports precise cancellation. A process mode gives true multi-core use across concurrent runs, crash isolation, and enforced timeouts, at the cost of coarser cancellation and a slower first run.
Plugins
The plugin API is a separate package under Apache-2.0, so a plugin depends on a published contract and not on Ciaren's internals. It defines ten provider base classes, from NodeProvider and ConnectorProvider to AuthProvider and LicenseProvider. Plugins can run nodes, connectors, and ML model types end to end today. The docs describe the remaining contracts, such as extra engines, exporters, and auth methods, as defined extension points for future plugins.
A plugin declares the permissions it wants in its manifest, such as network, credentials, or filesystem_write. A plugin that declares permissions is not imported until you approve it. Packages can carry a detached Ed25519 signature, and a package whose contents do not match its signature is refused. These are consent and integrity checks. Plugin code is not sandboxed and runs with the same access as the Ciaren process, which the security docs state up front.
Where to start reading
If you want to follow any of this in code, backend/app/engine/backends/base.py has the engine protocol, backend/app/engine/transformations/ has the nodes, and backend/app/engine/codegen.py and polars_codegen.py have the generators. The source is on GitHub, and the engines guide covers the user-facing side.