Ciaren Architecture Overview
How Ciaren works under the hood: a FastAPI backend, a React Flow canvas, a dual Polars/pandas execution engine, and a plugin system with ten extension points — all in one local install.
Ciaren promises a lot in one sentence — visual editing, live previews, two execution engines, code export, scheduling, ML tracking, plugins — while installing as a single Python package and running entirely on your machine. This post explains the architecture that makes that possible, at the level of detail you'd want before contributing or embedding it.
The one-process shape
The frontend — React 19, TypeScript, and React Flow for the node canvas, with Zustand and React Query for state — is built with Vite and shipped inside the PyPI wheel. One install serves both the UI and the API; there is no separate deployment, no Node.js requirement at runtime.
The backend is FastAPI with async SQLAlchemy 2.0 and Alembic migrations. Application state (projects, flows, datasets, connections, runs, schedules) lives in SQLite by default — a single ciaren.db file — with Postgres and MySQL supported when you outgrow it.
The execution engine
The interesting part. Flows are directed acyclic graphs of typed nodes, and execution goes through an engine abstraction with two backends: Polars (the default) and pandas, selectable per run. The engine choice is recorded on every run, so results are always attributable to the semantics that produced them.
A few deliberate decisions:
- Inputs are snapshotted first. Before execution, SQL queries and remote storage reads are materialized to local Parquet snapshots. Node execution then operates on stable local data — which is what makes per-node preview both fast and consistent with the eventual run.
- Execution stays off the event loop. Runs execute in a thread pool by default, or in a separate process per run for multi-core use, crash isolation, and real timeouts.
- Every node maps to real dataframe code. The same node definitions drive three code generators — pandas, eager Polars, and lazy Polars — which is what makes honest Python export possible.
The built-in catalog covers 80 nodes: transformation nodes for cleaning, reshaping, joins, window functions, calculated columns, and a Python script node, plus six data-quality assertion nodes, eight chart nodes, sixteen ML nodes, and the file, SQL, and cloud-storage inputs and outputs.
Connectors and secrets
Connectors are providers behind a common interface: PostgreSQL, MySQL/MariaDB, SQLite, DuckDB, SQL Server, Snowflake, MongoDB, local folders, S3, Azure Blob, GCS, and a read-only REST API connector — each an optional pip extra so the base install stays lean. Two security details worth noting: the REST connector ships with SSRF protection and path confinement, and credentials never enter the app database — they resolve from the OS keychain, environment variables, or secret files at use time.
ML without a platform
ML nodes (train/test split, scaling, encoding, training, prediction, evaluation, cross-validation, feature importance) wrap scikit-learn-compatible estimators — including XGBoost and LightGBM as extras — and log every training run to MLflow: parameters, metrics, and the model artifact. The default tracking store is a local directory; pointing at a remote MLflow or Databricks server is a configuration change, not an architecture change.
The plugin system
Everything above is also an extension point. The plugin API — a separate, Apache-2.0-licensed contract package — defines ten provider interfaces: nodes, connectors, models, storage, execution, exporters, validators, AI, auth, and licensing. Plugins ship as .ciarenplugin archives with a manifest, Ed25519 signatures, declared permissions (shown to the user before install), and a trust-tier model. A dedicated ciaren-plugin CLI handles packing, signing, verifying, and installing.
This is also the honest answer to "will Ciaren add AI features?": AI arrives as a provider you opt into, never as a required dependency of the core.
Why this shape
Every choice above serves the same three commitments: local-first (one process, local files, no mandatory network), no lock-in (real code export, open formats, AGPL core), and extensible without forking (stable plugin contracts). If you want to poke at any of it, the source is open — and the contributing guide marks the good first issues.