The `.flow` document format

Status: draft, schemaVersion 1.0.0. Implemented in backend/app/flow_schema/. This is the public, versioned description of a flow.

A .flow document is JSON. It describes a flow's identity and node graph in an environment-independent way (no absolute paths, no embedded secrets), so it can be committed to git, generated by external tools, validated without the full app, and migrated across versions.

Today: flows persist in the database as React Flow graph_json; the backend already exports/imports an unversioned ciaren.flow/v1 document (POST /api/flows/{id}/export/python → flow_document, POST /api/flows/import). This spec formalizes that into the versioned document below. The legacy document upgrades via from_legacy_document() and downgrades via to_legacy_document().

Shape

{
  "schemaVersion": "1.0.0",
  "ciarenVersion": "0.1.0",
  "project": {
    "id": "optional-stable-id",
    "name": "My Flow",
    "description": "optional"
  },
  "graph": {
    "nodes": [
      { "id": "a", "type": "fileInput", "data": { "config": { "format": "csv" } } },
      { "id": "b", "type": "fileOutput", "data": { "config": { "format": "csv" } } }
    ],
    "edges": [{ "id": "e1", "source": "a", "target": "b" }],
    "engine": "polars",
    "parameters": []
  },
  "metadata": {},
  "requirements": {
    "plugins": [{ "id": "ciaren.databricks", "version": ">=1.0,<2.0", "required": true }],
    "capabilities": ["connector.sql", "engine.polars", "exporter.python"]
  }
}

Fields

Field	Required	Notes
`schemaVersion`	yes (defaulted)	Semver of this document format. Defaults to the current version.
`ciarenVersion`	no	The Ciaren build that wrote it (informational).
`project.name`	yes	Human name. `id` is an optional stable identifier.
`graph.nodes` / `graph.edges`	yes	React Flow-compatible. Extra graph keys (`engine`, `parameters`) are preserved.
`metadata`	no	Free-form.
`requirements.plugins`	no	Plugins (with PEP 440 version specifiers) the flow needs to open/run.
`requirements.capabilities`	no	Capability strings the host must provide (e.g. `connector.sql`).

Validation

Two layers, both in app/flow_schema/validate.py:

Schema shape — validate_document(data) (Pydantic).
Graph structure — graph_structure_issues(graph): node ids present and unique, node types present, edges reference existing nodes.

validate(data) runs both. missing_node_types(document, available) reports node types the host does not provide — the basis for a "install plugin X to open this project" message.

CLI:

ciaren flow validate path/to/project.flow
ciaren flow validate path/to/project.flow --output json

Migrations

Forward migrations are registered as from_version -> to_version and applied as a chain (app/flow_schema/migrations.py). There are none yet (first version, still 1.0.0), but the framework exists from day one. Migration never writes to disk on its own; the CLI keeps a .bak backup when --write is used:

ciaren flow migrate project.flow --write

Rules: never silently mutate a user project without a backup; keep migration tests; keep old schemas available for validation.

The `.flow` document format

The .flow document format

Shape

Fields

Validation

Migrations

The `.flow` document format