Python transform — pythonTransform
Run arbitrary Python code on the input dataframe and return a transformed one.
Use cases
- One-off transformations too complex or unusual to express with built-in nodes.
- Applying a custom business rule that mixes multiple pandas/polars operations in a single logical step.
- Prototyping a transformation before it becomes a permanent node in the registry.
- Using an installed library (
scikit-learn,faker, a proprietary package) directly inside the pipeline.
What it does
You write the body of a transform(df) function. The node wraps it, calls
it, and routes the returned dataframe to the next node in the graph. You must
return the result. The function receives the incoming frame; with pandas,
copy it before assignment when you want to avoid modifying it in place.
Configuration
| Config key | Type | Required | Description |
|---|---|---|---|
script | string | Yes | Body of def transform(df): … — must contain a return statement |
Pre-injected names
The script runs in a namespace with the dataframe engine already imported:
| Engine | Available names |
|---|---|
| polars (default) | pl, polars |
| pandas | pd, pandas |
These aliases are available inside a Ciaren run. Portable exports import pd
or pl; use those names in the body, or import another name explicitly. Import
other installed libraries with a standard import statement inside the script.
Strict mode permits data and computation imports such as math and numpy,
while blocking known capability imports such as os, subprocess and socket.
Syntax validation
The node checks syntax before preview and execution. It wraps your script
body in a function definition and runs compile() on it, so invalid indentation
and return placement are caught before the body runs. Preview reports a syntax
error as a validation error (HTTP 400).
Saving a flow does not perform this node syntax check. Preview the node after editing; a successful save alone does not mean the Python body is valid.
Generated Python code
These separate File Input → Python Transform → File Output flows read source.csv and
write result.parquet. Both keep amount > 0 and add margin = revenue - cost; zero,
negative and missing amounts are excluded. The pandas body copies the selected rows
before assignment. Choose the body for your flow engine: Ciaren embeds it as written and
does not translate pandas operations into Polars. Put the original CSV beside the script
and run in an environment with Ciaren and its file-format dependencies installed. These
complete exports use pd or pl and eager frames.
pandas
Transform
import pandas as pd
df_source = pd.read_csv('source.csv')
def _transform(df):
df = df[df['amount'] > 0].copy()
df['margin'] = df['revenue'] - df['cost']
return df
df_source = _transform(df_source)
df_source.to_parquet('result.parquet', index=False)Polars
Transform
import polars as pl
import pandas as pd
df_source = pl.read_csv('source.csv')
def _transform(df):
df = df.filter(pl.col("amount") > 0)
df = df.with_columns((pl.col("revenue") - pl.col("cost")).alias("margin"))
return df
df_source = _transform(df_source)
df_source.write_parquet('result.parquet')Examples
Filter and add a column (polars):
df = df.filter(pl.col("amount") > 0)
df = df.with_columns(
(pl.col("revenue") - pl.col("cost")).alias("margin")
)
return dfCall an installed library (pandas):
from sklearn.preprocessing import StandardScaler
import pandas as pd
scaler = StandardScaler()
df[['price_scaled']] = scaler.fit_transform(df[['price']])
return dfMulti-step string cleaning (pandas):
df = df.copy()
df['email'] = df['email'].str.strip().str.lower()
df = df[df['email'].str.contains('@', na=False)]
return dfSecurity note
No sandboxing
The script runs with the same permissions as the Ciaren server process — full access to the filesystem, network, and any installed packages. This is intentional: Ciaren is local-first and sandboxing would break legitimate use cases (file access, custom library calls).
Only run scripts from sources you trust, and never expose the Ciaren API publicly without authentication when Python Transform nodes are in use.
Setting CIAREN_PYTHON_TRANSFORM_STRICT=true adds two opt-in, defense-in-depth
checks: an AST scan that rejects dangerous imports (os, sys, subprocess,
socket, ctypes, importlib, …), exec-style builtins, and dunder-attribute
traversal; and a restricted __builtins__ at execution time with open and
eval removed. Import statements use a guarded importer that enforces the
same module policy as the AST scan; direct calls to __import__ are rejected
by the scan. It's off by default so existing scripts
(e.g. import numpy) keep working, and it is not a sandbox — a determined
attacker can still find a bypass — so the real controls remain network auth
and running Ciaren as an unprivileged user.
This setting protects validation and execution inside Ciaren. The exported Python file contains your script body and normal imports; it does not include the server's AST checks or restricted builtins. Review exported code before running it under your own account.
Tips & common mistakes
- You must
returnthe dataframe. Forgetting thereturncauses a runtime error:pythonTransform: script returned None. The preview will catch this before a full run. - Use the selected engine's operations.
pdandplrefer to different dataframe libraries. Switching engines does not translate a saved body; usepdfor pandas andplfor Polars in scripts you also plan to export. - Do not call
.collect()inside Polars scripts unless you know the frame is lazy. In lazy Polars exports, the exporter materializes this node's inputs and returns its result to the lazy plan. - For complex transformations that you'll reuse across flows, consider contributing a new node to the registry so it gains a proper config UI, validation, and test coverage.