Ciaren

Storage input — storageInput

Read a file from a cloud object store (AWS S3, Google Cloud Storage, Azure Blob Storage) or a local folder via a reusable Storage Connection. On every run the file is downloaded once and materialized to a parquet snapshot so credentials never cross the process boundary.

  1. 1Input
    Storage Input
    s3://my-bucket/sales/2024-06.csv
  2. 2Clean
    Change Types
    amount→float · date→datetime
  3. 3Transform
    Group By + Aggregate
    revenue by region
  4. 4Output
    Storage Output
    s3://my-bucket/reports/summary.parquet

Use cases

  • Read a daily export file from S3 in a scheduled flow so every run picks up the latest data.
  • Process a shared file stored in a company Azure Blob container.
  • Point at a local folder for development or self-hosted setups.

Configuration

Config keyTypeRequiredDescription
connection_idstringYesThe storage connection to read from
pathstringYesFile path inside the bucket/container/folder (e.g. data/sales.csv)
formatstringNocsv (default), tsv, excel, parquet, json, jsonl, or text
delimiterstringNoCSV only: , ; \t or |. Unset means auto-detect
encodingstringNoCSV/TSV: utf-8, utf-8-sig, latin-1, cp1252, utf-16, utf-16-le, or utf-16-be. Unset means auto-detect
decimalstringNoCSV/TSV: . or ,. Unset means auto-detect

The connection defines the provider, bucket/container, and how credentials are resolved from environment variables. The node only needs the relative path within that bucket.

CSV dialect detection

When you pick a CSV or TSV file, the config panel reads the first 64 KB of the file and shows what it found, for example Detected: Semicolon (;) · cp1252 · decimal comma. The Separator, Encoding, and Decimal mark fields stay on Auto-detect and show the detected value, so the preview splits columns correctly without any manual setup. Opening the node never changes its configuration.

  • Auto-detect: a field left on Auto-detect is detected again on every preview and run. This helps when an upstream system may change the file's format.
  • Override: pick a value in any of the three fields, or click Use detected values to pin the current detection. A value you set always wins over detection, on both engines and in exported code.
  • Exported code only carries the dialect you set. A field left on Auto-detect uses the pandas/polars default (comma, UTF-8, .) in the export, so pin the values before exporting a non-default file.
  • Fallback: if the sample shows no clear dialect (an empty file, a single column, binary data, or bytes that are neither UTF-8 nor Windows-1252), the panel doesn't show a "Detected" value. Reads then use the defaults: comma, UTF-8, and . as the decimal mark.
  • Changing the file or format clears the three fields, and Ciaren detects the new file.
  • Where it applies: the built-in S3, GCS, Azure Blob, and Local Storage connectors. Plugin storage connectors don't support dialect options. A node that sets them on a plugin connection fails with a clear error.

Detection reads only a bounded sample, through the same path checks as the read itself. For a local folder, paths can't escape the connection root or the directories in CIAREN_STORAGE_ALLOWED_ROOTS.

Generated Python code

The exported script is portable, so it never embeds cloud credentials. Instead it reads the object by its file name and tells you to download it first:

python
# storageInput: download 'data/sales.csv' from your storage connection first
df_sales = pd.read_csv('sales.csv')

When the node sets a dialect, the read includes it. For polars, the script decodes non-UTF-8 files first:

python
df_ventas = pd.read_csv('ventas.csv', sep=';', encoding='cp1252', decimal=',')

Tips & common mistakes

  • Create the connection first. Go to Connections → Add connection and pick the storage provider. Set the key's secret reference — a bare name or env:NAME for an environment variable, keyring:NAME for the OS keychain (recommended on desktop), or file:/path for a mounted secret file. Ciaren never stores the secret itself; see Connections.
  • Path is relative to the bucket/container root. Do not include the s3://bucket-name/ prefix — that comes from the connection.
  • Preview uses a bounded sample. Preview and profile read the first N rows to stay fast; the full run reads the whole file.

See also