Ciaren

Storage output — storageOutput

Write the result of a flow to a cloud object store (AWS S3, Google Cloud Storage, Azure Blob Storage) or a local folder via a reusable Storage Connection. The result is serialized and uploaded after the rest of the flow succeeds — a write failure marks the run failed.

  1. 1Input
    Storage Input
    s3://raw-data/orders.csv
  2. 2Clean
    Drop Nulls
  3. 3Transform
    Group By + Aggregate
    revenue by region
  4. 4Output
    Storage Output
    s3://reports/summary.parquet · overwrite

Use cases

  • Land a cleaned or aggregated result in a company bucket for downstream consumers.
  • Write a daily report parquet to GCS on a schedule.
  • Archive processed outputs to Azure Blob alongside raw files.

Configuration

Config keyTypeRequiredDescription
connection_idstringYesThe storage connection to write to
pathstringYesDestination path inside the bucket/container/folder (e.g. reports/summary.parquet)
formatstringNoparquet (default), csv, tsv, excel, json, jsonl, or text
if_existsstringNooverwrite (default) or error

Generated Python code

The exported script is portable, so it never embeds cloud credentials or an upload call tied to one provider's SDK. Instead it leaves a marker for the write:

python
# storageOutput: write df_4 to your configured storage target

Tips & common mistakes

  • Parquet is the default format and the best choice for downstream data pipelines — it preserves types and compresses well. Use csv if the destination requires it.
  • overwrite replaces the object silently. Set if_exists: error to abort rather than overwrite an existing file, which is useful in append-style workflows where you'd rather fail loud than silently clobber old data.
  • The run is marked failed if the upload fails. No partial writes are left behind — Ciaren pushes the output only after every upstream node succeeds.

See also