Ciaren

Storage output (S3 / GCS / Azure Blob / Local)

Storage output — storageOutput

Write the result of a flow to a cloud object store (AWS S3, Google Cloud Storage, Azure Blob Storage) or a local folder via a reusable Storage Connection. The result is serialized and uploaded after the rest of the flow succeeds — a write failure marks the run failed.

Storage Input
s3://raw-data/orders.csv
input
Drop Nulls
clean
Group By + Aggregate
revenue by region
transform
Storage Output
s3://reports/summary.parquet · overwrite
output

Use cases

  • Land a cleaned or aggregated result in a company bucket for downstream consumers.
  • Write a daily report parquet to GCS on a schedule.
  • Archive processed outputs to Azure Blob alongside raw files.

Configuration

Config keyTypeRequiredDescription
connection_idstringYesThe storage connection to write to
pathstringYesDestination path inside the bucket/container/folder (e.g. reports/summary.parquet)
formatstringNoparquet (default), csv, tsv, excel, json, jsonl, or text
if_existsstringNooverwrite (default) or error

Generated Python code

The exported script is portable, so it never embeds cloud credentials or an upload call tied to one provider's SDK. Instead it leaves a marker for the write:

# storageOutput: write df_4 to your configured storage target

Tips & common mistakes

  • Parquet is the default format and the best choice for downstream data pipelines — it preserves types and compresses well. Use csv if the destination requires it.
  • overwrite replaces the object silently. Set if_exists: error to abort rather than overwrite an existing file, which is useful in append-style workflows where you'd rather fail loud than silently clobber old data.
  • The run is marked failed if the upload fails. No partial writes are left behind — Ciaren pushes the output only after every upstream node succeeds.

See also

PreviousStorage input (S3 / GCS / Azure Blob / Local)NextDrop columns