Remove Duplicate Rows

Use the Remove Duplicates node to drop repeated rows — either fully identical rows or duplicates of a key column.

You'll use: File Input → Remove Duplicates → File Output.

File Input

contacts.csv

input

Remove Duplicates

subset: email · keep first

clean

File Output

contacts_unique.csv

output

Steps

File Input — select your dataset.
Remove Duplicates — configure how to match:
- All columns (default): drops rows that are identical across every column.
- By key: set subset: ["email"] to treat rows with the same email as duplicates, and keep: "first" (or "last") to choose which one survives.
File Output — write the de-duplicated result.

Before

3 rows · 2 cols

Remove Duplicates (subset: email, keep: first)

After

name	email
Ada	[email protected]
Lin	[email protected]

2 rows · 2 cols

Order matters for keep. Add a Sort Rows node first so "first" or "last" means what you expect (e.g. keep the most recent).
Just want to find duplicates? Use an Assert Unique data-quality node to fail or warn when duplicates appear, instead of dropping them.