Remove duplicates — removeDuplicates

Drop duplicate rows.

Use cases

• Collapse exact duplicate records.
• Keep one row per key (e.g. one row per customer_id) by choosing a subset.

What it does

With subset: ["email"], any two rows sharing the same email are considered duplicates and only the first is kept.

Configuration

Generated Python code

df_2 = df_1.drop_duplicates()

Tips & common mistakes

• subset controls "duplicate by what". Without it, two rows must match on every column to count as duplicates.
• keep: false drops every copy of a duplicated row — use it to find rows that are truly unique.
• Pair with Sort rows first when keep: first/last should pick a specific record per key.

See also

• Sort rows · Group by + aggregate