CSV Deduplicator

Find and remove duplicate rows from CSV files based on selected columns.

Tested tool guide Tested browser tools Checked August 16, 2026

What CSV Deduplicator does, with a checked example

CSV Deduplicator identifies repeated records by comparing the columns you choose, then returns CSV data containing one row for each distinct selected-value combination. Selecting email checks uniqueness by email, while selecting email and account_id checks each pair. The common surprise is that rows can count as duplicates even when their unselected fields differ. Include every column needed to distinguish records, especially when other fields contain names, dates, statuses, balances, or notes that must not be discarded.

Worked example

A concrete input and expected output from the current implementation.

Input

CSV data:
id,name
1,Ada
1,Ada
2,Lin

Selected columns: id, name

Expected output

id,name
1,Ada
2,Lin

The two rows containing 1,Ada have identical values in both selected columns, so they form one duplicate group. The 2,Lin row has a different key and remains.

How the result is produced

Selected-column keys

The column selection defines the comparison key. For each data row, the values from those columns are considered together. Rows with the same combination belong to one duplicate group, and the deduplicated result contains one representative of that group. Selecting every column tests complete-row equality; selecting fewer columns tests uniqueness only at the chosen level.

CSV field boundaries

CSV syntax controls where one field ends and the next begins. Quoted commas belong inside a field rather than creating another column, and quote characters inside quoted fields require CSV escaping. A header, when present, identifies the available columns. Check that the input is divided into the intended fields, because structural problems can create false matches or hide real duplicates.

Good uses

Clean a mailing-list export by deduplicating on subscriber email before importing it into another system.
Reduce repeated order-line records by selecting the order identifier and line identifier together, rather than comparing descriptive text.
Produce one record per respondent from a CSV containing repeated survey submissions, using a stable respondent identifier as the key.

Limits and checks

Rows that agree in the selected columns can contain different information elsewhere. Collapsing their group may discard a meaningful version, so inspect conflicts or add more key columns.
Do not assume that spaces, capitalization, numeric forms such as 01 versus 1, or empty values are normalized. Test a small sample or standardize fields first when those distinctions matter.
The result establishes uniqueness only within the supplied CSV and the chosen columns. It cannot prove that an email, identifier, or transaction is unique in another file or source system.

Common questions

Must every field match for two rows to be duplicates?

No. Duplicate status follows the selected columns, not necessarily the entire row. Selecting only customer_id groups rows sharing that customer_id even if their address or balance differs. Selecting all columns narrows matching to complete parsed rows. Use a subset only when that subset genuinely defines the record's identity.

Which row is kept when matching rows differ elsewhere?

Do not infer a guaranteed survivor when duplicate-key rows differ outside the selected columns. For exact duplicate rows, the choice is irrelevant, as in the example. For conflicting records, inspect the result before depending on row order. Prefer resolving the conflicting values or adding distinguishing columns before deduplication.

References and verification

The example and behavioral notes were checked against the browser implementation. Standards and primary references below define the relevant format, formula, or platform behavior.