Deduplicate

Remove duplicate values in a column

Deduplicate removes any duplicate values from the selected columns in an input table. When removing duplicate values, the tool will remove the entire row of duplicates found.

Selection	Description
Fields	Select the field(s) in your table to search for duplicates in. Users can choose a single field or multiple, to remove duplicates of combinations of values across fields.
Keep (optional)	Choose between 2 options: First (default) When duplicates are found in your table, this option will choose to keep the row of the first duplicate value found in the set of duplicates Last When duplicates are found in your table, this option will choose to keep the row of the last value found in the set of duplicates

Configuration

Deduplicate allows users to easily remove duplicate values across one or multiple fields in the connected table. The Deduplicate tool is set to auto-run, which means that as soon as connecting to an input table, it will immediately run and search/remove duplicates across all fields in the table (aka any identical full rows of data). Simply open the tool and make your Fields and Keep selections to configure.

As seen above, the Fields prompt will allow you to choose from a dropdown list of all of the columns in your table. Select one or more to run the deduplicate tool on.

When configuring your Deduplicate tool, you will see two rows of numbers dynamically changing in between the Fields and Keep options.

Row counts represent the table size before and after removing duplicates

Finally, make your Keep selection - decide whether you want to keep the row of the first instance of your duplicates or the last.

Example

Let's say we have a dataset of baseball players that needs some clean up. The dataset has taken in data from various sources and as a result has multiple duplicate values. To better show how the tool works, we've color coded the duplicate values.

I

After connecting the above table to the Deduplicate tool, we'll configure by selecting playerID in the Fields prompt and "First" in the Keep Prompt.

As a result, we'll get the table below as our output. As you can see by the colors and data of the rows remaining, the Deduplicate tool removed all rows where there were duplicate playerID values and kept the first row of each duplicate value.

Outputs

The Deduplicate tool outputs two tables: one with all duplicate value rows removed and one with all duplicate value rows.