> For the complete documentation index, see [llms.txt](https://docs.cascade.io/cascade/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cascade.io/cascade/tools/transform/dedupe.md).

# Deduplicate

**Deduplicate** removes any duplicate values from the selected columns in an input table. When removing duplicate values, the tool will remove the entire row of duplicates found.

| Selection           | Description                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Fields**          | Select the field(s) in your table to search for duplicates in. Users can choose a single field or multiple, to remove duplicates of combinations of values across fields.                                                                                                                                                                                                                                                                                 |
| **Keep** (optional) | <p>Choose between 2 options:</p><ul><li><p>First <em>(default)</em> </p><ul><li>When duplicates are found in your table, this option will choose to keep the row of the <strong>first</strong> duplicate value found in the set of duplicates</li></ul></li><li><p>Last</p><ul><li>When duplicates are found in your table, this option will choose to keep the row of the <strong>last</strong> value found in the set of duplicates</li></ul></li></ul> |

### Configuration

Deduplicate allows users to easily remove duplicate values across one or multiple fields in the connected table. The Deduplicate tool is set to auto-run, which means that as soon as connecting to an input table, it will immediately run and search/remove duplicates across all fields in the table (aka any identical full rows of data). Simply open the tool and make your *Fields* and *Keep* selections to configure. &#x20;

<figure><img src="/files/jIg7XrKQcX7DQChLvYZZ" alt=""><figcaption></figcaption></figure>

As seen above, the *Fields* prompt will allow you to choose from a dropdown list of all of the columns in your table. Select one or more to run the deduplicate tool on.&#x20;

When configuring your Deduplicate tool, you will see two rows of numbers dynamically changing in between the *Fields* and *Keep* options.

<figure><img src="/files/XnXPU1xHAMF84p1L8VfG" alt=""><figcaption><p>Row counts represent the table size before and after removing duplicates</p></figcaption></figure>

Finally, make your *Keep* selection - decide whether you want to keep the row of the first instance of your duplicates or the last.&#x20;

### Example

Let's say we have a dataset of baseball players that needs some clean up. The dataset has taken in data from various sources and as a result has multiple duplicate values. To better show how the tool works, we've color coded the duplicate values.&#x20;

{% embed url="<https://datawrapper.dwcdn.net/6N2lX/2/>" %}
I
{% endembed %}

After connecting the above table to the Deduplicate tool, we'll configure by selecting `playerID` in the *Fields* prompt and "First" in the *Keep* Prompt.&#x20;

<div align="left"><figure><img src="/files/Oirm8rYvmTN4kMIzu2r3" alt=""><figcaption></figcaption></figure></div>

<div align="left"><figure><img src="/files/jENSvYSKP6OcaWYjC569" alt=""><figcaption></figcaption></figure></div>

As a result, we'll get the table below as our output. As you can see by the colors and data of the rows remaining, the Deduplicate tool removed all rows where there were duplicate `playerID` values and kept the first row of each duplicate value.&#x20;

{% embed url="<https://datawrapper.dwcdn.net/c0zP9/1/>" %}

### Outputs

The Deduplicate tool outputs two tables: one with all duplicate value rows removed and one with all duplicate value rows.