I Want to Remove Duplicates from a Python List • How Do I Do It?

Lists can have duplicate values. But sometimes, you don't want repeated items. Let's explore how to deal with this.

Another short article today to figure out ways to remove duplicate values from a list. The ideal solution depends on what you really need.

Let's explore.

Start With a List

Well, we need a list first–ideally, one with duplicate values. So, let's assume we have an online queue (line). But some people put their name in the queue more than once:

All code blocks are available in text format at the end of this article • #1 • The code images used in this article are created using Snappify. [Affiliate link]

Note how James and Kate were eager to ensure they were in the queue, so they put their name down twice.

Removing Duplicates: The Ugly Way

I was initially tempted not to include this section, but I changed my mind, as you can see. You can come up with several algorithms to perform this task "manually". It's only a few lines of code. Here's one option:

#2

You have a queue_unique empty list ready to collect unique names. Next, you iterate using enumerate() and add names to queue_unique if they don't appear in the rest of the original list. Note that I'm using slicing in the if statement to slice the list from index + 1 to the end of the list.

Let me show you another option. I'll discuss the outputs from these two manual versions later in this article:

#3

This time, you reverse the list so you can loop through the names in reverse order. The queue_unique doesn't start as an empty list this time but as a copy of the original reversed list.

In the loop, you remove names from queue_unique if the name appears later in the reversed list. A reminder that the .remove() list method only removes the first occurrence of an item. It doesn't remove all of them.

Both algorithms remove duplicates. Great. But compare the output from the two versions. The difference between these output lists gives a clue to what's coming next.

But I won't dwell on these versions any longer.

and PS: there are better versions of manual algorithms for this, but that's not the point of this first section, so let's move on!

---

---

Removing Duplicates: The Set Way

When you learn about data structures, you learn about the various characteristics they have. Then, you start comparing data structures based on these characteristics. For example, lists, dictionaries, tuples, and strings are all iterable. But lists and dictionaries are mutable, whereas tuples and strings are immutable. And lists, tuples, and strings are all sequences, but dictionaries are not–they're mappings. You can read more about some of these categories here: The Python Data Structure Categories Series

And some data structures enforce uniqueness while others don't. Lists, as you've seen above, can have several equal items–in the example above, you have several strings that are equal to each other.

However, sets are a Python data structure that can only have unique values:

#4

So, the easiest way to remove duplicates from a list is to cast it into a set:

#5

Or, if you prefer the output to still be a list, and perhaps you also want to overwrite the original variable name, then you can write the following:

#6

Now, that was easy! Much better than the several lines of code in the previous section.

However, there's an issue. If this is a queue of customers, then the order in which they joined the queue is somewhat important, I would say!

Note how the new queue list, the one without duplicates, no longer maintains the original order of the people within it. James was the first to join the queue, but Andy appears to have moved to the front when you removed duplicates.

Note that this also happened with the first of the manual algorithms in the previous section.

Sometimes, you don't care about the order of the elements in a list. If that's the case, you can cast the list into a set and then back into a list to remove duplicates.

But sometimes, the order matters. It certainly matters when dealing with a queue of customers. Let's look at another option.

Removing Duplicates: The Dictionary Way

A quick question before I carry on. Did you read the latest article I published just before this one? Here's the link: Are Python Dictionaries Ordered Data Structures?

If you haven't, now is a good time to read it. Like this one, it's a short article, so it won't take you too long.

You're back. Great.

So, you now know that since Python 3.7, there's a guarantee that the order of insertion of items in a dictionary is maintained. And dictionary keys must also be unique–you cannot have the same key appear twice in a dictionary.

Therefore, if you could create a dictionary from the elements in the list queue, you would remove duplicates but also maintain the order. And there's a dictionary class method for that:

#7

You create a dictionary from the list queue. The items in the list become keys, and each key has a default value of None. You can customise this default value, but you don't need to in this case, as you'll see in the next paragraph.

Great, you removed duplicates while maintaining order since dictionaries maintain order. The dictionary is created by iterating through the list, which explains why this version maintains the order of the items. But you don't want a dictionary, and you don't care about the values within it. So, you can cast this dictionary back into a list. You only keep the keys when you cast a dictionary into a list:

#8

You've now removed duplicates from the list and maintained the original order by converting the list into a dictionary and then back into a list.

Simple–once you know this idiom.

---

Do you want to join a forum to discuss Python further with other Pythonistas? Upgrade to a paid subscription here on The Python Coding Stack to get exclusive access to The Python Coding Place's members' forum. More Python. More discussions. More fun.

And you'll also be supporting this publication. I put plenty of time and effort into crafting each article. Your support will help me keep this content coming regularly and, importantly, will help keep it free for everyone.

---

A Limitation

Both the set and dictionary routes have an important limitation. Items in a set must be hashable objects. And keys in a dictionary must also be hashable. Therefore, you can't use these techniques if you have a list that includes non-hashable objects, such as a list that contains other lists.

You can read more about hashability and hashable objects in this post: Where's William? How Quickly Can You Find Him? • What's a Python Hashable Object?

Final Words

You may need to remove duplicates from a list in Python.

Don't write your own algorithm. Life's too short for that.

If you don't care about the order of the items in the list, cast the list into a set and then back into a list: list(set(queue))

If you do care about the order, create a dictionary from the list using dict.fromkeys() and then cast it back into a list: list(dict.fromkeys(queue)).

And the set and dictionary routes to removing duplicates are also more efficient than the manual ones shown above. So, it’s a win-win.

That's it.

Photo by Lisett Kruusimäe: https://www.pexels.com/photo/flowers-in-line-on-white-background-9510861/

Code in this article uses Python 3.13

The code images used in this article are created using Snappify. [Affiliate link]

You can also support this publication by making a one-off contribution of any amount you wish.

---

For more Python resources, you can also visit Real Python—you may even stumble on one of my own articles or courses there!

Also, are you interested in technical writing? You’d like to make your own writing more narrative, more engaging, more memorable? Have a look at Breaking the Rules.

And you can find out more about me at stephengruppetta.com

Further reading related to this article’s topic:

• A Slicing Story

• Iterable: Python's Stepping Stones

• Sequences in Python

• Finding Your Way To The Right Value • Python's Mappings

• Are Python Dictionaries Ordered Data Structures?

• Where's William? How Quickly Can You Find Him? • What's a Python Hashable Object?

---

Appendix: Code Blocks

Code Block #1

queue = ["James", "Kate", "Andy", "James", "Isabelle", "Kate"]

Code Block #2

queue_unique = []
for index, name in enumerate(queue):
    if name not in queue[index + 1:]:
        queue_unique.append(name)


queue_unique
# ['Andy', 'James', 'Isabelle', 'Kate']

Code Block #3

queue = ['James', 'Kate', 'Andy', 'James', 'Isabelle', 'Kate']
queue.reverse()
queue    
# ['Kate', 'Isabelle', 'James', 'Andy', 'Kate', 'James']

queue_unique = queue.copy()

for index, name in enumerate(queue):
    if name in queue[index + 1:]:
        queue_unique.remove(name)
       

queue_unique.reverse()
queue_unique
# ['James', 'Kate', 'Andy', 'Isabelle']

Code Block #4

set([1, 2, 3, 4, 3, 2, 1])
# {1, 2, 3, 4}

Code Block #5

queue = ["James", "Kate", "Andy", "James", "Isabelle", "Kate"]
set(queue)
# {'Andy', 'James', 'Kate', 'Isabelle'}

Code Block #6

queue = list(set(queue))
queue
# ['Andy', 'James', 'Kate', 'Isabelle']	

Code Block #7

queue = ["James", "Kate", "Andy", "James", "Isabelle", "Kate"]
dict.fromkeys(queue)
# {'James': None, 'Kate': None, 'Andy': None, 'Isabelle': None}

Code Block #8

queue = list(dict.fromkeys(queue))
queue
# ['James', 'Kate', 'Andy', 'Isabelle']

---

For more Python resources, you can also visit Real Python—you may even stumble on one of my own articles or courses there!

Also, are you interested in technical writing? You’d like to make your own writing more narrative, more engaging, more memorable? Have a look at Breaking the Rules.

And you can find out more about me at stephengruppetta.com