Dropping Values (#2 in The `itertools` Series • `dropwhile()` and `takewhile()`)

Yteria has another gig. She still can't use the word "for". This time it's `itertools.dropwhile()` that comes to the rescue

Yteria was still in a bad mood. She was frustrated and annoyed. Even though it had been five days since that unpleasant incident when she lost the word "for", she was still angry.

And it was also affecting her work now. It wasn't just her mindset, which hadn't yet recovered from the incident. The loss of "for" was having serious consequences on her work as a freelance Python programmer.

Yteria's world is similar to the one you and I live in. But not quite identical. People can lose words. And adults can't replace lost words easily. Babies are born with no words–just like in our world. Small children can gain new words easily from parents, school, and their daily interaction with the world. But the ability to freely absorb new words diminishes as they grow.

By the time they're adults, people in Yteria's world must purchase new words. Some obtain them through nefarious means, like the thugs who stole "for" from Yteria the other day. Yteria didn't have the means to purchase "for"–common words are expensive.

This missing word caused her problems with her Python coding.

---

You can read Yteria's first experience with coding without "for" in the first article in the itertools series: Loss of Words • A Nested Journey (# 1 in The 'itertools' Series • 'product()')

---

Yteria's Gig for the Local Manufacturing Company

Yteria got a gig from a local manufacturing company. Her clients had just purchased a new detector to improve their manufacturing process. The company's boss had spent half an hour explaining how this new device would improve efficiency and quality control and other stuff…

Yteria wasn't interested. She just wanted to know what they needed from her. What they do with the new detector is their problem, not hers!

Each time they turn on the detector, the device outputs a number of null readings before starting to give the proper readouts. These readings show as 0 in the detector's data.

One of their employees manually went through each series of data to delete the first few entries, the ones with a value of 0. But this was too time-consuming. They asked Yteria to help them automate this data cleaning.

Here's what the data looks like:

All code blocks are available in text format at the end of this article • #1 • The code images used in this article are created using Snappify. [Affiliate link]

There are always a number of items equal to 0 at the beginning. Yteria needs to write code to remove these.

An easy job. And one she'll be able to finish quickly (and get paid quickly, too, she hopes).

The solution Yteria couldn't use

But then Yteria remembered. She knew there was a coding tool she'd like to use, but she couldn't think of the word she needed. And she can't write it, say it, or type it either.

Here's the code Yteria would have written if she could still use "for":

#2

The for loop in the function iterates over the enumerate object returned by enumerate(data). We'll cover enumerate() in a later article in this series. In the meantime, here's a brief demo of what it does:

#3

The built-in enumerate() returns an iterator that yields tuples. Each tuple includes an index and the item from the original iterable. The built-in enumerate() is useful when you also need the index when iterating over an iterable.

In the trim_data() function, enumerate() keeps track of the index as you loop through the items in data. When the code finds the first non-zero element, the for loop breaks. The index at this point is the index of this element. So, the function returns the slice data[index:], which starts from the first non-zero element and runs through to the end of the list.

The output is a list of data starting from the first non-zero element:

[4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

Note that it's possible for values later in the data to be 0. The code should only remove the first series of zeros.

Even better, there's no need to return a list. The function could return an iterator–a generator in this case:

#4

The return statement now includes a generator expression. Note that you cast this into a list to display its values within print():

[4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

Another alternative that also produces a generator is the following:

#5

You create an iterator from the list data using the built-in iter() function. The for loop starts consuming elements from data, which is now an iterator. Therefore, when the loop breaks, data is still an iterator, but its first few elements have been used up.

However, you've also used up the first non-zero element. But this value is still stored in the for loop variable num. So, you start by yielding num. Then, you yield the rest of the items from data using yield from.

The function trim_data() is now a generator function since it includes yield or yield from. This code gives the same output:

[4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

Note that you can use yield from in the previous example, too, instead of creating the generator expression. But this article is about itertools.dropwhile(), so let's not dwell too much on these solutions here!

Great, we have a few solutions. But Yteria can't use any of these. They all use the for loop. And Yteria can't type "for"!

Yteria's itertools Solution

Ever since she lost the word "for", Yteria had discovered the itertools module. She found lots of iteration solutions already implemented in this module.

…and she found one to solve her detector problem, too:

#6

And the output from this code shows the data excluding the leading zeros:

[4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

The dropwhile() function in the itertools module performs a similar task to the trim_data() function you defined earlier. The dropwhile() function needs two arguments:

• A callable that returns True or False

• An iterable

Each element from the iterable is passed to the callable, which in this case is the 'lambda' function (lambda item: item == 0). Since the first few elements are 0, the lambda function returns True. The dropwhile() function drops these items. In fact, the dropwhile() function drops items while the condition returns True.

However, when dropwhile() finds the first item that returns False for the condition in the lambda function, it returns an iterator with the rest of the data.

And that's it, really!

---

Do you want to join a forum to discuss Python further with other Pythonistas? Upgrade to a paid subscription here on The Python Coding Stack to get exclusive access to The Python Coding Place's members' forum. More Python. More discussions. More fun.

And you'll also be supporting this publication. I put plenty of time and effort into crafting each article. Your support will help me keep this content coming regularly and, importantly, will help keep it free for everyone.

---

Returning Iterators and Other Patterns

As you go through the itertools series here on The Python Coding Stack, you'll note some recurring themes. The dropwhile() function returns an iterator. All functions in the itertools module return an iterator.

This fits in nicely with Python's iterator protocol.

You may also find the arguments required by itertools.dropwhile() familiar, especially if you've used tools such as map(). The requirement for a callable as the first argument and an iterable as the second is not unique to dropwhile(). You'll find this pattern elsewhere in the itertools module and in Python. Each element of the iterable–the second argument–is passed to the callable in the first argument. This callable is often a lambda function, but it doesn't have to be.

Exploring itertools.dropwhile() and itertools.takewhile()

Let's spend a few short paragraphs exploring dropwhile() and its sibling, takewhile().

And let's change things a bit:

#7

You need your data to start with a value that's a multiple of 5. So, you want to drop all the values at the beginning of your list of numbers that aren't divisible by 5:

#8

As we mentioned earlier, dropwhile() returns an iterator. You can use this iterator in a for loop or in any other iteration process. Or you can cast it to a list to display the values:

#9

The dropwhile() function drops values while they leave a non-zero remainder when divided by 5. As soon as the function finds the first value that's divisible by 5, it returns an iterator that yields all the remaining values. Note how the first two values from the original data, 2 and 6, are dropped.

The dropwhile() function drops values from the beginning of an iterable. What if you want to drop values from the end?

In this case, you can use takewhile(), which you'll also find in the itertools module. This function takes values while the values meet a certain condition.

Let's use the same list numbers and keep all the values until we encounter the first value that's divisible by 7:

#10

The first value that's a multiple of 7 is the seventh element in the list numbers, 14. Therefore, takewhile() keeps all the values up to but excluding 14.

The takewhile() function removes values from the end of the iterable.

Let's Compare the for Loop and itertools Solutions

Yteria was forced to use itertools.dropwhile() since she couldn't type "for". You don't have the same problem as she does. So why would you use this solution?

Simplicity and readability are part of the answer. Once you know about itertools.dropwhile(), it's easier to use it than to write your own code. It also makes your intent clearer, making your code more readable.

How about speed? Let's time the three solutions we explored in this article, two of which use a for loop. We'll only compare the solutions that use iterators, so we'll skip the first one that returned a list:

#11

You use a list with 10,000 random numbers, but you add a few zeros at the start.

The first two functions are two "homemade" solutions we explored earlier. The third one uses itertools.dropwhile(). They all return or create iterators, but to ensure a fair competition, you cast their outputs to a list when you time them using the timeit module.

Here's the output when I ran these. A reminder that you'll get different timings depending on your computer and what version of Python you're using:

First 'for' loop solution:
2.9326959580066614
​
Second 'for' loop solution:
2.3176030829999945
​
Using 'dropwhile()':
0.7492906670013326

As you can see, dropwhile() is the clear winner. The usual caveat applies: the improvement in performance depends on the size of your data and other factors.

Now, you might be thinking you can come up with a better solution than either of my first two solutions. And that's possibly true. But that is the point of dropwhile() and other tools in itertools. Instead of spending time and effort figuring out the optimal algorithm to solve your problem, you can use the option that's already included in Python. The itertools solution will almost certainly be the most efficient one!

We should also include a sanity check to make sure these functions return the same data:

#12

And here's the output:

True

Therefore, the three functions return the same data.

Until Next Time…

Yteria sent her code to her client. She was feeling better now. The thrill of finding a solution despite not being able to use the word "for" lifted her mood.

She brewed a cup of tea and went back to explore the itertools module. Clearly, she'll need to rely on it for a while longer.

Original Photo by Pixabay: https://www.pexels.com/photo/metal-pippings-with-pressure-gauge-372796/ Adapted to match other images in series

Code in this article uses Python 3.13

The code images used in this article are created using Snappify. [Affiliate link]

You can also support this publication by making a one-off contribution of any amount you wish.

---

For more Python resources, you can also visit Real Python—you may even stumble on one of my own articles or courses there!

Also, are you interested in technical writing? You’d like to make your own writing more narrative, more engaging, more memorable? Have a look at Breaking the Rules.

And you can find out more about me at stephengruppetta.com

Further reading related to this article’s topic:

• Loss of Words • A Nested Journey (# 1 in The itertools Series • product())

• What's All the Fuss About lambda Functions in Python

• The Anatomy of a for Loop

---

Appendix: Code Blocks

Code Block #1

data = [0, 0, 0, 0, 4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

Code Block #2

data = [0, 0, 0, 0, 4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

def trim_data(data):
    for index, num in enumerate(data):
        if num != 0:
            break
    return data[index:]

trimmed_data = trim_data(data)

print(trimmed_data)

Code Block #3

names = ["Yteria", "Silvia", "Jeff", "Stephen"]
enumerated_names = enumerate(names)
list(enumerated_names)
# [(0, 'Yteria'), (1, 'Silvia'), (2, 'Jeff'), (3, 'Stephen')]

Code Block #4

data = [0, 0, 0, 0, 4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

def trim_data(data):
    for index, num in enumerate(data):
        if num != 0:
            break
    return (value for value in data[index:])

trimmed_data = trim_data(data)

print(list(trimmed_data))

Code Block #5

data = [0, 0, 0, 0, 4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

def trim_data(data):
    data = iter(data)
    for num in data:
        if num != 0:
            break
    yield num
    yield from data

trimmed_data = trim_data(data)

print(list(trimmed_data))

Code Block #6

import itertools

data = [0, 0, 0, 0, 4, 6, 2, 34, 6, 2, 6, 0, 3, 4, 9]

trimmed_data = itertools.dropwhile(
    lambda item: item == 0,
    data,
)

print(list(trimmed_data))

Code Block #7

import itertools
numbers = [2, 6, 5, 4, 23, 10, 14, 15, 22, 29]

Code Block #8

# ...

itertools.dropwhile(lambda item: item % 5 != 0, numbers)
# <itertools.dropwhile object at 0x105a3f680>

Code Block #9

# ...

list(itertools.dropwhile(lambda item: item % 5 != 0, numbers))
# [5, 4, 23, 10, 14, 15, 22, 29]

Code Block #10

numbers = [2, 6, 5, 4, 23, 10, 14, 15, 22, 29]
list(itertools.takewhile(lambda item: item % 7 != 0, numbers))
# [2, 6, 5, 4, 23, 10]

Code Block #11

import random
import timeit

data = [0, 0, 0, 0, 0] + [random.randint(-50, 50) for _ in range(10_000)]

def trim_data_for_loop_1(data):
    for index, num in enumerate(data):
        if num != 0:
            break
    return (value for value in data[index:])

def trim_data_for_loop_2(data):
    data = iter(data)
    for num in data:
        if num != 0:
            break
    yield num
    yield from data

def trim_data_dropwhile(data):
    return itertools.dropwhile(
        lambda item: item == 0,
        data,
    )

repeats = 10_000
print("First 'for' loop solution:")
print(
    timeit.timeit(
        "list(trim_data_for_loop_1(data))",
        number=repeats,
        globals=globals(),
    )
)

print("Second 'for' loop solution:")
print(
    timeit.timeit(
        "list(trim_data_for_loop_2(data))",
        number=repeats,
        globals=globals(),
    )
)

print("Using 'dropwhile()':")
print(
    timeit.timeit(
        "list(trim_data_dropwhile(data))",
        number=repeats,
        globals=globals(),
    )
)

Code Block #12

# ...

print(
    list(trim_data_for_loop_1(data))
    == list(trim_data_for_loop_2(data))
    == list(trim_data_dropwhile(data))
)

---

For more Python resources, you can also visit Real Python—you may even stumble on one of my own articles or courses there!

Also, are you interested in technical writing? You’d like to make your own writing more narrative, more engaging, more memorable? Have a look at Breaking the Rules.

And you can find out more about me at stephengruppetta.com