A new pile from one line: Python comprehensions and generators
A list comprehension says what it wants first, and a generator gives back one item at a time. Following Chapters 5 and 9 of the official Python Tutorial: list, dict and set comprehensions, double for, nested comprehensions, * unpacking, yield, generator functions, generator expressions, and when to pick a list over a generator — plus a 100-mistake breakdown from a fictional bookstore script.

First, the boundaries of this guide. This is the sixth guide in the "Python Basics" series, continuing from the first five: the first covered variables, types and control flow, the second functions and modules, the third classes and inheritance, the fourth exception handling, and the fifth file input and output. This one is about list comprehensions and generators: writing "pick these out of a pile and compute a new pile" in a single line, and the "one at a time, computed as needed" approach. The "Morning Light Bookstore" below is still the same fictional practice project: a small street-side bookshop whose owner writes a little bookkeeping script that can already work out stock, apply discounts, manage books with classes, catch errors, and save the books to files. The new problem here is that the delivery note comes in as a pile of dictionaries, and jobs like "pick out the books with fewer than 10 copies in stock" or "work out each book's discounted price" should not take five or six lines of empty list plus a loop plus append() — one line does it. Everything said here about comprehensions and generators comes from Chapter 5 of the official Python Tutorial, "Data Structures", and sections 9.9 and 9.10 of Chapter 9, "Classes"; sources are listed in each guide's meta information and evidence record. Every Morning Light Bookstore operation, code sample and number in this guide was invented to explain the technique and is not a real record. By the end you should be able to do five things: write list comprehensions with if filters; write double-for and nested comprehensions, and know when to stop; write dict and set comprehensions; write a generator function with yield and know how it differs from a normal function; and use generator expressions to feed sum() and max() as you go. The boundary with the first guide is this: guide 1 only mentioned the names "list comprehension" and "dict comprehension" once, with a single stock-value example, and never explained the syntax, if filters, or generators; this guide covers the full syntax and generators in one place. The boundary with the third guide is: guide 3 wrote an iterator with __iter__ and __next__ in a class; the generator here is the short way of writing the same thing, and the iterator protocol is not re-taught.
Step 1: List comprehensions — the short form of "empty list, then a loop"
Section 5.1.3 of the tutorial says list comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition. The official example first computes the squares of 0 to 9:
>>> squares = []
>>> for x in range(10):
... squares.append(x**2)
...
>>> squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Note that this creates a variable named x that still exists after the loop completes. The tutorial says the same list of squares can be calculated without any side effects:
squares = list(map(lambda x: x**2, range(10)))
or, equivalently:
squares = [x**2 for x in range(10)]
The tutorial's verdict is that the comprehension is more concise and readable. A list comprehension consists of brackets containing an expression followed by a for clause, then zero or more for or if clauses. The result is a new list resulting from evaluating the expression in the context of the for and if clauses which follow it. The bookstore version: the total stock value needs no adding loop — the one-line form from guide 1 was a comprehension all along:
books = [('The Three-Body Problem', 59.0, 7), ('Twenty Thousand Leagues Under the Seas', 39.5, 12), ('To Live', 29.0, 5)]
totals = [price * stock for title, price, stock in books]
# totals is [413.0, 474.0, 145.0]
total = sum(totals) # 1032.0
The three books' stock values are 413.0, 474.0 and 145.0, adding up to 1032.0 — the same as the hand calculation and the file calculation in guide 5. The title, price, stock in the comprehension unpacks each row's tuple as the loop goes. My judgement is: a comprehension is not a showy shortcut, it is a way of "putting the intention first" — write what you want first, then where it comes from and what it has to satisfy, so the reader sees at a glance what the line produces.
Step 2: Add an if — filtering, and functions inside the expression
A comprehension can carry an if clause that keeps only the members that satisfy a condition. The official example uses a list with negative numbers:
>>> vec = [-4, -2, 0, 2, 4]
>>> # create a new list with the values doubled
>>> [x*2 for x in vec]
[-8, -4, 0, 4, 8]
>>> # filter the list to exclude negative numbers
>>> [x for x in vec if x >= 0]
[0, 2, 4]
>>> # apply a function to all the elements
>>> [abs(x) for x in vec]
[4, 2, 0, 2, 4]
>>> # call a method on each element
>>> freshfruit = [' banana', ' loganberry ', 'passion fruit ']
>>> [weapon.strip() for weapon in freshfruit]
['banana', 'loganberry', 'passion fruit']
The expression can hold a function call or a method call: abs(x) calls a function on each element, weapon.strip() calls a method on each element. Note that the loop variable in the freshfruit example is called weapon — that is how the official tutorial itself writes it. The name does not match the content, but the code runs fine; treat it as a reminder to name your own loop variables so people can understand them, and not to copy this example. The bookstore version: pick out the books with fewer than 10 copies in stock, then work out the 10%-off price of each:
books = [('The Three-Body Problem', 59.0, 7), ('Twenty Thousand Leagues Under the Seas', 39.5, 12), ('To Live', 29.0, 5)]
low_stock = [title for title, price, stock in books if stock < 10]
# low_stock is ['The Three-Body Problem', 'To Live']
discounted = [round(price * 0.9, 2) for title, price, stock in books if price > 50]
# discounted is [53.1]; only The Three-Body Problem costs more than 50
titles = [title.strip() for title, price, stock in books]
# titles is ['The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas', 'To Live']
The Three-Body Problem at 7 copies and To Live at 5 are short of 10; Twenty Thousand Leagues Under the Seas at 12 is fine; only The Three-Body Problem, at 59.0, costs more than 50, and 10% off is 53.1. My judgement is: an if inside a comprehension reads as "things like this, from here, but only the ones that satisfy this condition" — the more specific the condition, the more accurate the pick; but do not put more than two conditions in one comprehension, or split it into two steps.
Step 3: Double for and nested comprehensions — learn to read them before deciding to use them
A comprehension can have two for clauses. The official example pairs up the elements of two lists, keeping only the unequal pairs:
>>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
It is equivalent to two nested loops plus append():
>>> combs = []
>>> for x in [1,2,3]:
... for y in [3,1,4]:
... if x != y:
... combs.append((x, y))
...
>>> combs
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
The tutorial notes that the order of the for and if statements is the same in both snippets — whichever loop comes first in the loops comes first in the comprehension. If the expression is a tuple, such as (x, y) here, it must be parenthesized; leaving the parentheses out is a syntax error:
>>> [(x, x**2) for x in range(6)]
[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)]
>>> [x, x**2 for x in range(6)]
File "<stdin>", line 1
[x, x**2 for x in range(6)]
^^^^^^^
SyntaxError: did you forget parentheses around the comprehension target?
Two for clauses can also flatten a nested list:
>>> vec = [[1,2,3], [4,5,6], [7,8,9]]
>>> [num for elem in vec for num in elem]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
The initial expression in a list comprehension can itself be another list comprehension — a nested comprehension. The official example transposes a 3-by-4 matrix:
>>> matrix = [
... [1, 2, 3, 4],
... [5, 6, 7, 8],
... [9, 10, 11, 12],
... ]
>>> [[row[i] for row in matrix] for i in range(4)]
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
The inner list comprehension is evaluated in the context of the for that follows it, so the whole thing amounts to looping over columns outside and rows inside. The tutorial adds a practical note: in the real world, you should prefer built-in functions to complex flow statements — zip() does this job better:
>>> list(zip(*matrix))
[(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)]
The bookstore version: the delivery note is split into three groups, one list each, flattened into one master list:
groups = [['The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas'], ['To Live'], ['The Three-Body Problem', 'To Live']]
all_titles = [title for group in groups for title in group]
# all_titles is ['The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas', 'To Live', 'The Three-Body Problem', 'To Live'], 5 items
pairs = [(title, price) for title, price, stock in
[('The Three-Body Problem', 59.0, 7), ('Twenty Thousand Leagues Under the Seas', 39.5, 12)]]
# pairs is [('The Three-Body Problem', 59.0), ('Twenty Thousand Leagues Under the Seas', 39.5)]
My judgement is: a double-for comprehension reads as "outer loop first, inner loop second", exactly the same order as writing two nested loops; nested comprehensions are worth being able to read, not a daily writing style — past two levels, or past 80 characters on one line, split it into a plain loop, and when the tutorial itself says to use zip(), use zip().
Step 4: Dict and set comprehensions — change the brackets, change the product
Change the brackets of a comprehension and the product changes. A dict comprehension uses braces with a colon; the official example:
>>> {x: x**2 for x in (2, 4, 6)}
{2: 4, 4: 16, 6: 36}
A set comprehension uses braces with no colon; the official example:
>>> a = {x for x in 'abracadabra' if x not in 'abc'}
>>> a
{'r', 'd'}
Removing a, b and c from 'abracadabra' leaves only the distinct letters r and d; the set removes duplicates by itself. Note that an empty set cannot be written as {} — that is an empty dictionary; an empty set has to be written set(), as section 5.4 of the tutorial says. Dictionaries can also be merged with ** unpacking:
>>> odds = {i: i**2 for i in (1, 3, 5)}
>>> evens = {i: i**2 for i in (2, 4, 6)}
>>> {**odds, **evens}
{1: 1, 3: 9, 5: 25, 2: 4, 4: 16, 6: 36}
The bookstore version: a title-to-price dictionary, a title-to-sale-price dictionary, and the set of every title that has appeared:
books = [('The Three-Body Problem', 59.0, 7), ('Twenty Thousand Leagues Under the Seas', 39.5, 12), ('To Live', 29.0, 5)]
prices = {title: price for title, price, stock in books}
# prices is {'The Three-Body Problem': 59.0, 'Twenty Thousand Leagues Under the Seas': 39.5, 'To Live': 29.0}
sale_prices = {title: round(price * 0.9, 2) for title, price, stock in books if price > 50}
# sale_prices is {'The Three-Body Problem': 53.1}
groups = [['The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas'], ['To Live'], ['The Three-Body Problem', 'To Live']]
seen = {title for group in groups for title in group}
# seen is {'The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas', 'To Live'}; The Three-Body Problem appears twice but is kept once
My judgement is: the three comprehensions are one mapping to remember — square brackets make a list, braces with a colon make a dict, braces without a colon make a set; the expression and the for/if parts are written identically in all three, so learning one means learning all three. A dict comprehension has one more property: a repeated key is overwritten by the later value, the same rule as assigning to a dictionary directly.
Step 5: * unpacking — fine in a list, version-dependent inside a comprehension
Section 5.1.5 of the tutorial covers * unpacking: prefixing an expression by a * unpacks the result of that expression, adding each of its elements to the list being created:
>>> x = [1, 2, 3]
>>> [0, *x, 4, 5, 6]
[0, 1, 2, 3, 4, 5, 6]
What follows the * must evaluate to an iterable object; an integer raises an exception:
>>> x = 1
>>> [0, *x, 2, 3, 4]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
[0, *x, 2, 3, 4]
TypeError: Value after * must be an iterable, not int
Unpacking has one version trap to flag first. Section 5.1.5 of the tutorial also shows unpacking used inside a comprehension:
>>> x = [[1, 2, 3], [4, 5, 6], [], [7], [8, 9]]
>>> [*element for element in x]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
But that code does not run on a local Python 3.12 — it raises SyntaxError: iterable unpacking cannot be used in comprehension. Using * unpacking inside a comprehension is a form from the official 3.15 documentation that older versions do not accept. To combine several groups of book lists into one master list, use the double-for form from step 3, which works on every version:
groups = [['The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas'], ['To Live'], ['The Three-Body Problem', 'To Live']]
all_titles = [title for group in groups for title in group]
# all_titles is ['The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas', 'To Live', 'The Three-Body Problem', 'To Live'], 5 items
Unpacking in a list literal has no version problem, so use it freely when building a shelf:
shelf = ['The Three-Body Problem', *['Twenty Thousand Leagues Under the Seas', 'To Live'], 'The Three-Body Problem']
# shelf is ['The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas', 'To Live', 'The Three-Body Problem'], 4 items
The tutorial also notes that unpacking works for arbitrary iterable objects, not just lists — strings and dictionaries are unpacked too:
>>> x = [[1, 2, 3], 'cat', {'spam': 'eggs'}]
>>> [*element for element in x]
[1, 2, 3, 'c', 'a', 't', 'spam']
That is likewise a form from the official 3.15 documentation, and only runs on newer versions; 'cat' comes apart into three letters, and the dictionary comes apart leaving only its key, 'spam'. My judgement is: * unpacking itself has no version problem — it is putting it inside a comprehension that does. When writing code, first think about which Python version will run it; when in doubt, join things with a double for, which every version accepts.
Step 6: Generator functions — yield gives back one at a time
A comprehension builds the whole list at once; a generator does the opposite, giving back one item at a time and computing the next only when asked. Section 9.9 of the tutorial: generators are a simple and powerful tool for creating iterators. They are written like regular functions but use the yield statement whenever they want to return data. Each time next() is called on it, the generator resumes where it left off — it remembers all the data values and which statement was last executed. The official example yields a string backwards:
def reverse(data):
for index in range(len(data)-1, -1, -1):
yield data[index]
>>> for char in reverse('golf'):
... print(char)
...
f
l
o
g
The key difference is that calling reverse('golf') does not run the function body at once — it only creates a generator object. The body really starts running on the first next() call, or the first time it enters a loop. When generators terminate, they automatically raise StopIteration — a name guide 3 mentioned when it covered iterators, and the for loop relies on it to know when to stop. The tutorial also says that anything that can be done with generators can also be done with class-based iterators using __iter__ and __next__; what makes generators so compact is that those methods are created automatically, and local variables and execution state are automatically saved between calls. The bookstore version: yield one in-stock book at a time, skipping anything with zero stock:
def in_stock_books(books):
"""Yield one in-stock book at a time; skip anything with zero stock."""
for title, price, stock in books:
if stock > 0:
yield title
books = [('The Three-Body Problem', 59.0, 7), ('Twenty Thousand Leagues Under the Seas', 39.5, 12), ('To Live', 29.0, 0)]
gen = in_stock_books(books)
next(gen) # 'The Three-Body Problem'
next(gen) # 'Twenty Thousand Leagues Under the Seas'
next(gen) # raises StopIteration; To Live has zero stock and was skipped
list(in_stock_books(books)) # ['The Three-Body Problem', 'Twenty Thousand Leagues Under the Seas']
To Live has a stock of 0, so out of three books the generator yields only two. My judgement is: a generator function's body "reads like a normal function but runs like taking a numbered ticket" — each next() moves one step forward and stops at the yield to wait; to take everything at once, wrap it in list(), but that gives up the "compute one at a time" advantage.
Step 7: Generator expressions — the comprehension's brackets swapped for parentheses
Some simple generators can be coded succinctly as expressions using a syntax similar to list comprehensions but with parentheses instead of square brackets — generator expressions. Section 9.10 says these expressions are designed for situations where the generator is used right away by an enclosing function. Generator expressions are more compact but less versatile than full generator definitions and tend to be more memory friendly than equivalent list comprehensions. The official examples:
>>> sum(i*i for i in range(10)) # sum of squares
285
>>> xvec = [10, 20, 30]
>>> yvec = [7, 5, 3]
>>> sum(x*y for x,y in zip(xvec, yvec)) # dot product
260
>>> data = 'golf'
>>> list(data[i] for i in range(len(data)-1, -1, -1))
['f', 'l', 'o', 'g']
The sum of squares from 0 to 9 is 285: 0+1+4+9+16+25+36+49+64+81; the dot product is 260: 10 times 7 is 70, 20 times 5 is 100, 30 times 3 is 90, and they add up to 260. Note the parenthesis rule: in most cases, generator expressions must be wrapped in parentheses; but as a special case, when provided as the sole argument to a function — as in the sum() and list() examples above — the generator expression does not need an additional set of parentheses, so f(x for x in y) and f((x for x in y)) are semantically equivalent. The bookstore version: the total stock value without building a list first, letting sum() add as it goes:
books = [('The Three-Body Problem', 59.0, 7), ('Twenty Thousand Leagues Under the Seas', 39.5, 12), ('To Live', 29.0, 5)]
total = sum(price * stock for title, price, stock in books)
# total is 1032.0
most_expensive = max(price for title, price, stock in books)
# most_expensive is 59.0
low_titles = list(title for title, price, stock in books if stock < 10)
# low_titles is ['The Three-Body Problem', 'To Live']
My judgement is: a generator expression after sum(), max() or min() is the most common "one-line form" after comprehensions — the value is computed and used on the spot, and never sits around taking up memory; but if the result will be reused, indexed, or counted, skip the generator expression and write a list comprehension properly.
Step 8: List or generator — choose by "how many times, how much"
Comprehensions and generators look alike, and choosing wrong costs you. Three points decide it. First, how many times it is used: a list can be reused; a generator can only be walked once — walk it a second time and you get nothing:
gen = (x**2 for x in range(3))
list(gen) # [0, 1, 4]
list(gen) # [], it has already been walked
Second, how much memory it takes: a list builds all its results at once and keeps them; a generator computes one at a time. Section 9.10 of the tutorial says in so many words that generator expressions tend to be more memory friendly than equivalent list comprehensions. The ten-year transaction log from guide 5 fits here: a file object is itself an iterator that yields one line at a time, so for line in f reads a big file without loading it; to filter and compute as you read, put a generator expression after it:
# suppose each line of daily.txt is a number: no need to read the whole list first
with open('daily.txt', encoding='utf-8') as f:
total = sum(float(line.strip()) for line in f if line.strip())
Third, whether you need a count or an index: a list supports len() and indexing; a generator supports neither — calling len(gen) raises an error outright, because the generator itself does not know how many items are left. My judgement is: choosing takes only two questions — "how many times will I use this result?" and "how big is the data?" Used once and big: generator. Used many times, or counted and indexed: list. When the two answers fight — big data that still needs reusing — compute it once with a generator and store the result in a list, so the one-off calculation and the reusable list stay separate.
Step 9: Three traps — the loop variable, side effects, nesting too deep
The first trap is whether the loop variable "leaks out". A for loop from guide 1 leaves x behind afterwards; a comprehension does not:
squares = [x**2 for x in range(10)]
# the x inside the comprehension does not leak out
for x in range(3):
pass
# after this for loop, x is 2 and still there
Section 5.1.3 of the tutorial notes the same thing: the loop form creates a variable named x that still exists after the loop completes, while the comprehension has no such side effect. So do not count on using a comprehension's loop variable outside it. The second trap is side effects: a comprehension exists to "make new things", so do not put operations inside it that only run for their side effects, like append() or print() — the side effect runs once per expression evaluation, and anyone reading the code will think you are building a list. The third trap is nesting too deep: the transposed-matrix nested comprehension in step 3 takes a few seconds to understand; nest one more level and almost nobody can read it at a glance. My judgement is: a comprehension past two levels, or past 80 characters on one line, should be split into a plain loop or into two steps — the few lines saved are not worth the readability lost.
Step 10: Wrapping up — a comprehension says what it wants first, plus a mistake chart
Finally, this guide connects back to guide 5: the delivery note read in there is a list of dictionaries, and this guide's comprehensions are the natural tool for working with it — picking out the low-stock books, computing sale prices, building the set of titles, all in one line. One complete small example:
import csv
with open('stock.csv', newline='', encoding='utf-8') as f:
rows = list(csv.DictReader(f))
low_stock = [row['title'] for row in rows if int(row['stock']) < 10]
# low_stock is ['The Three-Body Problem', 'To Live']
total = sum(float(row['price']) * int(row['stock']) for row in rows)
# total is 1032.0
prices = {row['title']: float(row['price']) for row in rows}
# prices is {'The Three-Body Problem': 59.0, 'Twenty Thousand Leagues Under the Seas': 39.5, 'To Live': 29.0}
Note that DictReader returns price and stock as strings, so the comprehensions convert them with float() and int() before calculating — the loose end left by step 10 of guide 5.
Finally, this guide's mistake chart. Example: the fictional Morning Light Bookstore script, 100 comprehension and generator mistakes in its sixth month of learning Python, broken down below. The top mistake is using a generator a second time after it has already been walked, and getting nothing back.

Looking back over the ten steps, there are really only three things in this guide. First, a comprehension says what it wants first: square brackets make a list, braces with a colon make a dict, braces without a colon make a set, with the expression first and the for and if after it; a double for runs in the same order as two nested loops, and anything nested past two levels should be split apart. Second, a generator gives back one item at a time: writing yield in a function turns it into a generator function, calling it does not run it right away, each next() moves one step forward, and it raises StopIteration automatically when done; a generator expression is the short form with the square brackets swapped for parentheses, and it is at its best feeding sum() and max() as it goes. Third, choose by use: used once and big means generator; used many times, or counted and indexed, means list; a generator can only be walked once, and a comprehension's loop variable does not leak out. The next guide is a deep dive on dicts and sets: the dicts and sets that comprehensions produce — whether a list can be a key, whether order survives deduplication, and how get() and setdefault() divide the work — and the dict and set comprehensions from this guide will be back then.
Comments (0)
Sources
Related articles
A program that only keeps its books in memory loses them when it closes. Following Chapter 7 of the official Python Tutorial and the csv module docs: f-strings and format, str vs repr, open modes r/w/a/r+, with, five ways to read, write and tell/seek, JSON round-trips, and CSV with DictReader/DictWriter — plus a 100-mistake breakdown from a fictional bookstore script.
An exception is not a broken program: it is Python telling you what went wrong, with a name you can catch. Following Chapter 8 of the official Python Tutorial: syntax errors vs exceptions, try/except, matching by inheritance, which errors to catch and which to leave alone, raise, else, finally, with, user-defined exceptions, chaining, and ExceptionGroup — plus a 100-accident breakdown from a fictional bookstore script.
A class bundles data and its rules together: one Book class gives every book its own state, methods let objects do the work themselves, and inheritance lets an e-book stand on a paper book's shoulders. Following Chapter 9 of the official Python Tutorial: class and __init__, self, methods, class vs instance variables, attribute lookup, inheritance and overriding, super(), the underscore-private convention, dataclasses, __iter__ and __next__ — plus a 100-accident chart.