List Comprehensions and Generator Expressions In Python

development

List Comprehensions

List comprehensions are a common idiom in Python for creating a new list. The elements of the new list are the result of an operation applied to each element in another sequence or iterable. An overview can be found in the official Python documentation here.

List Comprehension v.s. For Loop

The basic form of a list comprehension looks like this:

>>> numbers = [i for i in range(10)]
>>> print(numbers)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

This can be thought of as shorthand for the code below, which creates an empty list, iterates over the range of numbers, and appends each number to the list:

>>> numbers = []
>>> for i in range(10):
>>>     numbers.append(i)
>>> print(numbers)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The code above is a pretty standard approach but it's not as concise nor as performant as the list comprehension version.

Calculations And Filtering In List Comprehensions

Other methods or operations can be applied to a list comprehension. The elements can either be modified in some way (with a calculation and/or method call) or filtered out based on certain criteria. The example below uses a list comprehension to square the same range of numbers and store them in a list (numbers). A second list is then created (even_numbers) using another list comprehension to filter out items from the first list (numbers):

>>> numbers = [i**2 for i in range(10)]
>>> print(numbers)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> even_numbers = [i for i in numbers if i % 2 == 0]
>>> print(even_numbers)
[0, 4, 16, 36, 64]

This example is contrived but it illustrates the syntax and some of the capabilities of list comprehensions.

List comprehensions are just another way to create a list and are pretty common. They aren't always appropriate but they can be a powerful tool when utilized in the right situations. Since the result of a list comprehension is a list this means, unsurprisingly, that they can be used as input to a function that accepts a list, such as the built-in sum function:

>>> sum([i for i in range(100)])
4950

Generator Expressions

In the example above the list comprehension creates a list containing the numbers from 0 to 99 and passes it to sum(). However, there is a better alternative to accomplish the same task. Enter the generator expression:

>>> sum(i for i in range(100))
4950

At first glance the example above looks exactly the same as the list comprehension version. The main difference in functionality is that a list comprehension creates the entire list in memory while a generator expression only creates an iterable. When an entire list is not required at any given time but rather just the individual elements in the list (such is the case for sum()) then a generator expression can be used.

In the example above, instead of creating the entire list at once for sum() to use, the result of the generator expression (an iterable) is sent to sum(). sum() gets the first number from the iterable, uses it in its calculation, gets the next number, uses it in its calculation, and repeats until the end is reached. At any point only the current number from the iterable exists in memory as opposed to the entire list. For a large sequence this means a much smaller memory footprint and potentially greater performance.

Conclusion

Both list comprehensions and generator expressions are useful tools. PEP 289 describes generator expressions as high performance, memory efficient generalizations of list comprehensions; however, the particular situation/problem will dictate which is more appropriate to use. If there is a need to iterate over a sequence more than once, store the items of a sequence, or use any list methods, then a list comprehension is the best choice. If only the individual elements of a list are required and just one iteration is needed, then a generator expression is a good option. The fact that the syntax is virtually identical between the two makes it easy to convert existing code from list comprehensions to generator expressions where it makes sense and vice versa.

View other posts