Skip to main content

How to Apply Generator Expressions to Any Iterable in Python

Python's Generator Expressions provide a memory-efficient way to process data. Unlike list comprehensions, which build the entire list in memory at once, generator expressions produce items one by one using lazy evaluation. This makes them ideal for processing large files, infinite streams, or complex transformation pipelines where intermediate results do not need to be stored.

This guide explains how to apply generator expressions to various iterable types (lists, files, dictionaries) and how to chain them for efficient data processing pipelines.

Basic Syntax (Parentheses vs. Brackets)​

The syntax for a generator expression is nearly identical to a list comprehension, but it uses parentheses () instead of square brackets [].

  • List Comprehension: [expr for item in iterable] -> Returns a list.
  • Generator Expression: (expr for item in iterable) -> Returns a generator object.
# List Comprehension: Creates the full list immediately
list_comp = [x**2 for x in range(5)]
print(f"List: {list_comp}")

# Generator Expression: Creates an iterator
gen_expr = (x**2 for x in range(5))
print(f"Generator: {gen_expr}")

# Accessing values
print(f"First value: {next(gen_expr)}")
print(f"Remaining values: {list(gen_expr)}")

Output:

List: [0, 1, 4, 9, 16]
Generator: <generator object <genexpr> at 0x7edefbc59d80>
First value: 0
Remaining values: [1, 4, 9, 16]

Applying to Different Iterables​

Generator expressions work on any iterable, not just lists.

Files (Reading Line by Line)​

This is the most common use case. Instead of loading a 5GB log file into RAM, process it line by line.

# Simulating a file object (normally open('file.txt'))
import io

file_content = "ERROR: Disk full\nINFO: Started\nERROR: Network down"
file_handle = io.StringIO(file_content)

# ✅ Create a generator to strip whitespace from lines
clean_lines = (line.strip() for line in file_handle)

# Iterate safely without memory spike
for line in clean_lines:
    print(f"Processing: {line}")

Output:

Processing: ERROR: Disk full
Processing: INFO: Started
Processing: ERROR: Network down

Dictionaries​

You can iterate over keys, values, or items.

prices = {'apple': 1.50, 'banana': 0.80, 'cherry': 2.00}

# ✅ Generator to calculate discounted prices
discounted = (price * 0.9 for price in prices.values())

print(f"Discounted prices: {list(discounted)}")

Output:

Discounted prices: [1.35, 0.7200000000000001, 1.8]

Custom Iterables​

Any class that implements __iter__ can be fed into a generator expression.

# Using a simple range object
r = range(10)

# ✅ Generator filtering even numbers
evens = (x for x in r if x % 2 == 0)

print(list(evens))

Output:

[0, 2, 4, 6, 8]

Chaining Generators (Pipelines)​

A powerful feature of generators is that you can chain them together. Data flows through the "pipeline" one item at a time. This allows you to break complex logic into small, readable steps without creating intermediate lists.

data = [1, 2, 3, 4, 5, 6]

# Step 1: Filter even numbers
evens = (x for x in data if x % 2 == 0)

# Step 2: Square them
squared = (x**2 for x in evens)

# Step 3: Convert to string
str_values = (f"Value: {x}" for x in squared)

# ✅ Execution happens HERE, one item at a time
for val in str_values:
    print(val)

Output:

Value: 4
Value: 16
Value: 36
note

In the example above, Python does not create three separate lists. It takes the first item 1, checks if even (no), takes 2 (yes), squares it (4), formats it ("Value: 4"), prints it, and only then moves to 3.

Memory Usage Comparison​

The memory difference becomes massive as dataset size increases.

import sys

# Large range
N = 1_000_000

# List Comprehension: Stores 1 million integers in RAM
list_obj = [x for x in range(N)]

# Generator Expression: Stores only the generation logic
gen_obj = (x for x in range(N))

print(f"List Size:      {sys.getsizeof(list_obj)} bytes")
print(f"Generator Size: {sys.getsizeof(gen_obj)} bytes")

Output (approximate):

List Size:      8697456 bytes
Generator Size: 112 bytes

Conclusion​

To use generator expressions effectively:

  1. Use () instead of [] to define them.
  2. Apply them to any iterable: Lists, Files, Dictionaries, or custom objects.
  3. Chain them to create efficient processing pipelines without intermediate storage.
  4. Use them for large datasets where loading everything into memory is impossible or slow.

Table of Contents