Iterators

Advanced ~25 min read

An iterator is an object that implements the iterator protocol, allowing you to traverse through a sequence of values one at a time. Unlike lists or tuples that store all values in memory, iterators generate values on-demand, making them memory-efficient for large datasets. Understanding iterators unlocks powerful patterns for custom iteration behavior!

Iterables vs Iterators

An iterable is any object you can iterate over (lists, tuples, strings, dicts). An iterator is an object that produces the next value when you call next() on it. When you use a for loop, Python automatically converts the iterable into an iterator.

# Iterables vs Iterators

# Iterables: objects you can iterate over
my_list = [1, 2, 3, 4, 5]
my_tuple = ("a", "b", "c")
my_string = "hello"

# You can get an iterator from any iterable using iter()
list_iter = iter(my_list)
tuple_iter = iter(my_tuple)
string_iter = iter(my_string)

# Iterators: objects that produce values one at a time
print("List iterator:")
print(next(list_iter))  # 1
print(next(list_iter))  # 2
print(next(list_iter))  # 3

print("\nTuple iterator:")
print(next(tuple_iter))  # 'a'
print(next(tuple_iter))  # 'b'

print("\nString iterator:")
print(next(string_iter))  # 'h'
print(next(string_iter))  # 'e'

# Iterators are iterables too (they have __iter__ that returns self)
# But iterables are NOT necessarily iterators until converted
print("\nIterator is iterable:")
print(hasattr(list_iter, '__iter__'))  # True
print(hasattr(list_iter, '__next__'))  # True
print(hasattr(my_list, '__iter__'))    # True
print(hasattr(my_list, '__next__'))    # False - list is NOT an iterator

# for loops automatically create iterators
print("\nUsing for loop (automatic iteration):")
for item in my_list:
    print(item)

Output

Click Run to execute your code

Key Difference:
- Iterable: Has __iter__() method that returns an iterator
- Iterator: Has both __iter__() (returns self) and __next__() (returns next value)

All iterators are iterables, but not all iterables are iterators. Lists are iterables but not iterators until converted.

The Iterator Protocol

To create a custom iterator, you must implement two methods: __iter__() and __next__(). The __iter__() method returns the iterator object itself (or a new iterator object). The __next__() method returns the next value and raises StopIteration when there are no more items.

# The Iterator Protocol: __iter__ and __next__

class NumberSequence:
    """Custom iterator that implements the iterator protocol."""
    
    def __init__(self, start, end):
        self.current = start
        self.end = end
    
    def __iter__(self):
        """Returns the iterator object itself."""
        return self
    
    def __next__(self):
        """Returns the next value or raises StopIteration."""
        if self.current < self.end:
            value = self.current
            self.current += 1
            return value
        else:
            raise StopIteration  # Signal that iteration is complete

# Using the custom iterator
print("NumberSequence from 1 to 5:")
nums = NumberSequence(1, 5)

# Manual iteration
print("Manual iteration:")
print(next(nums))  # 1
print(next(nums))  # 2
print(next(nums))  # 3

# Using for loop (handles StopIteration automatically)
print("\nFor loop iteration:")
for num in NumberSequence(1, 5):
    print(num)

# Using iter() and next() explicitly
print("\nUsing iter() and next():")
it = iter(NumberSequence(10, 13))
print(next(it))  # 10
print(next(it))  # 11
print(next(it))  # 12
# next(it) would raise StopIteration here

Output

Click Run to execute your code

Pro Tip: You can use the built-in iter() function to get an iterator from any iterable, and next() to manually advance through values. This gives you fine-grained control over iteration!

Creating Custom Iterators

Custom iterators let you define exactly how iteration works for your objects. This is powerful for lazy evaluation, infinite sequences, or custom traversal logic.

# Creating Custom Iterators

class SquareNumbers:
    """Iterator that yields squares of numbers up to a limit."""
    
    def __init__(self, limit):
        self.limit = limit
        self.num = 0
    
    def __iter__(self):
        self.num = 0  # Reset for new iteration
        return self
    
    def __next__(self):
        if self.num < self.limit:
            square = self.num ** 2
            self.num += 1
            return square
        else:
            raise StopIteration

# Using the custom iterator
print("Square numbers up to 5:")
squares = SquareNumbers(5)

for sq in squares:
    print(sq)  # 0, 1, 4, 9, 16

# Iterator is consumed after use
print("\nTrying to iterate again (will be empty):")
for sq in squares:
    print(sq)  # Nothing prints - iterator is exhausted

# Create a new iterator
print("\nNew iterator:")
for sq in SquareNumbers(5):
    print(sq)

class CountDown:
    """Iterator that counts down from start to 1."""
    
    def __init__(self, start):
        self.current = start + 1  # +1 because we decrement first
    
    def __iter__(self):
        return self
    
    def __next__(self):
        self.current -= 1
        if self.current > 0:
            return self.current
        else:
            raise StopIteration

print("\nCountdown from 5:")
for num in CountDown(5):
    print(f"T-minus {num}")

Output

Click Run to execute your code

Memory Efficiency of Iterators

Unlike lists that store all values in memory, iterators generate values on-demand. This makes them perfect for large datasets or infinite sequences that wouldn't fit in memory.

# Memory Efficiency: Iterators vs Lists

import sys

# List stores ALL values in memory at once
def create_large_list(n):
    """Creates a list with n elements - all in memory."""
    return list(range(n))

# Iterator generates values on-demand
def create_large_iterator(n):
    """Creates an iterator that generates values on-demand."""
    for i in range(n):
        yield i

# Compare memory usage
n = 1000000

print(f"Memory comparison for {n} elements:")
print("-" * 50)

# List: all values stored in memory
large_list = create_large_list(n)
list_size = sys.getsizeof(large_list)
print(f"List size: {list_size:,} bytes")

# Iterator: minimal memory (just the iterator object)
large_iter = create_large_iterator(n)
iter_size = sys.getsizeof(large_iter)
print(f"Iterator size: {iter_size:,} bytes")

print(f"\nIterator uses {list_size // iter_size}x less memory!")
print(f"List: {list_size:,} bytes")
print(f"Iterator: {iter_size:,} bytes")

# Practical example: processing large files
print("\n" + "-" * 50)
print("Practical example: Processing large data")

# This would use huge memory with a list
def process_with_list(data):
    """Processes all data at once - memory intensive."""
    results = []
    for item in data:
        results.append(item * 2)
    return results

# This processes one item at a time - memory efficient
def process_with_iterator(data):
    """Processes data one item at a time - memory efficient."""
    for item in data:
        yield item * 2

# With list: all results stored
# With iterator: results generated on-demand
numbers = range(100)
print("Using iterator (generates on-demand):")
for result in process_with_iterator(numbers):
    if result > 10:  # Can stop early!
        break
    print(result, end=" ")
print("\n(Stopped early - didn't process all items)")

Output

Click Run to execute your code

Important: Once an iterator is exhausted (raises StopIteration), you can't reuse it. To iterate again, you need to create a new iterator by calling __iter__() or using iter() again. This is why you can only iterate over a file handle once!

Common Mistakes

1. Forgetting to raise StopIteration

# Wrong - will loop forever or return None
class Counter:
    def __iter__(self):
        self.count = 0
        return self
    
    def __next__(self):
        if self.count < 5:
            self.count += 1
            return self.count
        # Missing: raise StopIteration

# Correct - explicitly raise StopIteration
class Counter:
    def __iter__(self):
        self.count = 0
        return self
    
    def __next__(self):
        if self.count < 5:
            self.count += 1
            return self.count
        raise StopIteration  # Signal end of iteration

2. Trying to iterate over an exhausted iterator

# Wrong - iterator is exhausted after first loop
numbers = iter([1, 2, 3])
for n in numbers:
    print(n)

for n in numbers:  # This won't print anything!
    print(n)

# Correct - create a new iterator for each loop
numbers = [1, 2, 3]
for n in numbers:  # Automatically creates iterator
    print(n)

for n in numbers:  # Creates a new iterator
    print(n)

3. Confusing iterables with iterators

# Wrong - list is iterable but not an iterator
my_list = [1, 2, 3]
next(my_list)  # TypeError: 'list' object is not an iterator

# Correct - convert to iterator first
my_list = [1, 2, 3]
my_iterator = iter(my_list)
print(next(my_iterator))  # Works: 1
print(next(my_iterator))  # Works: 2

Task: Create a countdown iterator that counts down from a given number to 1, then raises StopIteration.

Requirements:

Implement a Countdown class with __iter__ and __next__ methods
Initialize with a starting number
Return numbers in descending order (start, start-1, ..., 1)
Raise StopIteration when countdown reaches 0
Test it with both a for loop and manual next() calls

Output

Click Run to execute your code

Show Solution

class Countdown:
    def __init__(self, start):
        self.start = start
        self.current = start
    
    def __iter__(self):
        self.current = self.start  # Reset for new iteration
        return self
    
    def __next__(self):
        if self.current > 0:
            value = self.current
            self.current -= 1
            return value
        else:
            raise StopIteration


# Test with for loop
print("Countdown from 5:")
for num in Countdown(5):
    print(num)

# Test with manual next()
print("\nManual iteration from 3:")
counter = Countdown(3)
it = iter(counter)
print(next(it))  # 3
print(next(it))  # 2
print(next(it))  # 1
# next(it) would raise StopIteration

Summary

Iterable: Object with __iter__() that can be looped over (lists, tuples, strings, dicts)
Iterator: Object with both __iter__() and __next__() that produces values on-demand
Iterator Protocol: Implement __iter__() (returns self) and __next__() (returns next value, raises StopIteration when done)
Memory Efficiency: Iterators generate values lazily, perfect for large datasets
One-time Use: Iterators are consumed after one pass; create new ones for multiple iterations
Built-in Functions: Use iter() to get an iterator from an iterable, next() to manually advance
StopIteration: Must raise this exception to signal end of iteration

What's Next?

Now you understand the iterator protocol! Next, we'll explore generators, which provide a simpler way to create iterators using the yield keyword. Generators are iterators under the hood, but Python handles all the __iter__ and __next__ boilerplate for you!

Previous Dataclasses

Next Generators

Iterables vs Iterators

The Iterator Protocol

Creating Custom Iterators

Memory Efficiency of Iterators

Common Mistakes

1. Forgetting to raise StopIteration

2. Trying to iterate over an exhausted iterator

3. Confusing iterables with iterators

Summary

What's Next?

Enjoying these tutorials?