10 May 2018

Python Generators

Generator functions are a special kind of function that return a lazy generator. Unlike lists, lazy generators do not store their contents in memory. We use them to work with data streams or large files (eg. CSV files).

Example 1: Large Files

In the following code open(file_name, "r") returns a generator, but file.read() loads the whole file into memory.

open(file_name, "r"):
result = file.read().split("\n")

Instead, we can yield each row instead of returning it:

for row in open(file_name, "r"):
        yield row

Example 2: Infinite Sequences

def infinite_sequence():
        num = 0
        while True:
                yield num
                num += 1

We can use it in a for-loop:

for i in infinite_sequence():
        print(i, end=" ")

# Output: 0 1 2 3 4 5 6 7 8 ...

Or we can also call next():

next(gen)

# Output: 0

next(gen)

# Ouptut: 1

Generator Expression (Generator Comprehension)

Similar to list comprehension but with generators.

csv_gen = (row for row in open(file_name))