Part 3: Iterators and Iterables

Iterators and Iterables are key language features of Python 3, that help to make code easy to write and maintain.

First, we are going to look at some background from Python 2, before moving on to cover the main topic.

Background

Python used to be all about sequences — a good chunk of anything you did was stored in a sequence or involved manipulating one.

  • lists

  • tuples

  • strings

  • dict.keys()

  • dict.values()

  • dict.items()

  • zip()

In Python2 those are all sequences.  It turns out, however, that the most common operation for sequences is to iterate through them:

for item in a_sequence:
    do_something_with_item

So fairly early in Python2, Python introduced the idea of the “iterable”.  An iterable is something you can, well, iterate over in a for loop, but often does not keep the whole sequence in memory at once.  After all, why make a copy of something just to look at all its items?
For example, in python2: ``dict.keys()`` returns a list of all the keys in the dict.  But why make a full copy of all the keys, when all you want to do is:
for key in dict.keys():
    do_something_with(key)

Even worse dict.items() created a full list of (key,value) tuples — a complete copy of all the data in the dict.  Yet worse enumerate(dict.items()) created a whole list of
(index, (key, value)) tuples — lots of copies of everything.
Python2 then introduced “iterable” versions of a number of functions and methods:

itertools.izip
dict.iteritems()
dict.iterkeys()
dict.itervalues()

So you could now iterate through that stuff without copying anything.

Python 3


Python3 embraces iterables — now everything that can be an iterator is already an iterator — no unnecessary copies.  An iterator is an iterable that has been made more efficient by removing as much from memory as possible. Therefore, if you need a list, you have to make the list explicitly, as in:
list(dict.keys())

Also, there is an entire module: itertools that provides nifty ways to iterate through stuff.  So, while we used to think in terms of sequences, we can now think in terms of iterables.


Iterators and Iterables
Let’s see an example of how Iteration makes Python code so readable:
for x in just_about_anything:
    do_stuff(x)
An iterable is anything that can be looped over sequentially, so it does not have to be a “sequence”: list, tuple, etc.  For example, a string is iterable. So is a set.
An iterator is an iterable that remembers state. All sequences are iterable, but not all sequences are iterators. To make a sequence an iterator, you can call it with iter:
my_iter = iter(my_sequence)
Iterables
———
To make an object iterable, you simply have to implement the __getitem__ method.
class T:
    def __getitem__(self, position):
        if position > 5:
         return position
iter()
——
How do you get the iterator object from an “iterable”?  The iter() function will make any iterable an iterator.  It first looks for the __iter__() method, and if none is found, uses get_item to create the iterator.  The ``iter()`` function:
In []: iter([2,3,4])
Out[]: <listiterator at 0x101e01350>
In []: iter("a string")
Out[]: <iterator at 0x101e01090>
In []: iter( ('a', 'tuple') )
Out[]: <tupleiterator at 0x101e01710>

List as an Iterator

In []: a_list = [1,2,3]
In []: list_iter = iter(a_list)
In []: next(list_iter)
Out[]: 1
In []: next(list_iter)
Out[]: 2
In []: next(list_iter)
Out[]: 3
In []: next(list_iter)
--------------------------------------------------
StopIteration     Traceback (most recent call last)
<ipython-input-15-1a7db9b70878> in <module>()
----> 1 next(list_iter)
StopIteration:

Use iterators when you can

Consider the example from the trigrams problem: (http://codekata.com/kata/kata14-tom-swift-under-the-milkwood/) You have a list of words and you want to go through it, three at a time, and match up pairs with the following word. The *non-pythonic* way to do that is to loop through the indices:

for i in range(len(words)-2):
    triple = words[i:i+3]

It works, and is fairly efficient, but what about:

    for triple in zip(words[:-2], words[1:-1], words[2:-2]):

 zip() returns an iterable --- it does not build up the whole list, so
 this is quite efficient.  However, we are still slicing: ([1:]), which
 produces a copy --- so we are creating three copies of the list ---
 not so good if memory is tight.  Note that they are shallow copies, so
 this is not terribly bad.  Nevertheless, we can do better.

The ``itertools`` module has a ``islice()`` (iterable slice)
 function.  It returns an iterator over a slice of a sequence --- so no
 more copies:

   from itertools import islice
   triplets = zip(words, islice(words, 1, None), islice(words, 2,
None))
   for triplet in triplets:
       print(triplet)
   ('this', 'that', 'the')
   ('that', 'the', 'other')
   ('the', 'other', 'and')
   ('other', 'and', 'one')
   ('and', 'one', 'more')

The Iterator Protocol

The main thing that differentiates an iterator from an iterable (sequence) is that an iterator saves state.  An iterable must have the following methods:

    an_iterator.__iter__()
Usually returns the iterator object itself.
::

    an_iterator.__next__() Returns the next item from the container. If there are no further

items it raises the StopIteration exception.

Making an Iterator

A simple version of range()

class IterateMe_1:
    def __init__(self, stop=5):
        self.current = 0
        self.stop = stop
    def __iter__(self):
        return self
    def __next__(self):
        if self.current < self.stop:
            self.current += 1
            return self.current
        else:
            raise StopIteration

What does for do?

Now that we know the iterator protocol, we can write something like a for loop:

:download:`my_for.py <../examples/iterators_generators/my_for.py>`

def my_for(an_iterable, func):
    """
    Emulation of a for loop.
    func() will be called with each item in an_iterable
    """
    # equiv of "for i in l:"
    iterator = iter(an_iterable)
    while True:
        try:
            i = next(iterator)
        except StopIteration:
            break
        func(i)

Summary

Iterators and Iterables are fundamental concepts in Python. Although the language can be confusing, the underlying concepts are quite straightforward. In the lesson assignment you will have opportunities to practice and apply using them.