What exactly are iterator, iterable, and iteration?
What is the most basic definition of "iterable", "iterator" and "iteration" in Python?
I have read multiple definitions but I am unable to identify the exact meaning as it still won't sink in.
Can someone please help me with the 3 definitions in layman terms?
Iteration is a general term for taking each item of something, one after another. Any time you use a loop, explicit or implicit, to go over a group of items, that is iteration.
In Python, iterable and iterator have specific meanings.
An iterable is an object that has an __iter__
method which returns an iterator, or which defines a __getitem__
method that can take sequential indexes starting from zero (and raises an IndexError
when the indexes are no longer valid). So an iterable is an object that you can get an iterator from.
An iterator is an object with a next
(Python 2) or __next__
(Python 3) method.
Whenever you use a for
loop, or map
, or a list comprehension, etc. in Python, the next
method is called automatically to get each item from the iterator, thus going through the process of iteration.
A good place to start learning would be the iterators section of the tutorial and the iterator types section of the standard types page. After you understand the basics, try the iterators section of the Functional Programming HOWTO.
Here's the explanation I use in teaching Python classes:
An ITERABLE is:
- anything that can be looped over (i.e. you can loop over a string or file) or
- anything that can appear on the right-side of a for-loop:
for x in iterable: ...
or - anything you can call with
iter()
that will return an ITERATOR:iter(obj)
or - an object that defines
__iter__
that returns a fresh ITERATOR, or it may have a__getitem__
method suitable for indexed lookup.
An ITERATOR is an object:
- with state that remembers where it is during iteration,
- with a
__next__
method that:- returns the next value in the iteration
- updates the state to point at the next value
- signals when it is done by raising
StopIteration
- and that is self-iterable (meaning that it has an
__iter__
method that returnsself
).
Notes:
- The
__next__
method in Python 3 is speltnext
in Python 2, and - The builtin function
next()
calls that method on the object passed to it.
For example:
>>> s = 'cat' # s is an ITERABLE
# s is a str object that is immutable
# s has no state
# s has a __getitem__() method
>>> t = iter(s) # t is an ITERATOR
# t has state (it starts by pointing at the "c"
# t has a next() method and an __iter__() method
>>> next(t) # the next() function returns the next value and advances the state
'c'
>>> next(t) # the next() function returns the next value and advances
'a'
>>> next(t) # the next() function returns the next value and advances
't'
>>> next(t) # next() raises StopIteration to signal that iteration is complete
Traceback (most recent call last):
...
StopIteration
>>> iter(t) is t # the iterator is self-iterable
The above answers are great, but as most of what I've seen, don't stress the distinction enough for people like me.
Also, people tend to get "too Pythonic" by putting definitions like "X is an object that has __foo__()
method" before. Such definitions are correct--they are based on duck-typing philosophy, but the focus on methods tends to get between when trying to understand the concept in its simplicity.
So I add my version.
In natural language,
- iteration is the process of taking one element at a time in a row of elements.
In Python,
iterable is an object that is, well, iterable, which simply put, means that it can be used in iteration, e.g. with a
for
loop. How? By using iterator. I'll explain below.... while iterator is an object that defines how to actually do the iteration--specifically what is the next element. That's why it must have
next()
method.
Iterators are themselves also iterable, with the distinction that their __iter__()
method returns the same object (self
), regardless of whether or not its items have been consumed by previous calls to next()
.
So what does Python interpreter think when it sees for x in obj:
statement?
Look, a
for
loop. Looks like a job for an iterator... Let's get one. ... There's thisobj
guy, so let's ask him."Mr.
obj
, do you have your iterator?" (... callsiter(obj)
, which callsobj.__iter__()
, which happily hands out a shiny new iterator_i
.)OK, that was easy... Let's start iterating then. (
x = _i.next()
...x = _i.next()
...)
Since Mr. obj
succeeded in this test (by having certain method returning a valid iterator), we reward him with adjective: you can now call him "iterable Mr. obj
".
However, in simple cases, you don't normally benefit from having iterator and iterable separately. So you define only one object, which is also its own iterator. (Python does not really care that _i
handed out by obj
wasn't all that shiny, but just the obj
itself.)
This is why in most examples I've seen (and what had been confusing me over and over), you can see:
class IterableExample(object):
def __iter__(self):
return self
def next(self):
pass
instead of
class Iterator(object):
def next(self):
pass
class Iterable(object):
def __iter__(self):
return Iterator()
There are cases, though, when you can benefit from having iterator separated from the iterable, such as when you want to have one row of items, but more "cursors". For example when you want to work with "current" and "forthcoming" elements, you can have separate iterators for both. Or multiple threads pulling from a huge list: each can have its own iterator to traverse over all items. See @Raymond's and @glglgl's answers above.
Imagine what you could do:
class SmartIterableExample(object):
def create_iterator(self):
# An amazingly powerful yet simple way to create arbitrary
# iterator, utilizing object state (or not, if you are fan
# of functional), magic and nuclear waste--no kittens hurt.
pass # don't forget to add the next() method
def __iter__(self):
return self.create_iterator()
Notes:
I'll repeat again: iterator is not iterable. Iterator cannot be used as a "source" in
for
loop. Whatfor
loop primarily needs is__iter__()
(that returns something withnext()
).Of course,
for
is not the only iteration loop, so above applies to some other constructs as well (while
...).Iterator's
next()
can throw StopIteration to stop iteration. Does not have to, though, it can iterate forever or use other means.In the above "thought process",
_i
does not really exist. I've made up that name.There's a small change in Python 3.x:
next()
method (not the built-in) now must be called__next__()
. Yes, it should have been like that all along.You can also think of it like this: iterable has the data, iterator pulls the next item
Disclaimer: I'm not a developer of any Python interpreter, so I don't really know what the interpreter "thinks". The musings above are solely demonstration of how I understand the topic from other explanations, experiments and real-life experience of a Python newbie.
An iterable is a object which has a __iter__()
method. It can possibly iterated over several times, such as list()
s and tuple()
s.
An iterator is the object which iterates. It is returned by an __iter__()
method, returns itself via its own __iter__()
method and has a next()
method (__next__()
in 3.x).
Iteration is the process of calling this next()
resp. __next__()
until it raises StopIteration
.
Example:
>>> a = [1, 2, 3] # iterable
>>> b1 = iter(a) # iterator 1
>>> b2 = iter(a) # iterator 2, independent of b1
>>> next(b1)
1
>>> next(b1)
2
>>> next(b2) # start over, as it is the first call to b2
1
>>> next(b1)
3
>>> next(b1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>> b1 = iter(a) # new one, start over
>>> next(b1)
1
Here's my cheat sheet:
sequence
+
|
v
def __getitem__(self, index: int):
+ ...
| raise IndexError
|
|
| def __iter__(self):
| + ...
| | return <iterator>
| |
| |
+--> or <-----+ def __next__(self):
+ | + ...
| | | raise StopIteration
v | |
iterable | |
+ | |
| | v
| +----> and +-------> iterator
| ^
v |
iter(<iterable>) +----------------------+
|
def generator(): |
+ yield 1 |
| generator_expression +-+
| |
+-> generator() +-> generator_iterator +-+
Quiz: Do you see how...
- every iterator is an iterable?
- a container object's
__iter__()
method can be implemented as a generator? - an iterable that has a
__next__
method is not necessarily an iterator?
Answers:
- Every iterator must have an
__iter__
method. Having__iter__
is enough to be an iterable. Therefore every iterator is an iterable. -
When
__iter__
is called it should return an iterator (return <iterator>
in the diagram above). Calling a generator returns a generator iterator which is a type of iterator.class Iterable1: def __iter__(self): # a method (which is a function defined inside a class body) # calling iter() converts iterable (tuple) to iterator return iter((1,2,3)) class Iterable2: def __iter__(self): # a generator for i in (1, 2, 3): yield i class Iterable3: def __iter__(self): # with PEP 380 syntax yield from (1, 2, 3) # passes assert list(Iterable1()) == list(Iterable2()) == list(Iterable3()) == [1, 2, 3]
-
Here is an example:
class MyIterable: def __init__(self): self.n = 0 def __getitem__(self, index: int): return (1, 2, 3)[index] def __next__(self): n = self.n = self.n + 1 if n > 3: raise StopIteration return n # if you can iter it without raising a TypeError, then it's an iterable. iter(MyIterable()) # but obviously `MyIterable()` is not an iterator since it does not have # an `__iter__` method. from collections.abc import Iterator assert isinstance(MyIterable(), Iterator) # AssertionError