How do Python's any and all functions work?
I'm trying to understand how the any() and all() Python built-in functions work.
I'm trying to compare the tuples so that if any value is different then it will return True and if they are all the same it will return False. How are they working in this case to return [False, False, False]?
d is a defaultdict(list).
print d['Drd2']
# [[1, 5, 0], [1, 6, 0]]
print list(zip(*d['Drd2']))
# [(1, 1), (5, 6), (0, 0)]
print [any(x) and not all(x) for x in zip(*d['Drd2'])]
# [False, False, False]
To my knowledge, this should output
# [False, True, False]
since (1,1) are the same, (5,6) are different, and (0,0) are the same.
Why is it evaluating to False for all tuples?
You can roughly think of any and all as series of logical or and and operators, respectively.
any
any will return True when at least one of the elements is Truthy. Read about Truth Value Testing.
all
all will return True only when all the elements are Truthy.
Truth table
+-----------------------------------------+---------+---------+
| | any | all |
+-----------------------------------------+---------+---------+
| All Truthy values | True | True |
+-----------------------------------------+---------+---------+
| All Falsy values | False | False |
+-----------------------------------------+---------+---------+
| One Truthy value (all others are Falsy) | True | False |
+-----------------------------------------+---------+---------+
| One Falsy value (all others are Truthy) | True | False |
+-----------------------------------------+---------+---------+
| Empty Iterable | False | True |
+-----------------------------------------+---------+---------+
Note 1: The empty iterable case is explained in the official documentation, like this
any
Return
Trueif any element of the iterable is true. If the iterable is empty, returnFalse
Since none of the elements are true, it returns False in this case.
all
Return
Trueif all elements of the iterable are true (or if the iterable is empty).
Since none of the elements are false, it returns True in this case.
Note 2:
Another important thing to know about any and all is, it will short-circuit the execution, the moment they know the result. The advantage is, entire iterable need not be consumed. For example,
>>> multiples_of_6 = (not (i % 6) for i in range(1, 10))
>>> any(multiples_of_6)
True
>>> list(multiples_of_6)
[False, False, False]
Here, (not (i % 6) for i in range(1, 10)) is a generator expression which returns True if the current number within 1 and 9 is a multiple of 6. any iterates the multiples_of_6 and when it meets 6, it finds a Truthy value, so it immediately returns True, and rest of the multiples_of_6 is not iterated. That is what we see when we print list(multiples_of_6), the result of 7, 8 and 9.
This excellent thing is used very cleverly in this answer.
With this basic understanding, if we look at your code, you do
any(x) and not all(x)
which makes sure that, atleast one of the values is Truthy but not all of them. That is why it is returning [False, False, False]. If you really wanted to check if both the numbers are not the same,
print [x[0] != x[1] for x in zip(*d['Drd2'])]
How do Python's
anyandallfunctions work?
any and all take iterables and return True if any and all (respectively) of the elements are True.
>>> any([0, 0.0, False, (), '0']), all([1, 0.0001, True, (False,)])
(True, True) # ^^^-- truthy non-empty string
>>> any([0, 0.0, False, (), '']), all([1, 0.0001, True, (False,), {}])
(False, False) # ^^-- falsey
If the iterables are empty, any returns False, and all returns True.
>>> any([]), all([])
(False, True)
I was demonstrating all and any for students in class today. They were mostly confused about the return values for empty iterables. Explaining it this way caused a lot of lightbulbs to turn on.
Shortcutting behavior
They, any and all, both look for a condition that allows them to stop evaluating. The first examples I gave required them to evaluate the boolean for each element in the entire list.
(Note that list literal is not itself lazily evaluated - you could get that with an Iterator - but this is just for illustrative purposes.)
Here's a Python implementation of any and all:
def any(iterable):
for i in iterable:
if i:
return True
return False # for an empty iterable, any returns False!
def all(iterable):
for i in iterable:
if not i:
return False
return True # for an empty iterable, all returns True!
Of course, the real implementations are written in C and are much more performant, but you could substitute the above and get the same results for the code in this (or any other) answer.
all
all checks for elements to be False (so it can return False), then it returns True if none of them were False.
>>> all([1, 2, 3, 4]) # has to test to the end!
True
>>> all([0, 1, 2, 3, 4]) # 0 is False in a boolean context!
False # ^--stops here!
>>> all([])
True # gets to end, so True!
any
The way any works is that it checks for elements to be True (so it can return True), then it returnsFalseif none of them wereTrue`.
>>> any([0, 0.0, '', (), [], {}]) # has to test to the end!
False
>>> any([1, 0, 0.0, '', (), [], {}]) # 1 is True in a boolean context!
True # ^--stops here!
>>> any([])
False # gets to end, so False!
I think if you keep in mind the short-cutting behavior, you will intuitively understand how they work without having to reference a Truth Table.
Evidence of all and any shortcutting:
First, create a noisy_iterator:
def noisy_iterator(iterable):
for i in iterable:
print('yielding ' + repr(i))
yield i
and now let's just iterate over the lists noisily, using our examples:
>>> all(noisy_iterator([1, 2, 3, 4]))
yielding 1
yielding 2
yielding 3
yielding 4
True
>>> all(noisy_iterator([0, 1, 2, 3, 4]))
yielding 0
False
We can see all stops on the first False boolean check.
And any stops on the first True boolean check:
>>> any(noisy_iterator([0, 0.0, '', (), [], {}]))
yielding 0
yielding 0.0
yielding ''
yielding ()
yielding []
yielding {}
False
>>> any(noisy_iterator([1, 0, 0.0, '', (), [], {}]))
yielding 1
True
The source
Let's look at the source to confirm the above.
Here's the source for any:
static PyObject *
builtin_any(PyObject *module, PyObject *iterable)
{
PyObject *it, *item;
PyObject *(*iternext)(PyObject *);
int cmp;
it = PyObject_GetIter(iterable);
if (it == NULL)
return NULL;
iternext = *Py_TYPE(it)->tp_iternext;
for (;;) {
item = iternext(it);
if (item == NULL)
break;
cmp = PyObject_IsTrue(item);
Py_DECREF(item);
if (cmp < 0) {
Py_DECREF(it);
return NULL;
}
if (cmp > 0) {
Py_DECREF(it);
Py_RETURN_TRUE;
}
}
Py_DECREF(it);
if (PyErr_Occurred()) {
if (PyErr_ExceptionMatches(PyExc_StopIteration))
PyErr_Clear();
else
return NULL;
}
Py_RETURN_FALSE;
}
And here's the source for all:
static PyObject *
builtin_all(PyObject *module, PyObject *iterable)
{
PyObject *it, *item;
PyObject *(*iternext)(PyObject *);
int cmp;
it = PyObject_GetIter(iterable);
if (it == NULL)
return NULL;
iternext = *Py_TYPE(it)->tp_iternext;
for (;;) {
item = iternext(it);
if (item == NULL)
break;
cmp = PyObject_IsTrue(item);
Py_DECREF(item);
if (cmp < 0) {
Py_DECREF(it);
return NULL;
}
if (cmp == 0) {
Py_DECREF(it);
Py_RETURN_FALSE;
}
}
Py_DECREF(it);
if (PyErr_Occurred()) {
if (PyErr_ExceptionMatches(PyExc_StopIteration))
PyErr_Clear();
else
return NULL;
}
Py_RETURN_TRUE;
}
I know this is old, but I thought it might be helpful to show what these functions look like in code. This really illustrates the logic, better than text or a table IMO. In reality they are implemented in C rather than pure Python, but these are equivalent.
def any(iterable):
for item in iterable:
if item:
return True
return False
def all(iterable):
for item in iterable:
if not item:
return False
return True
In particular, you can see that the result for empty iterables is just the natural result, not a special case. You can also see the short-circuiting behaviour; it would actually be more work for there not to be short-circuiting.
When Guido van Rossum (the creator of Python) first proposed adding any() and all(), he explained them by just posting exactly the above snippets of code.
The code in question you're asking about comes from my answer given here. It was intended to solve the problem of comparing multiple bit arrays - i.e. collections of 1 and 0.
any and all are useful when you can rely on the "truthiness" of values - i.e. their value in a boolean context. 1 is True and 0 is False, a convenience which that answer leveraged. 5 happens to also be True, so when you mix that into your possible inputs... well. Doesn't work.
You could instead do something like this:
[len(set(x)) > 1 for x in zip(*d['Drd2'])]
It lacks the aesthetics of the previous answer (I really liked the look of any(x) and not all(x)), but it gets the job done.