python numpy ValueError: operands could not be broadcast together with shapes
In numpy, I have two "arrays", X
is (m,n)
and y
is a vector (n,1)
using
X*y
I am getting the error
ValueError: operands could not be broadcast together with shapes (97,2) (2,1)
When (97,2)x(2,1)
is clearly a legal matrix operation and should give me a (97,1)
vector
EDIT:
I have corrected this using X.dot(y)
but the original question still remains.
dot
is matrix multiplication, but *
does something else.
We have two arrays:
-
X
, shape (97,2) -
y
, shape (2,1)
With Numpy arrays, the operation
X * y
is done element-wise, but one or both of the values can be expanded in one or more dimensions to make them compatible. This operation is called broadcasting. Dimensions, where size is 1 or which are missing, can be used in broadcasting.
In the example above the dimensions are incompatible, because:
97 2
2 1
Here there are conflicting numbers in the first dimension (97 and 2). That is what the ValueError above is complaining about. The second dimension would be ok, as number 1 does not conflict with anything.
For more information on broadcasting rules: http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html
(Please note that if X
and y
are of type numpy.matrix
, then asterisk can be used as matrix multiplication. My recommendation is to keep away from numpy.matrix
, it tends to complicate more than simplifying things.)
Your arrays should be fine with numpy.dot
; if you get an error on numpy.dot
, you must have some other bug. If the shapes are wrong for numpy.dot
, you get a different exception:
ValueError: matrices are not aligned
If you still get this error, please post a minimal example of the problem. An example multiplication with arrays shaped like yours succeeds:
In [1]: import numpy
In [2]: numpy.dot(numpy.ones([97, 2]), numpy.ones([2, 1])).shape
Out[2]: (97, 1)
Per numpy docs:
When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when:
- they are equal, or
- one of them is 1
In other words, if you are trying to multiply two matrices (in the linear algebra sense) then you want X.dot(y)
but if you are trying to broadcast scalars from matrix y
onto X
then you need to perform X * y.T
.
Example:
>>> import numpy as np
>>>
>>> X = np.arange(8).reshape(4, 2)
>>> y = np.arange(2).reshape(1, 2) # create a 1x2 matrix
>>> X * y
array([[0,1],
[0,3],
[0,5],
[0,7]])
You are looking for np.matmul(X, y)
. In Python 3.5+ you can use X @ y
.
It's possible that the error didn't occur in the dot product, but after. For example try this
a = np.random.randn(12,1)
b = np.random.randn(1,5)
c = np.random.randn(5,12)
d = np.dot(a,b) * c
np.dot(a,b)
will be fine; however np.dot(a, b) * c
is clearly wrong (12x1 X 1x5 = 12x5
which cannot element-wise multiply 5x12
) but numpy will give you
ValueError: operands could not be broadcast together with shapes (12,1) (1,5)
The error is misleading; however there is an issue on that line.