Add numpy array as column to Pandas data frame

I have a Pandas data frame object of shape (X,Y) that looks like this:

[[1, 2, 3],
[4, 5, 6],
[7, 8, 9]]

and a numpy sparse matrix (CSC) of shape (X,Z) that looks something like this

[[0, 1, 0],
[0, 0, 1],
[1, 0, 0]]

How can I add the content from the matrix to the data frame in a new named column such that the data frame will end up like this:

[[1, 2, 3, [0, 1, 0]],
[4, 5, 6, [0, 0, 1]],
[7, 8, 9, [1, 0, 0]]]

Notice the data frame now has shape (X, Y+1) and rows from the matrix are elements in the data frame.


import numpy as np
import pandas as pd
import scipy.sparse as sparse

df = pd.DataFrame(np.arange(1,10).reshape(3,3))
arr = sparse.coo_matrix(([1,1,1], ([0,1,2], [1,2,0])), shape=(3,3))
df['newcol'] = arr.toarray().tolist()
print(df)

yields

   0  1  2     newcol
0  1  2  3  [0, 1, 0]
1  4  5  6  [0, 0, 1]
2  7  8  9  [1, 0, 0]

Consider using a higher dimensional datastructure (a Panel), rather than storing an array in your column:

In [11]: p = pd.Panel({'df': df, 'csc': csc})

In [12]: p.df
Out[12]: 
   0  1  2
0  1  2  3
1  4  5  6
2  7  8  9

In [13]: p.csc
Out[13]: 
   0  1  2
0  0  1  0
1  0  0  1
2  1  0  0

Look at cross-sections etc, etc, etc.

In [14]: p.xs(0)
Out[14]: 
   csc  df
0    0   1
1    1   2
2    0   3

See the docs for more on Panels.