Python: fastest way to create a list of n lists

Solution 1:

The probably only way which is marginally faster than

d = [[] for x in xrange(n)]

is

from itertools import repeat
d = [[] for i in repeat(None, n)]

It does not have to create a new int object in every iteration and is about 15 % faster on my machine.

Edit: Using NumPy, you can avoid the Python loop using

d = numpy.empty((n, 0)).tolist()

but this is actually 2.5 times slower than the list comprehension.

Solution 2:

The list comprehensions actually are implemented more efficiently than explicit looping (see the dis output for example functions) and the map way has to invoke an ophaque callable object on every iteration, which incurs considerable overhead overhead.

Regardless, [[] for _dummy in xrange(n)] is the right way to do it and none of the tiny (if existent at all) speed differences between various other ways should matter. Unless of course you spend most of your time doing this - but in that case, you should work on your algorithms instead. How often do you create these lists?

Solution 3:

Here are two methods, one sweet and simple(and conceptual), the other more formal and can be extended in a variety of situations, after having read a dataset.

Method 1: Conceptual

X2=[]
X1=[1,2,3]
X2.append(X1)
X3=[4,5,6]
X2.append(X3)
X2 thus has [[1,2,3],[4,5,6]] ie a list of lists. 

Method 2 : Formal and extensible

Another elegant way to store a list as a list of lists of different numbers - which it reads from a file. (The file here has the dataset train) Train is a data-set with say 50 rows and 20 columns. ie. Train[0] gives me the 1st row of a csv file, train[1] gives me the 2nd row and so on. I am interested in separating the dataset with 50 rows as one list, except the column 0 , which is my explained variable here, so must be removed from the orignal train dataset, and then scaling up list after list- ie a list of a list. Here's the code that does that.

Note that I am reading from "1" in the inner loop since I am interested in explanatory variables only. And I re-initialize X1=[] in the other loop, else the X2.append([0:(len(train[0])-1)]) will rewrite X1 over and over again - besides it more memory efficient.

X2=[]
for j in range(0,len(train)):
    X1=[]
    for k in range(1,len(train[0])):
        txt2=train[j][k]
        X1.append(txt2)
    X2.append(X1[0:(len(train[0])-1)])