Saving and loading multiple objects in pickle file?

Two additions to Tim Peters' accepted answer.

First, you need not store the number of items you pickled separately if you stop loading when you hit the end of the file:

def loadall(filename):
    with open(filename, "rb") as f:
        while True:
            try:
                yield pickle.load(f)
            except EOFError:
                break

items = loadall(myfilename)

This assumes the file contains only pickles; if there's anything else in there, the generator will try to treat whatever else is in there as pickles too, which could be dangerous.

Second, this way, you do not get a list but rather a generator. This will load only one item into memory at a time, which is useful if the dumped data is very large -- one possible reason why you may have wanted to pickle multiple items separately in the first place. You can still iterate over items with a for loop as if it were a list.

Using a list, tuple, or dict is by far the most common way to do this:

import pickle
PIK = "pickle.dat"

data = ["A", "b", "C", "d"]
with open(PIK, "wb") as f:
    pickle.dump(data, f)
with open(PIK, "rb") as f:
    print pickle.load(f)

That prints:

['A', 'b', 'C', 'd']

However, a pickle file can contain any number of pickles. Here's code producing the same output. But note that it's harder to write and to understand:

with open(PIK, "wb") as f:
    pickle.dump(len(data), f)
    for value in data:
        pickle.dump(value, f)
data2 = []
with open(PIK, "rb") as f:
    for _ in range(pickle.load(f)):
        data2.append(pickle.load(f))
print data2

If you do this, you're responsible for knowing how many pickles are in the file you write out. The code above does that by pickling the number of list objects first.

Try this:

import pickle

file = open('test.pkl','wb')
obj_1 = ['test_1', {'ability', 'mobility'}]
obj_2 = ['test_2', {'ability', 'mobility'}]
obj_3 = ['test_3', {'ability', 'mobility'}]

pickle.dump(obj_1, file)
pickle.dump(obj_2, file)
pickle.dump(obj_3, file)

file.close()

file = open('test.pkl', 'rb')
obj_1 = pickle.load(file)
obj_2 = pickle.load(file)
obj_3 = pickle.load(file)
print(obj_1)
print(obj_2)
print(obj_3)
file.close()

If you're dumping it iteratively, you'd have to read it iteratively as well.

You can run a loop (as the accepted answer shows) to keep unpickling rows until you reach the end-of-file (at which point an EOFError is raised).

data = []
with open("data.pickle", "rb") as f:
    while True:
        try:
            data.append(pickle.load(f))
        except EOFError:
            break

Minimal Verifiable Example

import pickle

# Dumping step
data = [{'a': 1}, {'b': 2}]
with open('test.pkl', 'wb') as f:
    for d in data:
        pickle.dump(d, f)

# Loading step
data2 = []
with open('test.pkl', 'rb') as f:
    while True:
        try:
            data2.append(pickle.load(f))
        except EOFError:
            break

data2
# [{'a': 1}, {'b': 2}]

data == data2
# True

Of course, this is under the assumption that your objects have to be pickled individually. You can also store your data as a single list of object, then use a single pickle/unpickle call (no need for loops).

data = [{'a':1}, {'b':2}]  # list of dicts as an example
with open('test.pkl', 'wb') as f:
    pickle.dump(data, f)

with open('test.pkl', 'rb') as f:
    data2 = pickle.load(f)

data2
# [{'a': 1}, {'b': 2}]

I will give an object-oriented demo using pickle to store and restore one or multi object:

class Worker(object):

    def __init__(self, name, addr):
        self.name = name
        self.addr = addr

    def __str__(self):
        string = u'[<Worker> name:%s addr:%s]' %(self.name, self.addr)
        return string

# output one item
with open('testfile.bin', 'wb') as f:
    w1 = Worker('tom1', 'China')
    pickle.dump(w1, f)

# input one item
with open('testfile.bin', 'rb') as f:
    w1_restore = pickle.load(f)
print 'item: %s' %w1_restore

# output multi items
with open('testfile.bin', 'wb') as f:
    w1 = Worker('tom2', 'China')
    w2 = Worker('tom3', 'China')
    pickle.dump([w1, w2], f)

# input multi items
with open('testfile.bin', 'rb') as f:
    w_list = pickle.load(f)

for w in w_list:
    print 'item-list: %s' %w

output:

item: [<Worker> name:tom1 addr:China]
item-list: [<Worker> name:tom2 addr:China]
item-list: [<Worker> name:tom3 addr:China]