how to store a complex object in redis (using redis-py)

The hmset function can set the value of each field, but I found that if the value itself is a complex structured object, the value return from hget is a serialized string, not the original object

e.g

images= [{'type':'big', 'url':'....'},
     {'type':'big', 'url':'....'},
     {'type':'big', 'url':'....'}]   

redis = Redis()
redis.hset('photo:1', 'images', images)

i = redis.hget('photo:1', 'images')
print type(i)

the type of i is a string, not a python object, is there any way to solve this problem besides manually parse each fields?

Solution 1:

Actually, you can store python objects in redis using the built-in module pickle.

Here is example.

import pickle
import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)
obj = ExampleObject()
pickled_object = pickle.dumps(obj)
r.set('some_key', pickled_object)
unpacked_object = pickle.loads(r.get('some_key'))
obj == unpacked_object

Solution 2:

If your data is JSON-serializable, then that may be the better option than saving python pickles to an external database, since it's a more common standard outside of Python, is more human-readable on its own, and avoids a rather large attack vector.

JSON Example:

import json
import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)

images= [
    {'type':'big', 'url':'....'},
    {'type':'big', 'url':'....'},
    {'type':'big', 'url':'....'},
]

# Convert python dict to JSON str and save to Redis
json_images = json.dumps(images)
r.set('images', json_images)

# Read saved JSON str from Redis and unpack into python dict
unpacked_images = json.loads(r.get('images'))
images == unpacked_images

python 3:

unpacked_images = json.loads(r.get('images').decode('utf-8'))
images == unpacked_images

Solution 3:

You can't create nested structures in Redis, meaning you can't (for example) store a native redis list inside a native redis hash-map.

If you really need nested structures, you might want to just store a JSON-blob (or something similar) instead. Another option is to store an "id"/key to a different redis object as the value of the map key, but that requires multiple calls to the server to get the full object.

Solution 4:

I created a library, SubRedis, which lets you create much more complex structures/hierarchies in redis. If you give it a redis instance and a prefix, it gives you a nearly fully capable and independent redis instance.

redis = Redis()
photoRedis = SubRedis("photo:%s" % photoId, redis)
photoRedis.hmset('image0', images[0])
photoRedis.hmset('image1', images[1])
...

SubRedis just ends up prepending the string passed into it as a prefix onto the flat redis data structure. I find this to be a convenient wrapper for a pattern I end up doing a lot in redis -- prepending some id to nest some data.