TypeError: ObjectId('') is not JSON serializable

My response back from MongoDB after querying an aggregated function on document using Python, It returns valid response and i can print it but can not return it.

Error:

TypeError: ObjectId('51948e86c25f4b1d1c0d303c') is not JSON serializable

Print:

{'result': [{'_id': ObjectId('51948e86c25f4b1d1c0d303c'), 'api_calls_with_key': 4, 'api_calls_per_day': 0.375, 'api_calls_total': 6, 'api_calls_without_key': 2}], 'ok': 1.0}

But When i try to return:

TypeError: ObjectId('51948e86c25f4b1d1c0d303c') is not JSON serializable

It is RESTfull call:

@appv1.route('/v1/analytics')
def get_api_analytics():
    # get handle to collections in MongoDB
    statistics = sldb.statistics

    objectid = ObjectId("51948e86c25f4b1d1c0d303c")

    analytics = statistics.aggregate([
    {'$match': {'owner': objectid}},
    {'$project': {'owner': "$owner",
    'api_calls_with_key': {'$cond': [{'$eq': ["$apikey", None]}, 0, 1]},
    'api_calls_without_key': {'$cond': [{'$ne': ["$apikey", None]}, 0, 1]}
    }},
    {'$group': {'_id': "$owner",
    'api_calls_with_key': {'$sum': "$api_calls_with_key"},
    'api_calls_without_key': {'$sum': "$api_calls_without_key"}
    }},
    {'$project': {'api_calls_with_key': "$api_calls_with_key",
    'api_calls_without_key': "$api_calls_without_key",
    'api_calls_total': {'$add': ["$api_calls_with_key", "$api_calls_without_key"]},
    'api_calls_per_day': {'$divide': [{'$add': ["$api_calls_with_key", "$api_calls_without_key"]}, {'$dayOfMonth': datetime.now()}]},
    }}
    ])


    print(analytics)

    return analytics

db is well connected and collection is there too and I got back valid expected result but when i try to return it gives me Json error. Any idea how to convert the response back into JSON. Thanks

Solution 1:

Pymongo provides json_util - you can use that one instead to handle BSON types

def parse_json(data):
    return json.loads(json_util.dumps(data))

Solution 2:

You should define you own JSONEncoder and using it:

import json
from bson import ObjectId

class JSONEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, ObjectId):
            return str(o)
        return json.JSONEncoder.default(self, o)

JSONEncoder().encode(analytics)

It's also possible to use it in the following way.

json.encode(analytics, cls=JSONEncoder)

Solution 3:

>>> from bson import Binary, Code
>>> from bson.json_util import dumps
>>> dumps([{'foo': [1, 2]},
...        {'bar': {'hello': 'world'}},
...        {'code': Code("function x() { return 1; }")},
...        {'bin': Binary("")}])
'[{"foo": [1, 2]}, {"bar": {"hello": "world"}}, {"code": {"$code": "function x() { return 1; }", "$scope": {}}}, {"bin": {"$binary": "AQIDBA==", "$type": "00"}}]'

Actual example from json_util.

Unlike Flask's jsonify, "dumps" will return a string, so it cannot be used as a 1:1 replacement of Flask's jsonify.

But this question shows that we can serialize using json_util.dumps(), convert back to dict using json.loads() and finally call Flask's jsonify on it.

Example (derived from previous question's answer):

from bson import json_util, ObjectId
import json

#Lets create some dummy document to prove it will work
page = {'foo': ObjectId(), 'bar': [ObjectId(), ObjectId()]}

#Dump loaded BSON to valid JSON string and reload it as dict
page_sanitized = json.loads(json_util.dumps(page))
return page_sanitized

This solution will convert ObjectId and others (ie Binary, Code, etc) to a string equivalent such as "$oid."

JSON output would look like this:

{
  "_id": {
    "$oid": "abc123"
  }
}

Solution 4:

Most users who receive the "not JSON serializable" error simply need to specify default=str when using json.dumps. For example:

json.dumps(my_obj, default=str)

This will force a conversion to str, preventing the error. Of course then look at the generated output to confirm that it is what you need.