How to speed up process of loading and reading JSON files in Python?

First, find out where your bottlenecks are.

If it is on the json decoding/encoding step, try switching to ultrajson:

UltraJSON is an ultra fast JSON encoder and decoder written in pure C with bindings for Python 2.5+ and 3.

The changes would be as simple as changing the import part:

try:
    import ujson as json
except ImportError:
    try:
        import simplejson as json
    except ImportError:
        import json

I've also done a simple benchmark at What is faster - Loading a pickled dictionary object or Loading a JSON file - to a dictionary?, take a look.

swith from

import json

import ujson

https://artem.krylysov.com/blog/2015/09/29/benchmark-python-json-libraries/

or switch to orjson

import orjson

https://github.com/ijl/orjson

Related