How to get JSON from webpage into Python script

Got the following code in one of my scripts:

#
# url is defined above.
#
jsonurl = urlopen(url)

#
# While trying to debug, I put this in:
#
print jsonurl

#
# Was hoping text would contain the actual json crap from the URL, but seems not...
#
text = json.loads(jsonurl)
print text

What I want to do is get the {{.....etc.....}} stuff that I see on the URL when I load it in Firefox into my script so I can parse a value out of it. I've Googled a ton but I haven't found a good answer as to how to actually get the {{...}} stuff from a URL ending in .json into an object in a Python script.

Solution 1:

Get data from the URL and then call json.loads e.g.

Python3 example:

import urllib.request, json 
with urllib.request.urlopen("http://maps.googleapis.com/maps/api/geocode/json?address=google") as url:
    data = json.loads(url.read().decode())
    print(data)

Python2 example:

import urllib, json
url = "http://maps.googleapis.com/maps/api/geocode/json?address=google"
response = urllib.urlopen(url)
data = json.loads(response.read())
print data

The output would result in something like this:

{
"results" : [
    {
    "address_components" : [
        {
            "long_name" : "Charleston and Huff",
            "short_name" : "Charleston and Huff",
            "types" : [ "establishment", "point_of_interest" ]
        },
        {
            "long_name" : "Mountain View",
            "short_name" : "Mountain View",
            "types" : [ "locality", "political" ]
        },
        {
...

Solution 2:

I'll take a guess that you actually want to get data from the URL:

jsonurl = urlopen(url)
text = json.loads(jsonurl.read()) # <-- read from it

Or, check out JSON decoder in the requests library.

import requests
r = requests.get('someurl')
print r.json() # if response type was set to JSON, then you'll automatically have a JSON response here...

Solution 3:

This gets a dictionary in JSON format from a webpage with Python 2.X and Python 3.X:

#!/usr/bin/env python

try:
    # For Python 3.0 and later
    from urllib.request import urlopen
except ImportError:
    # Fall back to Python 2's urllib2
    from urllib2 import urlopen

import json


def get_jsonparsed_data(url):
    """
    Receive the content of ``url``, parse it as JSON and return the object.

    Parameters
    ----------
    url : str

    Returns
    -------
    dict
    """
    response = urlopen(url)
    data = response.read().decode("utf-8")
    return json.loads(data)


url = ("http://maps.googleapis.com/maps/api/geocode/json?"
       "address=googleplex&sensor=false")
print(get_jsonparsed_data(url))

See also: Read and write example for JSON

Solution 4:

I have found this to be the easiest and most efficient way to get JSON from a webpage when using Python 3:

import json,urllib.request
data = urllib.request.urlopen("https://api.github.com/users?since=100").read()
output = json.loads(data)
print (output)

Solution 5:

you need import requests and use from json() method :

source = requests.get("url").json()
print(source)

Of course, this method also works:

import json,urllib.request
data = urllib.request.urlopen("url").read()
output = json.loads(data)
print (output)

json.loads will decode it into a Python object using this table, for example a JSON object will become a Python dict.