Python requests getting 403 but on mobile device works fine
When requesting an HTML page in Python, I get a 403 forbidden response every time.
import requests
url = 'https://shop.rewe.de/mydata/login'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
result = requests.get(url, headers=headers)
>>> result.status_code
403
When I use the iOs app, for example "Rest", I just send GET with the same domain and no other additional settings and get a 200 OK response. I don't know what the problem is when I run this in Python.
Edit: This shows the metric within the rest app. When searching for the protocol I found the h2 or simply http/2 protocol. I think this is the problem. Postman and also the requests lib do not support this.
I have changed the cipher of the request session. Temporarily it works. I do not know how long ...
url = 'https://shop.rewe.de/mydata/login'
import ssl
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
from requests.packages.urllib3.util import ssl_
CIPHERS = (
'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:AES256-SHA'
)
class TlsAdapter(HTTPAdapter):
def __init__(self, ssl_options=0, **kwargs):
self.ssl_options = ssl_options
super(TlsAdapter, self).__init__(**kwargs)
def init_poolmanager(self, *pool_args, **pool_kwargs):
ctx = ssl_.create_urllib3_context(ciphers=CIPHERS, cert_reqs=ssl.CERT_REQUIRED, options=self.ssl_options)
self.poolmanager = PoolManager(*pool_args,
ssl_context=ctx,
**pool_kwargs)
session = requests.session()
adapter = TlsAdapter(ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1)
session.mount("https://", adapter)
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0',
}
try:
r = session.request('GET', url, headers=headers)
print(r)
except Exception as exception:
print(exception)