certbort commands return ModuleNotFoundError: No module named '_cffi_backend'

I followed a guide to get my python flask app running and I am at the last step where I change http into https with certbot. But when I run my certbot command sudo certbot --nginx -d domainname -d www.domainname I get ModuleNotFoundError: No module named '_cffi_backend'

The whole error is:

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/home/mc-obfuscator/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/mc-obfuscator/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2859, in load_entry_point
    return ep.load()
  File "/home/mc-obfuscator/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2450, in load
    return self.resolve()
  File "/home/mc-obfuscator/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 10, in <module>
    import josepy as jose
  File "/usr/lib/python3/dist-packages/josepy/__init__.py", line 44, in <module>
    from josepy.interfaces import JSONDeSerializable
  File "/usr/lib/python3/dist-packages/josepy/interfaces.py", line 8, in <module>
    from josepy import errors, util
  File "/usr/lib/python3/dist-packages/josepy/util.py", line 4, in <module>
    import OpenSSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 12, in <module>
    from cryptography import x509
  File "/usr/lib/python3/dist-packages/cryptography/x509/__init__.py", line 8, in <module>
    from cryptography.x509.base import (
  File "/usr/lib/python3/dist-packages/cryptography/x509/base.py", line 16, in <module>
    from cryptography.x509.extensions import Extension, ExtensionType
  File "/usr/lib/python3/dist-packages/cryptography/x509/extensions.py", line 18, in <module>
    from cryptography.hazmat.primitives import constant_time, serialization
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/primitives/constant_time.py", line 9, in <module>
    from cryptography.hazmat.bindings._constant_time import lib
ModuleNotFoundError: No module named '_cffi_backend'

I hope someone can help as I have found a lot of people asking questions about this mysterious _cffi_backend thing.

Some more info:

If I do python3 -m pip install cffi it says requirement already satisfied.

I have also gotten this error when installing other things and trying different peoples solutions. 'ModuleNotFoundError: No module named 'apt_pkg' that seems to be fixed by doing sudo apt-get install python3-apt --reinstall but now I get: ImportError: cannot import name '_gi' from partially initialized module 'gi' (most likely due to a circular import) (/usr/lib/python3/dist-packages/gi/__init__.py)

I also made a symbolic link with /usr/lib/python3/dist-packages/apt_pkg.so -> apt_pkg.cpython-36m-x86_64-linux-gnu.so something other people said would work. I am running python 3.8 but probably have 3.6 on the server as well. If I do python it opens the 3.8 shell. Also I am running ubuntu 18.04.4.

ls -al /usr/bin | grep python gives :

-rwxr-xr-x  1 root   root        1056 Apr 16  2018 dh_python2
lrwxrwxrwx  1 root   root          23 Nov  7 10:07 pdb2.7 -> ../lib/python2.7/pdb.py
lrwxrwxrwx  1 root   root          23 Nov  7 10:44 pdb3.6 -> ../lib/python3.6/pdb.py
lrwxrwxrwx  1 root   root          23 Nov  7 10:50 pdb3.7 -> ../lib/python3.7/pdb.py
lrwxrwxrwx  1 root   root          23 Oct 28 16:14 pdb3.8 -> ../lib/python3.8/pdb.py
lrwxrwxrwx  1 root   root          31 Oct 25  2018 py3versions -> ../share/python3/py3versions.py
lrwxrwxrwx  1 root   root          24 Jun 19  2019 python -> /etc/alternatives/python
lrwxrwxrwx  1 root   root          16 Apr 16  2018 python-config -> python2.7-config
lrwxrwxrwx  1 root   root           9 Apr 16  2018 python2 -> python2.7
lrwxrwxrwx  1 root   root          16 Apr 16  2018 python2-config -> python2.7-config
-rwxr-xr-x  1 root   root     3637096 Nov  7 10:07 python2.7
lrwxrwxrwx  1 root   root          33 Nov  7 10:07 python2.7-config -> x86_64-linux-gnu-python2.7-config
lrwxrwxrwx  1 root   root          25 Jan  5 10:38 python3 -> /etc/alternatives/python3
-rwxr-xr-x  1 root   root         384 Feb  5  2018 python3-futurize
-rwxr-xr-x  1 root   root         388 Feb  5  2018 python3-pasteurize
-rwxr-xr-x  1 root   root         152 Nov 11  2017 python3-pbr
-rwxr-xr-x  2 root   root     4526456 Nov  7 10:44 python3.6
-rwxr-xr-x  2 root   root     4526456 Nov  7 10:44 python3.6m
-rwxr-xr-x  2 root   root     4873376 Nov  7 10:50 python3.7
-rwxr-xr-x  2 root   root     4873376 Nov  7 10:50 python3.7m
-rwxr-xr-x  1 root   root     5203488 Oct 28 16:14 python3.8
lrwxrwxrwx  1 root   root          10 Oct 25  2018 python3m -> python3.6m
lrwxrwxrwx  1 root   root          29 Apr 16  2018 pyversions -> ../share/python/pyversions.py
lrwxrwxrwx  1 root   root          10 Sep 27  2018 uwsgi_python36 -> uwsgi-core
lrwxrwxrwx  1 root   root          33 Apr 16  2018 x86_64-linux-gnu-python-config -> x86_64-linux-gnu-python2.7-config
-rwxr-xr-x  1 root   root        2971 Nov  7 10:07 x86_64-linux-gnu-python2.7-config

The files do exist on my system because:

dpkg -l python3-cffi-backend python3-cryptography
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                  Version                         Architecture                    Description
+++-=====================================================-===============================-===============================-===============================================================================================================
ii  python3-cffi-backend                                  1.11.5-1                        amd64                           Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-cryptography                                  2.1.4-1ubuntu1.3                amd64                           Python library exposing cryptographic recipes and primitives (Python 3)

This fixes the problem:

pip install -U cffi

On an Ubuntu 18.04 system there were multiple version of python3 installed, with the default python3 (3.9) being different to the (apt managed) system version (3.6). Using update-alternatives to set the default to the system version allowed certbot to run as expected. E.g.:

update-alternatives --config python3
There are 3 choices for the alternative python3 (providing /usr/bin/python3).

  Selection    Path                Priority   Status
------------------------------------------------------------
* 0            /usr/bin/python3.9   3         auto mode
  1            /usr/bin/python3.6   1         manual mode
  2            /usr/bin/python3.8   2         manual mode
  3            /usr/bin/python3.9   3         manual mode

Press <enter> to keep the current choice[*], or type selection number: 1

Then switch it back when done.

The solution ended up being to install sudo apt install python3-pip python3-dev build-essential libssl-dev libffi-dev python3-setuptools but not in any virtual environment and then to install cffl with pip3 also not in any virtual environment.