Google Cloud - Permission denied (publickey)

I have a GCP instance running debian-8-jessie-v20170619 (deprecated). After a reboot, I lost ssh access to the machine (cannot ssh via Web, Terminal, or gcloud cli). Now my MySQL database has issue for my Wordpress blog and I need to ssh. There are tens of questions with the same title, so I read most of them and tried:

1. Regenerating ssh key, adding to Metadata (both on instance, and in project)
2. Turning off OS Login and trying ssh. Turning on OS Login and trying ssh.
3. Serial console method (user is not being created by startup script; so login fails)
4. Using ssh -i key user@host instead of gcloud cli
5. Adding key to authorized_keys

None of these worked. Here is the out of $ gcloud compute ssh MyVM --ssh-flag="-vvv"

OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/Jash/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug2: resolve_canonicalize: hostname 35.237.214.57 is address
debug2: ssh_connect_direct
debug1: Connecting to 35.237.214.57 [35.237.214.57] port 22.
debug1: Connection established.
debug1: identity file /Users/Jash/.ssh/google_compute_engine type 0
debug1: identity file /Users/Jash/.ssh/google_compute_engine-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u8
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 35.237.214.57:22 as 'Jash'
debug1: using hostkeyalias: compute.2217424771252360577
debug3: hostkeys_foreach: reading file "/Users/Jash/.ssh/google_compute_known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/Jash/.ssh/google_compute_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from compute.2217424771252360577
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:/8U8cYg6EIhbKq5aDfb9DySyrP6JEli2B3iUpLKx6lE
debug1: using hostkeyalias: compute.2217424771252360577
debug3: hostkeys_foreach: reading file "/Users/Jash/.ssh/google_compute_known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/Jash/.ssh/google_compute_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from compute.2217424771252360577
debug1: Host 'compute.2217424771252360577' is known and matches the ECDSA host key.
debug1: Found key in /Users/Jash/.ssh/google_compute_known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/Jash/.ssh/google_compute_engine RSA SHA256:dHe8jHKOHpUXq/rriZMV3ya/HgaJeyrLqGLTe2+OJnc explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/Jash/.ssh/google_compute_engine RSA SHA256:dHe8jHKOHpUXq/rriZMV3ya/HgaJeyrLqGLTe2+OJnc explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Unfortunately you didn't provide serial port logs. I provided typical SSH troubleshooting steps below. Feel free to update your question to get more specific guidance.

Check serial port logs at Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM -> at the VM instance details find section Logs and click on Serial port 1 (console). It's better to reboot VM instance to get the full picture.

If you found errors/warning related to disk space you can try to resize it accordingly to the documentation Resizing a zonal persistent disk or follow this article Recovering an inaccessible instance or a full boot disk:

Try again to troubleshoot your VM instance via serial console (I've used this startup script many times and it works):

1. Enable serial console connection at Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> click on EDIT -> go to section Remote access and check Enable connecting to serial ports

2. Create temporary user and password to login: shutdown your VM and set a startup script by adding at the section Custom metadata key startup-script and value:

    #!/bin/bash
    useradd --groups google_sudoers tempuser
    echo "tempuser:password" | chpasswd
   

   and then start your VM.

3. Connect to your VM via serial port at Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> and click on Connect to serial console

   if no luck with connection again, you can check serial port logs again to see details about startup-script execution.

4. Check what went wrong.

5. Disable access via serial port at Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> click on EDIT -> go to section Remote access and uncheck Enable connecting to serial ports. More details you can find at the documentation Interacting with the serial console.

If you weren't able to connect via serial console, try follow the documentation Troubleshooting SSH section Inspect the VM instance without shutting it down and inspect the disk of your VM on another VM. Same way you can transfer your data to another working VM instance.