OpenSSH : Key-based authorization, maximum key length

I'am using Putty on windows with key-based authentication to access some of mine servers.

It works totally fine with ~3700-bit key, but with ~17000-bit key it thinks for like 20 seconds on client-side and then just says "Access denied" and asks for a password.

Is there any key length limit or timeout in OpenSSH for key-based authentication?

I understand that using such large keys have not much practical sence, especially when looking at these 20 seconds of calculation, just trying to solve any problems I face :-)...

At one point I looked into the source of OpenSSL for Diffie-Hellman keys, and found there was an "arbitrary" 10K limit on the size of DH keys. I changed the source for a test, and found that worked. I wrote a bug to the authors, and they replied back that it was design intent to prevent DoS by using massive keys.

Wouldn't surprise me to see something similar in OpenSSH.

There is no maximum key size or timeout defined in the protocol (or at least none that you'd be hitting), but an implementation might not support such long keys. A 20-second processing time with the private key doesn't sound high for a 17kbit RSA key. Then the server might not want to spend too much computing power on an unauthenticated user: refusing very large keys is a protection against DoS attacks.

Currently 2048 bits is considered reasonable for an RSA key; 4096 bits is higher than necessary but usually supported; beyond this you shouldn't be surprised if some programs to reject the key.