How can I set a limit for incorrect password attempts for GDM lock screen before action is taken?

Solution 1:

Some background info:

User log-ins are handled by the PAM system (Pluggable Authentication Module) and in this particular case by the pam_tally subsystem (to tally the user accounts).

As pam_tally itself will be deprecated, you should use pam_tally2, which comes in two parts:

1. pam_tally2.so being the module that does the authentication itself and which you need to add to the PAM system
2. pam_tally2 being a stand-alone program that you'll need to display, change and clear individual counts

The parameters of pam_tally2 you're interested in are:

deny=n
    Deny access if tally for this user exceeds n.
lock_time=n
    Always deny for n seconds after failed attempt.
unlock_time=n
    Allow access after n seconds after failed attempt. If this
    option is used the user will be locked out for the specified
    amount of time after he exceeded his maximum allowed attempts.
    Otherwise the account is locked until the lock is removed by a
    manual intervention of the system administrator.
magic_root
    If the module is invoked by a user with uid=0 the counter is
    not incremented. 
even_deny_root
    Root account can become unavailable.

The configuration:

Add the following lines to /etc/pam.d/login to lock the account after 3 failed logins (also for the root account) and have it unlocked automatically after 50 minutes.

auth     required       pam_securetty.so
auth     required       pam_tally2.so deny=3 even_deny_root lock_time=60 unlock_time=3000
auth     required       pam_env.so
auth     required       pam_unix.so
auth     required       pam_nologin.so
account  required       pam_unix.so
password required       pam_unix.so
session  required       pam_limits.so
session  required       pam_unix.so
session  required       pam_lastlog.so nowtmp
session  optional       pam_mail.so standard

After adding the above congiguration, test it and log out and do try to log in 3 times with a bogus password: 60 seconds to wait after first and second attempt and 50 minutes after the third try.