Regular expression to enforce complex passwords, matching 3 out of 4 rules

Don't use one regex to check it then.

if (password.length < 8)
  alert("bad password");
var hasUpperCase = /[A-Z]/.test(password);
var hasLowerCase = /[a-z]/.test(password);
var hasNumbers = /\d/.test(password);
var hasNonalphas = /\W/.test(password);
if (hasUpperCase + hasLowerCase + hasNumbers + hasNonalphas < 3)
  alert("bad password");

If you must use a single regex:

^(?:(?=.*[a-z])(?:(?=.*[A-Z])(?=.*[\d\W])|(?=.*\W)(?=.*\d))|(?=.*\W)(?=.*[A-Z])(?=.*\d)).{8,}$

This regex is not optimized for efficiency. It is constructed by A·B·C + A·B·D + A·C·D + B·C·D with some factorization. Breakdown:

^
(?:
    (?=.*[a-z])       # 1. there is a lower-case letter ahead,
    (?:               #    and
        (?=.*[A-Z])   #     1.a.i) there is also an upper-case letter, and
        (?=.*[\d\W])  #     1.a.ii) a number (\d) or symbol (\W),
    |                 #    or
        (?=.*\W)      #     1.b.i) there is a symbol, and
        (?=.*\d)      #     1.b.ii) a number ahead
    )
|                     # OR
    (?=.*\W)          # 2.a) there is a symbol, and
    (?=.*[A-Z])       # 2.b) an upper-case letter, and
    (?=.*\d)          # 2.c) a number ahead.
)
.{8,}                 # the password must be at least 8 characters long.
$

You could write a really sophisticated regex to do that. Instead, I’d suggest writing four distinct regexes, one for each rule, and testing them one by one, counting how many of them matched. If three out of four did, accept the password.