Is OpenLDAP a viable alternative to Windows LDAP?

Microsoft Active Directory is (mainly) just an LDAP server, some pre-cooked schemas and some tools for interacting with it and Kerberos. OpenLDAP is pretty much a drop-in replacement for the former.

While you could design your own AD compatible schemas (or if you already have an AD schema, just import it into openLDAP) and write your own tools for managing the datasets there are solutions runing on top of openLDAP available, e.g. GoSA, freeIPA see the samba wiki for more stuff

Not sure where Samba 4 has got to (its expected to add drop-in replacement functnioality for MSAD).

Absolutely. This is generally provided by pam_ldap. There is a specific schema that needs to be imported to the OpenLDAP directory to get the right attributes you need to use it as a Unix authentication source, but it works very reliably.