Allow users to install certificates in their home directory?

If you mean applications using OpenSSL library for SSL, each application can either specify the (concatenated) file and/or (hash-linked) directory to be used for trusted certs, or it can invoke OpenSSL's defaults, or it could offer the choice. In the first case, you need to (be able and) configure the app what to specify. For example, in curl use --cacert and/or --capath per http://curl.haxx.se/docs/manpage.html . In the second case, the compiled-in OpenSSL defaults, which are system and possibly build dependent, can be overridden by environment variables SSL_CERT_FILE and SSL_CERT_DIR respectively.

If you mean applications using OpenSSL library for other things (that use certs) like CMS/SMIME, OpenSSL has a less simple API; basically the application must directly build up an X509_STORE to be used for validation, although I think it can still invoke the same defaults.

If you mean the commandline program openssl the picture is a little more complicated. Some utilities (subcommands) don't use truststore (or even certs at all); those that do have options to specify one usually -CAfile and -CApath; see the man pages for s_client, verify, ocsp etc as applicable. However, the logic that is supposed to use the defaults if you don't specify the options has long been coded inconsistently; there was discussion on the support list a few months ago and I believe a fix has (finally) been agreed, but as of 1.0.1j 15 Oct 2014 it isn't released.