Solution 1:

OK looks like there are too many mismatch in your postfix, roundcube and dovecot configuration. Let's break it down one by one.

STARTTLS error

This log line

warning: TLS library problem: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:s3_pkt.c:1293:SSL alert number 

indicated that PHP fails to verify peer certificate because unknown CA. This is exact duplicate problem with this question: Roundcube & Postfix SMTP: SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:s3_pkt.c. And from the question you already solves it by add these lines in roundcube

$config['smtp_conn_options'] = array(
  'ssl'         => array(
     'verify_peer'      => false,
     'verify_peer_name' => false,
  ),
);

Authentication failed error

To do this, we must look in postfix and roundcube logs to identify the problem. The real message was captured in postfix log file

Jan 18 20:18:21 steelhorse postfix/smtpd[1942]: warning: localhost.localdomain[127.0.0.1]: SASL LOGIN authentication failed: Invalid authentication mechanism

Now because the postfix SASL mechanism was provided by dovecot then you should check the mechanism that dovecot offer by run dovecot -a | grep auth_mechanism.

By looking to roundcube $config['smtp_auth_type'] and dovecot auth_mechanisms config, you can see a mismatch between these two. Dovecot only offer PLAIN mechanism. But the roundcube was configure to use LOGIN mechanism.

Solution can be one or both of these

  • Offering LOGIN mechanism from dovecot side by adding

    auth_mechanisms = plain login
    

    in dovecot config.

  • Set PLAIN login to roundcube side by change the parameter to

    $config['smtp_auth_type'] = 'PLAIN';