How can I learn more about AWS's RDS Aurora PostgreSQL 9.6.19 upgrade failure?

The pg_upgrade_internal log file will usually contain details on any failures/errors.

You can take a look on these logs using the command line:

aws rds describe-db-log-files --db-instance-identifier my-db-instance

Or via console, or RDS API.

For more information take a look on these links: Upgrading the PostgreSQL DB engine for Amazon RDS, Viewing and listing database log files