Error: "invalid use of incomplete type ‘RSA {aka struct rsa_st}" in OpenSSL 1.1.0

crypto.cpp:158:13: error: invalid use of incomplete type ‘RSA {aka struct rsa_st}’
     if( (prv->p==0 || prv->q==0) ) {
             ^~

As you are aware, OpenSSL 1.1.0 changed the visibility of a lot of struct members. You can no longer access the members directly. Instead, you have to use getter and setter functions.

Try RSA_get0_factors. The get0 means the reference counts are not incremented. Do not BN_free them.

void RSA_get0_factors(const RSA *r, const BIGNUM **p, const BIGNUM **q);

If the code supports multiple versions of OpenSSL, then you will need a guard because RSA_get0_factors is for OpenSSL 1.1.0 and above. Maybe something like the following. Also see OPENSSL_VERSION_NUMBER man page.

#include <openssl/opensslv.h>

#if OPENSSL_VERSION_NUMBER < 0x10100000L

    /* OpenSSL 1.0.2 and below (old code) */

#else

    /* OpenSSL 1.1.0 and above (new code) */

#endif

After 1.1.1 OpenSSL Supports getter what return each parameter like this.

const BIGNUM *RSA_get0_n(const RSA *d);
const BIGNUM *RSA_get0_e(const RSA *d);
const BIGNUM *RSA_get0_d(const RSA *d);
const BIGNUM *RSA_get0_p(const RSA *d);
const BIGNUM *RSA_get0_q(const RSA *d);
const BIGNUM *RSA_get0_dmp1(const RSA *r);
const BIGNUM *RSA_get0_dmq1(const RSA *r);
const BIGNUM *RSA_get0_iqmp(const RSA *r);

So if you don't need to considerate below version of OpenSSL less than 1.1.1 these code will make simple code. AND other structures support kind of this getter too. You can find more information of functions this. https://www.openssl.org/docs/man1.1.1/man3/

#if OPENSSL_VERSION_NUMBER < 0x10100005L
static void RSA_get0_key(const RSA *r, const BIGNUM **n, const BIGNUM **e, const BIGNUM **d, const BIGNUM **p, const BIGNUM **q)
{
    if(n != NULL)
        *n = r->n;

    if(e != NULL)
        *e = r->e;

    if(d != NULL)
        *d = r->d;

    if(p != NULL)
        *p = r->p;

    if(q != NULL)
        *q = r->q;
}
#endif

const BIGNUM *bn_p;
const BIGNUM *bn_q;

RSA_get0_key(key, NULL, NULL, NULL, &bn_p, &bn_q);
/*   if( (prv->p==0 || prv->q==0) ) { */
if( (prv_p==0 || prv_q==0) ) {