How do I unlock a greater variety of relics and arcana?

Basically, suck it up and buy stuff you don't want.

It turns out that both relics and arcana are organized into "pools". You begin with pool 1 of both relics and arcana open. The arcana pool is independent of the relic pool.

Once you've unlocked half of the items in your latest pool with chaos gems, the next pool becomes available both to unlock with chaos gems and for gold during the trials. Items purchased with gold during the trials do not count toward unlocking pools.

In order to efficiently unlock all the pools, purchase only items in the latest pool available to you. There is a list of both arcana and relics by pool on the wiki, but generally items from later pools are more expensive than items in earlier pools.

If you only purchase the most expensive items available to you, and refresh the stores by returning to your base when there are only cheap items for sale, you can unlock all the pools quickly.

Note that you permanently unlock a random Signature arcana whenever you defeat one of the three main bosses. As such it's inefficient to purchase Signature arcana if all you want to do is unlock pools.