Is there an upper limit of how much money you can throw in a gold hole?

Yes and no. When you plant 10,000 bells, you will always receive 3 bell bags with the amount you planted. Above 10,000 bells, you may receive either 3 bell bags containing the amount you buried, or 3 bell bags containing 10,000 bells.

Supposing there is a p_full chance for a full payout, the expected profit for any amount b planted above 10,000 bells is:

(1 - p_full) * (30k - b) + (p_full) * (b * 2)

According to some data mining, there is a 30% chance for a full payout, so if you consistently plant 99,000 bells, you have an expected profit of 11,100, vs the guaranteed profit planting 10,000 bells of 20,000.

The expected profit function can be reduced to 21,000 - 0.1 * b, a linear function with negative slope. 10,000 bells is the optimal amount to plant.