Maximizing compound interest with fee per compound
Solution 1:
Presuming that both $p$ and $t$ are measured in years, the number of compoundings per year is $\frac 1p$, and the interest rate for each period is the yearly interest rate divided by the number of compounding per year, or $r_p = \dfrac r{\frac 1p} = pr$.
At the end of the first period, an amount equal to $Ppr - x$ is added to the account (where $r$ is expressed as a fraction out of $1$, not as a percent), totalling
$$P_1 = P + Ppr - x = P(1+pr) - x$$
To make this a little easier, let $a = (1+pr)$, so $P_1 = Pa - x$.
At the end of the second period, the calculation is the same, except the beginning principle $P$ is replaced by the amount $P_1$ in the account at the beginning of the period. $$\begin{align}P_2 &= P_1a - x\\&= (Pa - x)a - x\\&= Pa^2 - xa - x\\&=Pa^2 - x(a +1)\end{align}$$ Continuing we see that $$P_3 = Pa^3 - x(a^2 + a + 1)\\P_4 = Pa^4 - x(a^3 + a^2 + a + 1)\\\vdots$$
Now this sum of powers is well known. If you multiply it by $a-1$ you get $$\begin{align}(a-1)(a^{n-1} + a^{n-2} + \cdots + a + 1) &= a(a^{n-1} + a^{n-2} + \cdots + a + 1)\\&\qquad-(a^{n-1} + a^{n-2} + \cdots + a + 1)\\&=(a^n + a^{n-1} + \cdots + a^2 + a)\\&\qquad - (a^{n-1} + a^{n-2} + \cdots + a + 1)\\&=a^n - 1\end{align}$$ so $$a^{n-1} + a^{n-2} + \cdots + a + 1 = \dfrac{a^n - 1}{a - 1}$$ And we can write $$P_n = Pa^n - x\dfrac{a^n - 1}{a - 1}$$
The number $n$ of periods is going to be $n = \frac tp$. And noting that $a - 1 = pr$, this becomes $$\begin{align}A &= Pa^{t/p} - x \frac{a^{t/p} - 1}{pr}\\&=Pa^{t/p}-\frac x{pr}a^{t/p}+\frac x{pr}\\&=\left(P-\frac x{pr}\right)a^{t/p} + \frac x{pr}\end{align}$$ $$\bbox[5px,border:2px solid] {A =\left(P-\frac x{pr}\right)\left(1+pr\right)^{t/p} + \frac x{pr}}$$