$A\oplus C \cong B \oplus C$. Is $A \cong B$ when $C$ is finite, A and B infinite.

So my question is simply that for groups $A, B, C,$ if C is finite, A and B infinite and $A\oplus C \cong B \oplus C$, is $A \cong B$? My gut tells me this must be the case, and logically I can find no reason they shouldn't be, but I can not seem to derive a formal proof of this for the life of me.

Note that I am well aware that this is not always true for infinite C.

Yes, this is true. A group $C$ is cancellable if $A\times C\cong B\times C\Rightarrow A\cong B$. Hence, the result you are after is "finite groups are cancellable". This holds, and does not depend on the groups $A$ and $B$.

That finite groups are cancellable was first proven by B. Jónsson and A. Tarski (1947), who proved it in the more general case of "algebras" (sets with operations, but so far as I can tell this is not the modern notion of an algebra - but I may be wrong). See their book Direct Decompositions of Finite Algebraic Systems, Theorem 3.11. If you are not convinced that their general result holds for groups, see the introduction, or read Definition 1.1 from Chapter 1, where they define an algebra and explain how to view a group in this way (the purpose of the book is to generalise results on direct products of groups to direct products of "algebras", but they note in the introduction that the result we are discussing here seems not to have been proven before).

For a purely group-theoretic argument, see the paper On cancellation in groups by R. Hirshon (the link is behind a paywall). This paper also proves that $\mathbb{Z}$ is not cancellable.