How do we know whether certain mathematical theorems are circular?

The shortest answer is this: because theorem $A$ can only be proven using theorem $B$ if theorem $B$ is already proven. This way, your circular chain can never happen, since $B$ can only be proven using already proven theorems, meaning $A$ cannot be used to prove neither $B$ nor any theorems used in the proof of $B$ (or any theorem used in the proof of a theorem used in the proof of a theorem used in the.... .... used in the proof of $B$)

You should take a look at the Stacks Project. There is a neat feature due to the way theorems and lemmas are organized that allows you took look at dependency graphs for all the results.

In other words, if you look at a certain theorem or lemma (for example here), you can look in the "dependency graphs" section in the lower right to look at every result which is used in the proof, and every result used to prove those results and so on (Here is a dependency graph for the example before).

These graphs would make it clear if there were a circular dependency, but they're also fun just to look at and explore. This is just for algebraic geometry though so I don't know if something like this exists for other parts of mathematics.

I think the argument given by 5xum is the basic reason we trust most proofs. However, I know of several scenarios where circular proofs could possibly get into the literature.

One scenario is a mathematician writing a number of papers published in parallel. If the papers cite each other, there could be a circularity. A prominent mathematician (a Fields medalist) once told me of a case of an apparently circular argument in a series of papers by a famous mathematician. He believed it was actually a deliberate deception. Since I haven't personally verified this, I'm not going to name names. While this is possible, it is a rather rare circumstance.

Another scenario is theorem A is published. Later theorems B, C, ... are proven based on theorem A. Later, other proofs of theorem A are given possibly based on some of the theorems B, C, ... Such later proofs are circular and invalid, but at least the original proof is valid, so we may have invalid proofs but at least the theorems are all correct. Except ... what if the original proof of theorem A was erroneous and no one discovered the flaw? Or, even if the flaw in the original proof was discovered, but the later proofs are still accepted as valid. It may be very hard to sort out what is valid and invalid if there's a lot of intervening work.

Similar things can happen when textbooks are written that cite each other. I know of cases of erroneous statements given as "theorems" in well regarded and frequently cited texts. Often, such "theorems" are left as exercises or are claimed to be easy to prove.

So, ..., I generally trust the mathematical literature, but it's quite possible there are circular proofs out there. I also think there are many undiscovered errors in published proofs. Anyone who's done any computer programming knows that bugs are unavoidable and sometimes maddeningly difficult to find, even in seemingly simple and clear code. I have no doubt the same is true of math, except we don't have to pass our proofs through compilers and run them.

In the long run, I think the answer will be that some day we will run our proofs through compilers, in the form of automated proof verifiers. Gradually, we'll codify all of mathematics into machine verifiable systems that won't allow circular proofs.

There is no guarantee, however as mentioned in the other answers, pivotal results come under close scrutiny which reduces the chance of any circularity. This reflects the fact that proofs are more of a social process than dry formal verification. Assumptions and theorem statements are often tweaked as as consequence.

Many results are based on an intuition, which can be misleading, but in general tends to be a reasonable guide.

Some proofs (the four-colour theorem) are computer based, which adds a whole new dimension to your question.

In general, the longer a result has been around and the more it has been 'used', the more likely it is not circular.