Teacher claims this proof for $\frac{\csc\theta-1}{\cot\theta}=\frac{\cot\theta}{\csc\theta+1}$ is wrong. Why?

I would at least have skipped the first line and started with $\dfrac{\cos^2\theta}{\sin^2\theta} = \cot^2\theta.$

One reason why an instructor might have qualms about this argument is the frequency with which students do this in the opposite direction, i.e. they write $$ \require{cancel} \xcancel{ \begin{align} \frac{\csc(\theta)-1}{\cot(\theta)}&=\frac{\cot(\theta)}{\csc(\theta)+1} \\[8pt] \frac{\csc^2(\theta)-1}{\cot(\theta)}&=\cot(\theta)\\[8pt] \csc^2(\theta)-1&=\cot^2(\theta)\\[8pt] \frac{1-\sin^2(\theta)}{\sin^2(\theta)}&=\cot^2(\theta)\\[8pt] \frac{\cos^2(\theta)}{\sin^2(\theta)}&=\cot^2(\theta)\\[8pt] \cot^2(\theta)&=\cot^2(\theta) \end{align} } $$ This would be logically correct if one wrote "This equality is true if the one below it is true." on each line. But one must be clear about the correct direction of "If ... then ...". Without those explicit words, the sequence of equalities can be taken to mean "If ... then ..." in just the opposite order.

Often what instructors want is something like this: $$ \frac{\csc\theta-1}{\cot\theta} = \cdots\cdots\cdots\cdots = \frac{\cot\theta}{\csc\theta+1}. $$ in which each "equals" sign asserts the equality of things already known to be equal.

I would give full credit to the answer that your son wrote if each line had some brief explanation of how it is deduced from the line before it. For example, he could write this:

$$ \cot^2\theta = \frac{\cos^2\theta}{\sin^2\theta} = \frac{1-\sin^2\theta}{\sin^2\theta} = \csc^2\theta-1. $$ Then, dividing both sides of the equality $\cot^2\theta = \csc^2\theta-1$ by $\cot\theta,$ we get $$ \cot\theta = \frac{\csc^2\theta-1}{\cot\theta}. $$ Finally, dividing both sides of that by $\csc\theta+1$ we get $$ \frac{\csc\theta-1}{\cot\theta} = \frac{\cot\theta}{\csc\theta+1}. $$

Note that I wrote words above, not just mathematical notation. Well written answers do that, except perhaps in fairly simple cases.

There are multiple motivations for writing proofs: perhaps to pass an exam (from a regular student's viewpoint), or perhaps to test understanding (from a teacher's perspective). Here, I'd like to highlight another important motivation: to enjoy the process of discovery.

Here's a geometric proof of the trigonometric identity. It is not practical to use in an exam, but it makes the abstract symbols more concrete. Coming up with multiple proofs of the same result can help one develop intuition and allow one to appreciate the connections between different areas of mathematics.

Problem statement: $$\frac{\csc\theta-1}{\cot\theta}=\frac{\cot\theta}{\csc\theta+1}$$

In this picture involving the unit circle, we construct some lengths which correspond to expressions in the problem statement. We start with $\theta$ and $\phi = \frac{\pi}{2} - \theta$.

Basic trigonometric definitions lead to the blue and orange lengths. By some angle-chasing (or a circle theorem), we have the $\color{red}{\text{red}}$ angle. Then, more angle-chasing gives:

The $\color{red}{\text{red}}$ angles are the same $\implies\triangle ADB\sim \triangle BDE \implies \frac{AD}{DB}=\frac{DB}{ED}\iff$ the problem statement.

I would assume that the teacher considered it an invalid answer, because your son assumed that the equality is true before showing that it is, if that even makes any sense. This is obviously a guess, but in the mind of the teacher, what he did was not "showing" that the equality is true, but merely "verifying" that it is. In other words, what the teacher likely wanted is for the students to get the answer on the right-hand side by exclusively using the left-hand side. More explicitly, even though the students knew what the end result should look like, they should have pretended they don't, manipulated the left-hand side and exclaimed "Eureka! It's true!" when they got the RHS.

With that logic, all he needed to do was to multiply numerator and denominator by $\csc(\theta)+1$, which immediately yields the desired result after simplifications.

All of these methods of proof are valid, and should be graded as such (unless the question specifically said to use a certain method).

Method $1$ - Start with one side, simplify to the other.

$$\frac{\csc(\theta)-1}{\cot(\theta)}=\frac{(\csc(\theta)-1)(\csc(\theta)+1)}{\cot(\theta)(\csc(\theta)+1)}=\frac{\csc^{2}(\theta)-1}{\cot(\theta)( \csc(\theta)+1)}=\frac{\cot^{2}(\theta)}{\cot(\theta)(\csc(\theta)+1)}$$$$=\boxed{\frac{\cot(\theta)}{\csc(\theta)+1}}$$

Method $2$ - Start with the asserted identity, simplify to a known identity.

$$\frac{\csc(\theta)-1}{\cot(\theta)}=\frac{\cot(\theta)}{\csc(\theta)+1}\Longleftrightarrow(\csc(\theta)-1)(\csc(\theta)+1)=\cot^{2}(\theta)\Longleftrightarrow\csc^{2}(\theta)-1$$$$=\cot^{2}(\theta)\ \blacksquare$$

Method $3$ - Start with a known identity, turn it into the asserted identity (which is what your son did):

$$\csc^{2}(\theta)-1=(\csc(\theta)+1)(\csc(\theta)-1)=\cot^{2}(\theta)\Longrightarrow\boxed{\frac{\csc(\theta)-1}{\cot(\theta)}=\frac{\cot(\theta)}{\csc(\theta)+1}}$$

That said, however, perhaps the teacher took points off due to the length of your son's proof, which was more roundabout than it could have been. For instance, he could have started with a Pythagorean identity instead of starting with a '$x=x$' identity and using a Pythagorean identity, and took a few less steps in simplifying to the desired identity.

It's disappointing to see that the teacher believes there is only one way to prove trigonometric identities. Actually, there are a number of approaches, all equally valid and useful in different circumstances. Sometimes it is more elegant to 'start with one side and prove that it is equal to the other', but a proof is a proof.

Your son is actually using quite an advanced technique, where he shows that a statement $P$ implies another statement $Q$, and then shows that all of the steps are invertible, meaning that $P \iff Q$. As long as you are able to show that all of the steps are indeed invertible, this is a very useful method. Let's look at his method in more detail. He started by assuming that $$ \frac{\csc \theta - 1}{\cot\theta} = \frac{\cot\theta}{\csc\theta+1} $$ and working from there. If we cross mutliply both sides, look at what happens: \begin{align} &\frac{\csc \theta - 1}{\cot\theta} = \frac{\cot\theta}{\csc\theta+1} \\[6pt] \implies & (\csc\theta-1)(\csc\theta+1) = \cot^2\theta \\[6pt] \implies & \csc^2\theta-1=\cot^2\theta \\[6pt] \implies & \frac{1-\sin^2\theta}{\sin^2\theta} = \frac{\cos^2\theta}{\sin^2\theta} \\[6pt] \implies & \frac{\cos^2\theta}{\sin^2\theta}=\frac{\cos^2\theta}{\sin^2\theta} \\[6pt] \implies & \cot^2\theta = \cot^2\theta \, . \end{align} Now, try reading the proof 'backwards'. Does $\cot^2\theta = \cot^2\theta$ imply $$ \frac{\cos^2\theta}{\sin^2\theta} = \frac{\cos^2\theta}{\sin^2\theta} \, ? $$ Does $\frac{\cos^2\theta}{\sin^2\theta} = \frac{\cos^2\theta}{\sin^2\theta}$ imply $$ \frac{1-\sin^2\theta}{\sin^2\theta} = \frac{\cos^2\theta}{\sin^2\theta} \, ? $$ Keep going. If the answer to all of these question is 'yes', then we have successfully proven that $$ \frac{\csc \theta - 1}{\cot\theta} = \frac{\cot\theta}{\csc\theta+1} \iff \cot^2\theta = \cot^2\theta \, . $$ And since the RHS is an identity, the LHS must also be an identity. There's just one tiny snag to this proof. When we are going in a backwards direction, and conclude that $$ (\csc\theta-1)(\csc\theta+1) = \cot^2\theta $$ implies $$ \frac{\csc \theta - 1}{\cot\theta} = \frac{\cot\theta}{\csc\theta+1} \, , $$ we are assuming that both $\cot\theta$ and $\csc\theta + 1$ are non-zero. But, your son directly addresses this by stating at the start of the proof, 'for all real $\theta$ not equal to an integer multiple of $\pi/2$...', meaning that this is never a problem. This kind of attention to detail is very impressive for a high-school student.