Most ambiguous and inconsistent phrases and notations in maths

Solution 1:

1. 'The function $f(x)$'. No, the function is $f$.
2. Let $f$ and $g$ be real differentiable functions defined in $\mathbb R$. Some people denote $(f\circ g)'$ by $\dfrac{\mathrm df(g(x))}{\mathrm dx}$. Contrast with the above. I discuss this in greater detail here.
3. The differential equation $y'=x^2y+y^3$. Just a minor variant of 1. Correct would be $y'=fy+y^3$ where $f\colon I\to \mathbb R, x\mapsto x^2$, for some interval $I$.
4. This is one I find particularly disgusting. "If $t(s)$ is a function of $s$ and it is invertible, then $s(t)$ is the inverse", lol what? The concept of 'function of a variable' isn't even definable in a satisfiable way in $\sf ZFC$. Also $\left(\frac{\mathrm dy}{\mathrm dx}\right)^{-1}=\frac{\mathrm dx}{\mathrm dy}$. Contrast with 1.
5. In algebra it's common to denote the algebraic structure by the underlying set.
6. When $\langle \,\cdot\,\rangle$ is a function which takes sets as their inputs, it's common to abuse $\langle\{x\} \rangle$ as $\langle x\rangle$. More generally it's common to look at a finite set $\{x_1, \ldots ,x_n\}$ as the finite sequence $x_1, \ldots ,x_n$. This happens for instance in logic. Also in linear algebra and it's usual to go even further and talk about 'linearly independent vectors' instead of 'linearly independent set' — this is only an abuse when linear (in)dependence is defined for sets instead of 'lists'.
7. 'Consider the set $A=\{x\in \mathbb R\colon P(x)\}$'. I'm probably the only person who reads this as the set being the whole equality $A=\{x\in \mathbb R\colon P(x)\}$ instead of $A$ or $\{x\in \mathbb R\colon P(x)\}$, in any case it is an abuse. Another example of this is 'multiply by $1=\frac 2 2$'.
8. Denoting by $+$ both scalar addition and function addition.
9. Instead of $((\varphi\land \psi)\to \rho)$ people first abandon the out parentheses and use $(\varphi\land \psi)\to \rho$ and then $\land$ is given precedence over $\to$, yielding the much more common (though formally incorrect) $\varphi\land \psi\to \rho$.
10. Even ignoring the problem in 1., the symbol $\int x\,\mathrm dx=\frac {x^2}2$ is ambiguous as it can mean a number of things. Under one of the common interpretations the equal sign doesn't even denote an equality. I allude to that meaning here, (it is the same issue as with $f=O(g)$).
11. There's also the very common '$\ldots$' mentioned by Lucian in the comments.
12. Lucian also mentions $\mathbb C=\mathbb R^2$ which is an abuse sometimes, but not all the time, depending on how you define things.
13. Given a linear map $L$ and $x$ on its domain, it's not unusual to write $Lx$ instead of $L(x)$. I'm not sure if this can even be considered an abuse of notation because $Lx$ is meaningless and we should be free to define $Lx:=L(x)$, there's no ambiguity. Unless, of course, you equate linear maps with matrices and this is an abuse. On the topic of matrices, it's common to look at $1\times 1$ matrices as scalars.
14. Geometers like to say $\mathbb R\subseteq \mathbb R^2\subseteq \mathbb R^3$.
15. Using $\mathcal M_{m\times n}(\mathbb F)$ and $\mathbb F^{m\times n}$ interchangeably. On the same note, $A^{m+ n}=A^m\times A^n$ and $\left(A^m\right)^n=A^{m\times n}$.
16. I don't know how I forgot this one. The omission of quantifiers.
17. Calling 'well formed formulas' by 'formulas'.
18. Saying $\forall x(P(x)\to Q(x))$ is a conditional statement instead of a universal conditional statement.
19. Stuff like $\exists yP(x,y)\forall x$ instead of (most likely, but not certainly) $\exists y\forall xP(x,y)$.
20. The classic $u=x^2\implies \mathrm du=2x\mathrm dx$.
21. This one disturbs me deeply. Sometimes people want to say "If $A$, then $B$" or "$A\implies B$" and they say "If $A\implies B$". "If $A\implies B$" isn't even a statement, it's part of an incomplete conditional statement whose antecedent is $A\implies B$. Again: mathematics is to be parsed with priority over natural language.
22. Saying that $x=y\implies f(x)=f(y)$ proves that $f$ is a function.
23. Using $f(A)$ to denote $\{f(x)\colon x\in A\}$. Why not stick to $f[A]$ which is so standard? Another possibility is $f^\to(A)$ (or should it be square brackets?) which I learned from egreg in this comment.

Solution 2:

The inconsistent treatment of raising trig functions to powers: $$ \sin^n x \,.$$

Seriously, starting ab inito $$\sin^2 x$$ could mean either $$\sin( \sin(x) )$$ if you are a quantum mechanic and like to see everything as an operator or as $$(\sin x)^2$$ which is the conventional meaning.

So why is $$\sin^{-1} x$$ used for $$\arcsin x$$ (which is vaguely consistent with the former) instead of $$(\sin x)^{-1} $$ in keeping with the latter.