Prove/disprove: if $\det(A+X) = \det(B + X)$ for all $X$, then $A=B$

I have to prove/disprove this:

If $\det(A+X) = \det(B + X)~ \forall X \in M_{n \times n} (\mathbb F) \rightarrow A = B$

I believe it is true but I can not think of a direct way to prove it. Any help is appreciated!

Let $M_1$ be the set of matrices such that the first row of all the matrices in $M_1$ is the negative of the first row of $A$. Then $\det(A+X)=0$ for all $X\in M_1$, so we also have $\det(B+X)=0$ for all $X\in M_1$.

I claim that the first row of $B$ must be equal to the first row of $A$. Assume contrariwise that this is not the case. Then all the matrices of the form $B+X, X\in M_1$ share the same non-zero first row (= the first row of $B-A$). But by a suitable choice of $X$ from $M_1$ we can make the remaining $n-1$ rows of $B+X$ to be anything we want. As the first row of $B+X$ was assumed to be non-zero, we can find an $X\in M_1$ such that $\det(B+X)\neq0$. This is a contradiction.

Clearly we can repeat the argument for any other row, so we can conclude that $A=B$.

A characterization of the determinant is that $$ \det M=\sum_{s\in\mathfrak S_n}\varepsilon(s)\cdot\prod_{i=1}^nM_{is(i)}, $$ where $\varepsilon(s)$ is the signature of the permutation $s$ in $\mathfrak S_n$. Applying this to $M=A+X$, one gets a polynomial in $\mathbb F[X_{ij},1\leqslant i,j\leqslant n]$, namely, $$ \det(A+X)=\sum_{s\in\mathfrak S_n}\varepsilon(s)\cdot\prod_{i=1}^n(X_{is(i)}+A_{is(i)}). $$ Fix $t$ in $\mathfrak S_n$ and $1\leqslant k\leqslant n$. The coefficient of $$ \prod_{i\ne k}X_{it(i)}, $$ in $\det(A+X)$ is $\varepsilon(t)A_{kt(k)}$ hence the polynomial $\det(A+X)$ fully determines $A$.

I just see this problem today, so this is a late answer.

We can use the same trick I used in an answer to another question. By elementary row/column reductions, we have $B-A=P(I_r\oplus0)Q$ for some invertible matrices $P$ and $Q$, with $r=\operatorname{rank}(B-A)$. Therefore, by putting $Y = P^{-1}(A+X)Q^{-1}$, the given assumption is equivalent to $\det(Y)=\det((I_r\oplus0) + Y)$ for all $Y$. Since this condition does not hold if $r\neq0$ and $Y=0\oplus I_{n-r}$, we conclude that $r$ must be zero, i.e. $B=A$.