What is the "right" extra information needed to define the trace of a map between two different vector spaces?

As you observe, if we choose an isomorphism $A:W\to V$, we can define the trace of $f:V\to W$ to be $$\mathrm{Tr}_A(f):=\mathrm{Tr}(fA)=\mathrm{Tr}(A f).$$ If $B:W\to V$ is another isomorphism, then $B=MA$ for some $M\in GL(V)$, and $$ \mathrm{Tr}_B(f)=\mathrm{Tr}(MA f).$$ If $\mathrm{Tr}_A=\mathrm{Tr}_B$, then $\mathrm{Tr}(MC)=\mathrm{Tr}(C)$ for every $C\in\mathrm{End}(V)$. This implies $\mathrm{Tr}((M-\mathrm{Id}_V)C)=0$ for all $C$, which implies $M=\mathrm{Id}_V$. So the trace function determines the isomorphism from $W$ to $V$.

By contrast, $det(MA)=det(A)$ for all $A$ if and only if $det(M)=1$. So knowing the determinant function only defines an isomorphism of $W$ with $V$ modulo determinant $1$ maps. The orbit space $\mathrm{Iso}(W,V)/SL(V)$ can be identified with $\mathrm{Iso}(\bigwedge^d W,\bigwedge^d V)$.