How to answer mathematical questions in the proper way

You mention mathematical writing and symbols/notation. Though I’ve never taken a linear algebra course, here are some relevant tips:

1. Make sure every line of work has a meaningful relation (symbol) in it. This helps you label your answers.
2. If you define your own notations, explain that.
3. Aesthetically organize your work.
4. Do not be afraid to blend English (or whatever language you write in) with mathematics.
5. Use an ample and appropriate vocabulary of mathematical jargon.

_{I will stick to calculus examples, because it is something I know and that most readers can probably follow.}

1. Meaningful Relations & Labeled Answers

This practice is the easiest to implement and comes directly from an instructor of my own, so I’ve put it first and foremost. I don’t have the clearest explanation, but connecting mathematical phrases without a relation symbol is illogical.

Differentiate $\newcommand{\b}{\boldsymbol} \b{x^2+e}$ with respect to $\b x$.
$\newcommand{\r}{\color{red}}\qquad\qquad\quad\r{ ↳ 2x}$

This is a bad answer because there no is meaningful relation specified—namely equality. You need that equals sign!

Differentiate $\b{x^2+e}$ with respect to $\b x$. $$\newcommand{\g}{\color{green}} \g{\frac{d}{dx}\left(x^2+e\right)=2x}$$

2. Explain Notations

You can’t just assume your grader will intuitively follow your approach to problem solving. This is especially true with notation, where no absolutes exist! As a result you need to explain the meaning of any notation you make up (that wasn’t given in the problem).

Prove that $\b{\displaystyle\frac{d}{dx}\int_a^x f(t) \, dt = f(x)}$ for constant $\b a$. $$\r{\frac{d}{dx}\bigl(F(x) - F(a)\bigr) = F’(x)-(F(a))’=f(x)}$$

The problem here is that the meaning of $F$ has not been explained explicitly. (Yes, you can intuit the meaning here, but that’s beside the point.) Also, the final value $f(x)$ has not been attached to the initial integral, so since there is no appropriate labeling, the whole thing is irrelevant! A better response might go something like:

Prove that $\b{\displaystyle\frac{d}{dx}\int_a^x f(t) \, dt = f(x)}$ for constant $\b a$.

$\g{\text{Taking $F’(t)=f(t)$},}$ $$\begin{align} \g{\frac{d}{dx}\int_a^x f(t)\,dt} &\g= \g{\frac{d}{dx}\left(F(t)\bigr|_a^x\right)} \\[2ex] &\g= \g{\frac{d}{dx}\bigl(F(x)-F(a)\bigr)} \\[2ex] &\g= \g{F’(x) - \frac{d\,F(a)}{dx}} \\[2ex] &\g= \g{F’(x)-0} \\[2ex] &\g= \g{f(x)} \end{align}$$ $\g{\text{since $F(a)$ evaluates as a constant.}}$

The above example also demonstrates point 3 by spacing lines of work appropriately and point 4 with its last corollary.

3. Aesthetically Organize Your Work

Do not be afraid to use a bunch of paper! (I promise that deforestation of luscious ecosystems is not caused by mathematicians but rather greedy capitalists purchasing plots of the Amazon on which they implement slash-and-burn agriculture and which they abandon after one season of soy.)

This is how I record my answers on math tests:

This setup allows me to use ink (which is more visible), switch back and forth between questions, et cetera, because I merely cross out erroneous work with a single line and get more paper when I need it.

If you’re doing stuff with matrices, you might invest in graph paper.

DON’T try to solve an entire problem in the blank space under the prompt. That’s a mess!

Also, be sure to align your writing appropriately.

4. Blend In Spoken Language

Sometimes, a blend between spoken language and mathematical notation is advantageous as a more natural way to communicate yourself. You want your math and your language to supplement one another.

What is the definition of a limit? $$\r{\lim_{x\to c}f(x)=L \iff \forall\epsilon>0:\exists\delta>0:\lvert x-c\rvert<\delta\Rightarrow\lvert f(x)-L\rvert<\epsilon}$$

If you were a grader, would you (a) spend time sorting through that mess and (b) be satisfied that you the student have an understanding of whatever that mess is?

What is the definition of a limit?

$\newcommand{\G}[1]{\color{green}{\text{#1}}} \G{Let $f(x)$ be a function,}$ $\G{and let $c$}$ $\G{be an $x$-value}$ $\G{to approach.}$ $\G{It is}$ $\G{defined that}$ $\G{$\lim_{x\to c}f(x)=L$}$ $\G{if the values}$ $\G{of $f(x)$ get}$ $\G{arbitrarily closer to $L$}$ $\G{(generally notated }$ $\G{as $\lvert f(x)-L\rvert<“\epsilon”$)}$ $\G{when $x$ gets}$ $\G{arbitrarily closer}$ $\G{to $c$ (generally}$ $\G{notated as}$ $\G{$\lvert x-c\rvert<“\delta”$).}$

5. Augment Your Mathematical Vocabulary

You need obviously to know the terms of whatever unit you’re studying, but in addition to that, speaking like a mathematician will both prove to the grader that you are knowledgeable clarify your explanations.

Some terms that come to mind include:

• if and only if (iff / $\iff$)
• for some ($\exists$)
• for all ($\forall$)
• implies ($\implies$)
• arbitrary

_{I invite other users to expand this list with what comes to their minds.}

I hope these tips help, and I will expand this answer some more as more ideas come to mind.

I don't have any literature recommendations, but I'll give an impression of my personal ideas. Note: I'm a student, not a teacher, so this will be a mix of my opinions and my experience of what works well with "real-world professors". ;)

Most questions in a mathematics course ask for a proof. If they don't, they often ask to simplify a certain expression to produce a value, which can be seen as a proof question simply by giving a proof that your answer is correct. In proof questions, I believe the most important aspects of quality are clarity and rigour.

Clarity

Clarity is about making sure the reader actually understands what you're trying to explain. Since a proof (in anything but formal logic) is really nothing more than an explanation to a mathematician of why a statement is correct, having this reader understand what you're saying is important. This entails defining any new notation or variables you introduce and properly linking together your ideas in English (or whatever language you're using).

Remember that it's fine for you answer to be a mix of symbols and normal text, and you can use this to your advantage. If you see that a certain equality holds for reasons A and B, say something like "We have the equality $\ldots = \ldots$ because A and B." If you can show your equality using a few intermediate steps ($\ldots = \ldots = \ldots = \ldots$), but one of the equalities there requires some explanation, make a little mark above the relevant equals sign (e.g. $\stackrel{\star}=$) and afterwards explain why the equality at $\star$ holds.

Rigour

Rigour is about making sure that every step you take is clearly true. If the statement you're required to prove was obvious, you'd be able to say so and be done, but usually (always? Education seems to work that way) that's not the case, so you'll have to explain why the statement is true. (Or why the given values are equal, or why your answer is correct, etc.)

In mathematics, a proof is really a series of statements, each of which follows clearly from the previous statements, the last of which is the thing you're required to prove and the first of which are the things you're allowed to assume.^[1] Often you'll be able to omit your assumptions in the proof on an exam, but for structure, to make sure you're not forgetting any assumptions, and for allowing the professor to check whether you read the assumptions in the question correctly, it might be useful to actually write down your assumptions explicitly before you start on your proof.

Then in the proof, as said above, each of the statements you make, each of your claims, should follow "obviously" from your previously made claims. What steps are obvious and which aren't is completely dependent on your mathematical experience and exposure to the specific subject you're studying, but in general I like to say that if you feel you need to say explicitly that something is "trivial", you apparently think it's easy, so it should be easy to write down an actual proof as well. :) If you feel you need to reach for intuitive arguments, it's probably worth it to try giving a formal argument anyway, that doesn't use anything but logic and theorems that you can assume to be true. (A basic truth is also a theorem, of course; you won't need to argue why $3 + 5 = 8$, since that follows from the definition of addition that your reader is expected to know. In linear algebra, you also won't need to argue why, for a matrix $A$ and a vector $v$, $Av = \vec 0$ implies that $v \in \mathrm{ker}A$, since the definition of the kernel of a matrix is enough to make that clear.)

Putting everything together, and more

Clarity and rigour have a lot to do with each other, of course, and only together do they make a good proof. Decomposing your argument into simple steps that are clearly true will surely help with clarity, since those simple steps won't require any involved explanations. But clearly explaining what you're doing and why that works should also help with rigour, since a good explanation is far more likely to convince the reader that what you think is correct is actually correct; and convincing the reader that your argument is correct is really convincing them that the statement you're proving is correct, which was the goal of your proof in the first place.

Of course, there are other aspects to consider, like good structure, i.e. an aesthetically pleasing and simple layout. If you write longer mathematical expressions, give them their own space: don't write them inline with text, since that just reduces visual clarity. But if you have a short mathematical expression that really makes sense to read in the middle of a sentence, by all means write it inline; after all, that's how the reader is supposed to read your sentence.

Using the right vocabulary is also important: don't explain a concept in general terms when there is a well-defined term that means what you want to say. But don't overdo this one: if it is really clearer to use less dense language, do so! If the teacher was testing for your mastery of terminology, he/she would have asked for that explicitly, not asked for this proof.

Sometimes, in longer proofs or proofs that use induction, you'll make a claim first and only then prove that claim, after which you'll use the fact that your claim was true to work the rest of your proof. This may not seem to directly fit in the model described above of a series of statements where each follows from the last ones, but it does if you see this claim and its proof as a separate theorem (that is only "by chance" stated and proved in the middle of your proof), after which you invoke your claim, then being obvious because it's an already proven theorem. In proofs using induction, this often happens with the two sub-proofs for the induction basis and the induction step, but otherwise this is usually reserved only for more involved proofs that you won't likely write on an exam. But knowing the technique is useful, since it may come in handy once in a while.

Practice

I hope the above helps you a bit. If you can, practice a lot, and have your proofs read by e.g. older students, or teachers if you can get them to (hint: they're more likely to proofread a short and clear proof than a long and convoluted proof, which goes back exactly to the points I mentioned above!). In my experience, proof writing is just experience and heuristics; there's usually umpteen ways to write down a specific mathematical proof. But on the other hand, many mathematicians have (in their minds) pretty clear/strong ideas on what proofs are nice and what proofs aren't, and as you mature mathematically, you'll both learn how to convince other mathematicians efficiently and what proofs you like to read. Mix both together, and you might just produce a good proof!

[1]: Tip: take a course in formal logic! There you'll learn to do this really formally, specifying exactly what previous statements you're using, and why (i.e. using what rule of inference) the current statement follows from the previous ones.

(EDIT: Footnote [1] and paragraph on claims in proofs)

Its all about what and why! the best tip I can provide is that after writing your answer, read it carefully and try to find loose ends, that is, when you write a sentence ask yourself these two questions: (1) Why have you written this sentence? (this is to check whether the sentence is at all necessary or not) (2) why the sentence is valid? ( explanation of the sentence). Now the most important part: is the explanation to the question (2) written in your answer? The fundamental part is that you must provide reasons behind your claims and most importantly, the mistakes newbies make: check whether you have clearly mentioned which element belongs to which set and before you use an alphabet to denote something, state it. For example, if you write "Let $x \in V$", your answer must contain a sentence like "Let V be a vetor space over the field F" before you write that. In short, your answer must be complete in explanations, you are likely to get penalized if not written to the point and with reasons. Also use complete sentences with proper grammar.

I have been doing mathematics for a little over three years i do some marking of first year classes as well and this is one of the most difficult i have/still struggle with on occasion is how to write up my solutions.

To make it clear what is going on in mathematics the key point is yes finding the correct solutions but unlike in most other classes that is not sufficient as we wish to teach you how to communicate the correct solution in an understandable way.

Though i agree with the other posters adding mathematical language in the correct context can help save time or make a solution more aesthetically pleasing thats really NOT what this is about.

When writing up your solution communicate what your thinking for example say i am going to X because of Y then do it then say for example this is not quite the solution because (reason) we must do Z still then do Z and say thus or therefore or hence this is the solution.

Writing up your solutions in words or technobable isn't important its communicate that you actually understand the conclusion you reached and didn't copy it for example by communicating your thought process to the marker you will get those marks your losing and in the case where you have all the correct method but make a small arithmetic error resulting in an incorrect answer you can still score 3 maybe even 3.5/4 marks on a question if its clear you understood the method but just made a minor mistake (something you may not get any marks for in anthore class.)

In the short term you can also goto your prof or math help center and say hey i think i got the correct solution how can i communicate it in a way that will get me full marks any decent prof will happily help you with this.

You asked for a book. Well Kevin Houston's "How to Think Like a Mathematician: A Companion to Undergraduate Mathematics" seems a good choice.

For more info check out his website for the book, where alongside sample chapters you can also find a solutions pdf.