Is everything in mathematics built on add, subtract, multiply and divide?

Not really. For one thing, before you can "add", "subtract", "multiply", and "divide", you have to have something that you are "adding", "subtracting", "multiplying", or "dividing." So that "something" has to precede these operations. And for another, there are many notions that cannot really be expressed in these terms. To take a familiar one if you've taken Calculus, the idea of "limit" is not one that can be expressed in terms of adding, subtracting, multiplying, or dividing, but rather is expressed in terms of "approaching", or of being "arbitrarily close to". (This is one reason why Newton and Leibnitz are so rightly hailed as revolutionaries: the ideas of Calculus were very much unlike what mathematics had been up to that point.)

However, one has to start somewhere. Historically, mathematics started from different starting points in different cultures; counting was certainly one of the firsts things that comes up. But some cultures then realized that they need some more general ideas that went beyond simply counting (with its addition, multiplication, etc). Greeks settled on geometry as their underlying structure, and in fact translated everything about numbers into statements about geometry: lines, points, circles, squares, etc. Other cultures did not.

These days we usually take the route of axiomatic theories, where we have some "primitive notions" that are not defined, and some rules about how we can talk about them and their properties, and then we try to build everything on top of that. One can start from the natural numbers (the "counting numbers") using, for instance, the Peano axioms, and start building from there. Then you are actually starting before you get to "addition", "multiplication", "subtraction", and "division", with only the notion of "number", "next one", and induction. Or we can have other starting points.

Now, today, there is usually one of two starting points: Set Theory, and Category Theory. Both presume that we agree on the basics of logic (the meaning of things like "and", "or", "if... then...", and "not", the meaning of "is equal to", and things like that); and the rules of inference, which are rules that let you go from some statements others; for instance, the rule that says that if you know that A implies B, and you also know that A is true, then you know that B is true.

In Set Theory, you start with the idea of "set" and "element of"; these ideas are not actually defined, you just give some axioms that describe things you can say about them. From these axioms one can define numbers, addition, multiplication, etc. And also build other notions that are orthogonal to natural numbers and their usual operations.

In Category Theory, you start with the idea of "objects" and "maps" between objects (again, not defined; you only get some axioms that tell you how you can talk about them); then you can use these notions to construct everything else in terms of them.

So, rather, in modern mathematics everything is either "sets all the way down", or everything is "categories all the way down". If there is one common set of presuppositions, it's just basic logic.

This is an interesting rhetorical question. On the one hand, many theoretical mathematical concepts transcend arithmetic. However, computer algebra systems (such as Mathematica) are able to express answers to those sorts of questions using a CPU that performs a limited range of simple computations. So the answer is a matter of opinion. Or there is no answer?