'Linux' math program with interactive terminal?
Sage is basically a Python program/interpreter that aims to be an open-source mathematical suite (ala Mathematica and Magma etc.). There are many algorithms implemented as a direct part of Sage, as well as wrapping many other open-source mathematics packages, all into a single interface (the user never has to tell which package or algorithm to use for a given computation: it makes the decisions itself). It includes GP/Pari and maxima, and so does symbolic manipulations and number theory at least as well as them.
It has a command-line mode, as well as a web notebook interface (as an example, a public server run by the main developers).
And, although this might not be relevant since your use-case sounds very simple, the syntax is just Python with a small preprocessing step to facilitate some technical details and allow some extra notation (like [1..4] which expands to [1,2,3,4]), so many people already know it and, if not, learning it is very easy.
As a slight tangent, Sage is actually the origin of the increasingly popular Cython language for writing fast "Python", and even offers an easy and transparent method of using Cython for sections of code in the notebook.
I am aware of two packets that might help You:
- Octave - free clone of Matlab,
- maxima - a packet for symbolic computations.
Octave is good when you want to do numerical calculations and you can accept numerical errors. Especially good for matrix/vector operations. I think this is what you want. Maxima is rather for symbolic computations. Both of them have text interface. If you want more user-friendly interface for the latter one, try wxMaxima.
For modular arithmetic and much more by way of number theory, I would strong recommend GP/Pari from here.
It does a huge variety of operations mostly relevant to number theory, but not limited to that area. It should be easily installable on most versions of Linux.
Added: just checked, and it also does a wide variety of vector/matrix and linear algebra stuff.