allocate matrix in C

Well, you didn't give us a complete implementation. I assume that you meant.

int **mat = (int **)malloc(rows * sizeof(int*));
for(int i = 0; i < rows; i++) mat[i] = (int *)malloc(cols * sizeof(int));

Here's another option:

int *mat = (int *)malloc(rows * cols * sizeof(int));

Then, you simulate the matrix using

int offset = i * cols + j;
// now mat[offset] corresponds to m(i, j)

for row-major ordering and

int offset = i + rows * j;
// not mat[offset] corresponds to m(i, j)

for column-major ordering.

One of these two options is actually the preferred way of handling a matrix in C. This is because now the matrix will be stored contiguously in memory and you benefit from locality of reference. Basically, the CPU cache will a lot happier with you.

The other answers already covered these, but for completeness, the comp.lang.c FAQ has a relevant entry:

How can I dynamically allocate a multidimensional array?

what you can do is

int (*mat)[col];
mat=(int (*)[col])malloc(sizeof(*mat)*row);

and then use this new matrix as mat[i][j]

You may also use calloc, which will additionally zero initialize the matrix for you. The signature is slightly different:

int *mat = (int *)calloc(rows * cols, sizeof(int));