Matrix A of shape (5, 9) or with shape(5, 9)?
This might be on-topic here; it might be on-topic on Mathematics.
One question there indicates that of is acceptable:
Consider a matrix, A, of shape m × n.
However, both the other options are entirely understandable, and grammatical.