Origin and Literal Breakdown of "Court-Martial"
The term "court-martial" refers to military institutions whereby those accused of breaking the law are brought to trial.
This word is used in both the way we would use "trial" to refer to the proceeding ("I went to trial/I face a court-martial"), but it is also used like a verb ("I was court-martialled"). I have seen reference to "courts-martial" as is standard for pluralizing hyphenated terms.
Usage of the words "court" and "martial" is fairly simple. The former referring to the institution and/or the physical location, while the latter is a modifier to make something refer to war/military/fighting: martial arts, martial law, etc.
It is the combination of the two is where I would like some additional insight. It seems to me that "Martial Court" would be the natural way to link the two concepts, but here we are with a hyphenated word...and I would otherwise expect a more formal term than "court-martialling" to use as a verb for so severe an action.
@Josh gave the answer as a comment, but for good form:
Court martial is a calque of French cour martiale (Italian corte marziale). The word cour is the equivalent of court (tribunal), whereas martiale is a feminine adjective. It means litterally "a war tribunal".
A more standard way of translating it would have been martial court.
Perhaps it is a grammatical attraction (definition 1.4): it started as a plain French phrase (as e.g. fait accompli) but since it was commonly used in the military forces (where not everyone was highly educated), some people thought it wise to add a final "t" to court, and then others decided to remove the final "e" from martiale to make it fit together.