Is "setup" an acceptable noun in formal writing?

I'm editing a draft of a scientific paper which repeatedly uses the word "setup" to refer to the, well, experimental setup.

Example:

The dimensions of the setup are 250 mm × 250 mm × 50 mm.

Every time I see the word "setup" appear by itself, I cringe a little, thinking of, say, Microsoft Windows' SETUP.EXE. However, the phrase "experimental setup" sounds OK.

Is "setup" accepted as a valid word to use for an experimental apparatus, in formal scientific writing in American English?

Yes, it's a common word in scientific literature. Experimental setup is a technical term referring to the collective apparati, procedures, and environmental conditions needed to carry out the experiment in a fashion expected to reproduce the original results. Setup is considered an acceptable shorthand for it, but I'd use "Experimental Setup" at least once before "Setup".

ODO insists on set-up with a hyphen.

OED has this, again under set-up:

2. a. The way in which something is organized, arranged, or constituted; an organization, arrangement, system, or situation; U.S., personal bearing or carriage; Billiards, etc., a position of the balls (e.g. as left by the last player) from which it is easy to score.
Set-up occurs in a vague and indefinite sense in a large number of contexts, but several reasonably distinct areas of use can be isolated: (a) a business or administrative structure or organization; also, an economic, social, or political system (both with reference to the system or the persons involved); (b) a domestic situation, as determined either by lifestyle or personal relationships; (c) a team (esp. in Sport); (d) the layout of some mechanical apparatus or equipment.

but provides a citation

1953 W. S. Burroughs Junkie 7 "But these people were jerks for the most part and, after an initial period of fascination, I cooled off on the setup."

So yes, setup is a recognised, if somewhat deprecated, form.

The omission of hyphens is not uncommon, though. And as coleopterist has commented, ODO does not insist on the hyphen in American English (the American English sentence was a later addition to the question).