Experimental or experiment?

Saying "experiment setup" refers to the setup of an experiment, like setting up the beakers, thermometers, and whatnot so that everything is ready for conducting the experiment. The setup itself isn't experimental but is merely a task you perform in preparation for conducting an experiment in which you will use what you've setup.

Saying "experimental setup" refers to a setup procedure itself being experimental, like if you try setting things up a new way to see if it's more effective or efficient, you're experimenting with a new setup. That's what an "experimental setup" is, a new setup that you're trying out.

With these two definitions in mind, you could even have an experimental experiment setup. That would be when you experiment with a new setup for conducting an experiment or experiments.

This reminds me of the argument I sometimes have over whether it's better to say 'resonance frequency' or 'resonant frequency'. I go for 'resonance frequency' because it's not the frequency that is resonant but some oscillatory system. Likewise, I prefer 'experiment set-up in the cases you cite, and for rather similar reasons. In both cases, though, I think that the adjective ('resonant', 'experimental') is acceptable instead of the noun.