A simple permutation question - discrete math

Solution 1:

You are going in the right direction, but don't think in terms of slots. Just organize the books on the floor, then put them on the shelf.

First, order the computer science books. There are $5!$ ways of putting them in. Then put the two art books on both ends; there are $2$ ways of doing it.

Now you have $7$ books, with eight spaces between and to the left and right of them. We just need to put in the three math books. Let's say they are a Calculus I, a Calculus II, and a Calculus III textbooks.

We put in the Calc I book first. There are 8 places where it can go, relative to the other 7 books. Next, put the Calc II book in; there are 9 places where it can go, relative to the other 8 books that have already been placed. Finally, put in the Calc III book.

Solution 2:

It's unclear (to me) from your question if the first art book must always be on the left and the second art book must always be on the right? Let's assume that's the case.

Order the computer books first. There are $5!$ ways, since they are distinct. Next put the art books on the ends, and there is only one way to do this (because of my assumption above). Finally, there are $3$ math books left with 8 slots to choose from; there are no restrictions on these math books.

You have $8$ choices for the first math book, $9$ for the second, and 10 for the last. So in total,

$$ 5! \cdot 8 \cdot 9 \cdot 10 = 86400 $$

You should double this if my first assumption is wrong.