In how many ways can 4 males and 4 females sit together in a round table with such that there are exactly 3 males who are seated next to each other?

Since the table is circular we can use the group of three men as a reference point, after which there are just three possibilities for the position of the fourth man – one, two or three women between him and the nearest man to his left (this can be counted manually). Thus the $(6-1)!-(5-1)!$ should be just $3$ instead, and all other factors are correct, leading to $1728$.

Here is another approach you can take.

Take the quietest man and seat him first and then seat women in $4!$ ways. Now the remaining $3$ men must be seated together between any two adjacent women - there are $3$ such spaces between women.

Finally arrange men in $4!$ ways.

So the answer is $ \displaystyle 4! \cdot 3 \cdot 4! = 1728$

For your subtracted off case (where four men are seated together), the fourth man could be left or right of the group of three men designated to be seated together. So you need to double your subtracted off term.