Team cohesion in football manager starting squad/reserves
Yes it is a thing. I cannot say to what extent, but the game does track how much you change your starting lineup.
My own (completely anecdotal) experience with FM2014 indicates the effect is small, maybe even non-existent as long as you only rotate a single player or two. It is unlikely to be your main reason for bad performances.