How do you manage the compatibility between mods in Skyrim?
Solution 1:
You have a few options. Basically: it's not too terribly difficult to identify mods that might conflict. You can then either remove conflicting mods or learn how to make your own compatibility patch.
Use someone else's mod list
As you mentioned, you could run a list of mods provided by someone else, in the order they specify, using their patches. So you would follow the STEP guide and never deviate from it. This should give you a stable game, if you're happy with their choice of mods.
Hope for the best
Try to choose mods that you think will work well together. Avoid mods that look like they edit the same thing, and avoid running more than one overhaul mod at a time unless you can find a compatibility patch or a statement that they're compatible without one. Start up your game and look for problems. Personally, if I make it to level 30 without any serious problems, I assume that my game is stable.
Check for compatibility yourself
File compatibility
Use Mod Organizer and set it to unpack all BSAs. Mod Organizer will show you which mods overwrite the files of other mods (if their BSAs are unpacked). It isn't usually a problem if a mod overwrites USLEEP, but for anything else, you might want to look at it. Sometimes overwriting mods is harmless (e.g. overwriting a couple textures), and sometimes it indicates an incompatibility. If mods are overwriting each other's script files and you don't expect them to, you might run into problems.
ESP compatibility
Run LOOT and make a bashed patch with Wrye Bash. Next, start up TES5Edit and click OK with the default selection. Wait for it to finish and then browse your entire mod list.
TES5Edit will highlight records in each file that conflict with other files. If you see tons of conflicts in a particular mod, it may not be compatible with your other mods. Some conflicts are not a problem: practically everything edits Worldspace 3C (Skyrim), for example. If you have a compatibility patch for specific mods, expect the patch to overwrite/conflict with both of the mods it's patching for.
If you see a lot of conflicts, you can either remove mods until TES5Edit doesn't highlight so many records in red, or you can trek down the road of becoming a Skyrim modder and use TES5Edit to create a compatibility patch yourself.
Script compatibility
Honestly... other than checking for overwritten script files with Mod Organizer, I don't know how you'd check for script conflicts. I just hope that nothing bad happens. In general, just try to find well-written mods and don't use too many script-heavy mods.
Solution 2:
Use ModOrganizer - it will display the conflicts.
When you look at the conflicts you have a few choices. Quickest is to pick one mod to be the "winner" (usually by re-ordering your lists although there are more advanced mechanisms too). That's the most common decision because many conflicts will be things like a mod replacing an original Bethesda texture with something better. Another choice is to go look for compatibility patches. If there's no compatibility patch, another choice is to follow the ordering recommendations of the LOOT tool - but be aware that MO is one of the few tools that would allow the "loose" textures and scripts and other files to be in a different order than the LOOT driven plugin list of esps and esms.
And do what you're already doing - start with S.T.E.P. and think carefully about what you want to change.
And lastly - For stability, try to avoid removing or even upgrading stuff during a play-through. If you do, follow the "clean save" procedure on the S.T.E.P. troubleshooting page.