Practical limit on groups in AD?
Solution 1:
You don't really define your replication topology, which can come into play here. Assuming you have a single site with all DCs in the same LAN, replication won't be your issue. Simply having thousands of groups normally isn't a problem, unless you have severe restrictions on replication (like you do it across the country over two soup cans and a piece of string).
The problem that you may face is that a user's access token can only contain 1024 SIDs. Once the user is a member of about 1000 groups, some SIDs can't be added to the token, which will cause an access failure when trying to use a resource that requires that token.
In short, if you have a user being a member of 1,000 groups you'll have problems. If not, you're fine.
This TechNet article covers the problem pretty well and this Microsoft document explains it in depth (warning: word document direct link).
Solution 2:
It depends on how you plan to scale.
First you should use nested groups and a RBAC approach.
For scaling you will need other domain controllers to take the load from the primary one. You will need to design the application to be able to fail-over for DNS, LDAP, Kerberos services. You might need to split your structure in different OU. This will help you to delegate and to have less objects in an OU.
Before going to do big changes, searc, try to reproduce your current setup in a test environment and do load testing based on the current queries (make a baseline). Then do the changes in the test environment and do the load test again and assume a higher number of requests.