How to properly remove lingering objects when -strict has been set on a large number of DCs for a long time?
I was recently in an environment where there were 120 domain controllers in over 100 sites around the world. This domain began in the Windows 2000 era and was upgraded over time, so strict replication consistency was never set as the default for new DCs and has never been enabled on any DCs. There are lingering objects in the directory and you regularly see a decent number of conflicted objects because of this.
Using repadmin /removelingeringobjects requires knowing two things:
Which DC(s) have lingering objects in the database
A DC that has no lingering objects on it to use as a reference DC.
Obviously, in the future, this environment should be set so that all new DCs have strict replication consistency enforced and repadmin /options * +strict should be run to make all current DCs use strict replication consistency, but that will break replication now without cleaning the objects.
So, my question is this: In such a massive environment where I would have no idea which DCs have lingering objects and which ones do not, how can I identify a good reference DC for repadmin /removelingeringobjects to use, and how can I ensure that all 120+ DCs are clean of lingering objects before enforcing strict replication consistency and breaking replication? Or, is it just easier to turn on strict mode and watch repadmin /replsum to see what breaks and deal with it?
This will take some time to fix.
To stop all replication, run:
repadmin /options +DISABLE_OUTBOUND_REPL
On all DCs. Remember that the above setting does not prevent manual replication actions such as an admin (you) running repadmin /syncall /APed, etc. But that's a good thing because it allows you to get all your DCs back into sync totally before re-enabling regular replication.
Repadmin determines that it's a lingering object if the object exists on ServerA but not on ServerB, where ServerB is the reference DC. The difference between replicating newly created objects and replicating updates to already existing objects is the key. Replicating newly created objects = good. Replicating updates to already existing objects = good. Replicating updates to objects that don't exist on the destination DC = bad.
You only need to lather, rinse, repeat until all DCs match up with your one good reference DC. Then turn on strict consistency everywhere, then re-enable replication. Yes, you do run the risk of deleting legitimate objects that were created on other remote DCs that have not replicated to your reference DC.
From the great "How the Active Directory Replication Model Works" article:
Replication Consistency Setting
If the attributes on a lingering object never change, the object is never considered for replication. However, if an attribute changes, the attribute is considered for outbound replication. Because the destination domain controller does not hold the object for the attribute that is being replicated, an update cannot be performed. How this condition is resolved depends on the replication consistency setting on the domain controller.
A registry setting on domain controllers that are running Windows Server 2003 or Windows 2000 Server with SP3 provides a consistency value that determines whether a domain controller replicates and reanimates an updated object that has been deleted from all other replicas, or whether replication of such objects is blocked. The default settings are different on domain controllers that are running Windows 2000 Server with SP3 and Windows Server 2003.
Strict Replication Consistency
To avoid problems with reanimating objects that have been deleted, a domain controller that is running Windows Server 2003 in a newly created (not upgraded) Windows Server 2003 forest blocks inbound replication by default when it receives an update to an object that it does not have.
Note • Active Directory replication uses update tracking to differentiate between replicating a newly created object and updating an attribute for an existing object. Replication of a lingering object is an attempt to update an attribute or attributes of an object that the destination domain controller cannot update because the object does not exist.
Replication is halted in the directory partition for the object until the lingering object is removed from the source domain controller or the strict replication consistency setting is disabled.
When ServerB says to ServerA: "Hey, some updates have been made to existing objectA." Then ServerA says: "Wait what? I don't even have objectA at all. Send me the entire object!" If no strict consistency. If strict consistency, ServerA says: "Wait what? How do you expect me to update an object that doesn't exist? Go get bent!"
To find if you have lingering objects on a domain controller:
repadmin /removelingeringobjects ServerName ServerGUID DirectoryPartition /advisory_mode
ServerGUID is the known good reference DC. I know you already know this... and how to script the above line to run it on all DCs... (foreach ($DC In $(Get-ADDomain).ReplicaDirectoryServers) { })...
You need a good source DC to compare against, bottom line. If you don't have a known good source DC or don't know, you just have to pick one. It should be a writable GC of course. It's relative - if all domain controllers agree on the existence of an object, and that object's attributes... then it's not a lingering object.
foreach($GC In $(Get-ADForest).GlobalCatalogs) { repadmin /removelingeringobjects $_.name 85d158d2-a006-4fff-b1e5-f9b6eaabab2b '$directoryPartition'
That's resyncing that directory partition of every GC in the forest with the known good source which you need to specify as the GUID.
Then after you've got all your domain controllers once again all in agreement, and replication is happy... then you go and start flipping on strict consistency on all of them.
Edit: This is Microsoft's party line on the issue, and what they'll likely talk you through were you to call them.
Finally, this may be more trouble to fix than it's worth unless it's causing you problems. I hate to say it, but AD can still function normally with lingering objects in it.
The general principle in a case where you can't identify a clean DC is as follows:
for each $sourceDC in $allDCs
for each $targetDC in $allDCs
if ($targetDC <> $sourceDC) then
run repadmin with $sourceDC and $targetDC
end if
next
next
This process is described here: http://blogs.technet.com/b/glennl/archive/2007/07/26/clean-that-active-directory-forest-of-lingering-objects.aspx
However, take a look at ReplDiag. It automates the process by running repadmin for you against all combinations of source and target DC's. It then follows up with /advisory_only to check for any further lingering objects.