Items in minecraft item filter stacking
What I've found is that the sorters work best if you keep the placeholder blocks at 1 per slot, and increase the item you want to sort up to 41.
With the setup you're using, the redstone signal coming from the comparator can easily increase past the amount needed to open the hopper below. This makes it so that the hoppers on either side will open, dropping your placeholder blocks. If the signal from one of the sorters gets big enough, it will start a cascade effect through your sorters, and they'll all start dropping the placeholder blocks.