"Replace with" versus "replace by"

OP is right to suspect active/passive has a bearing on preferred usage. From Google Books...

1: Active voice favours with...
The company replaced workers by machines - 9 results
The company replaced workers with machines - at least 180 results¹

2: Passive voice favours by...
Workers were replaced by machines - at least 160 results¹
Workers were replaced with machines - 8 results

To be honest, I can't say I think there's anything wrong with the "less favoured" versions above, and it would be ridiculous to suggest there's any semantic difference. But note that whereas...

Tom replaced Dick by Harry
Tom replaced Dick with Harry

...are both equivalent (manager Tom took Dick off the team, and put Harry in instead), if we want to put that into the "passive" voice, we can only recast it as...

Dick was replaced by Tom with Harry
...or (more likely, imho)...
Dick was replaced with Harry by Tom

That's to say, if the "passive" form actually specifies the "agent", we have to use by for that agent. So we can only use with for the "replacement" in such (slightly contrived) constructions.

¹ Google Books has changed somewhat since I originally did these searches - the smaller values are easy to count so they're accurate, but the "at least" values just reflect how many pages of 10 hits each I could scroll through before GB stopped returning any more (sometimes it just truncates relatively large result sets for no obvious reason). Whatever - that huge reversal of ratios is still unmistakable.

In many, (maybe even most?), contexts, replace with and replace by will be interchangeable.

I don't think this has to do with active or passive voice.

Two types of objects can follow replace with and replace by -- the means / method of replacement or the new content. I would say there is a bias for using by to indicate means and with to indicate the substituted content, but I don't think this is an absolute rule:

an example of the former:

Replace numbers by doubling them.

An example of the latter:

Replace all words with codes.