macOS Content Caching serving lots of data to unknown clients

TL;DR:

Hidden/unknown client has downloaded 300GB of data from my macOS Content Caching (from origin, not cache). Traffic didn't come from my devices according to my testing. This stopped after I disabled SSH and SMB, changed password, and enabled firewall (I did them all at once). What happened? Why and how was it serving so much data?

Full details:

I have setup a headless late 2014 Mac mini on my local network for Apple content caching. It runs macOS Mojave, it’s connected to the network using Ethernet, it’s freshly formatted and up to date as of this writing.

Setup was super straightforward: check a box on sharing, disable sleeping on energy saving, wait for clients to connect.

I had also enabled SSH, VNC and SMB, changed users to “administrators only” on each of them, installed the Server app and few other utilities (docker, AirServer, DriveDx, server apps from the Mac App Store).

I have about 5 clients (Macs, iPhone, iPad, Apple TV) that usually connect to it. None of them normally consume much data: the first 7 days I got it running, they transferred a total of 37GB of data. I have an AirPort TC (WPA2 Personal with DHCP, not NAT) which, every time I checked, only showed these 5 devices on the Wi-Fi network, and the Mac mini is the only one on Ethernet. The ISP router has NAT but wireless is disabled. The macOS Server app shows “reachable, no services”.

But on the 8th day, around the time I decided to do a full SMART disk self-test on the HDD, the content caching server started serving 12GB per hour, every hour, non-stop, from origin (not cache). Seemed strange. I turned off Wi-Fi on all of my devices to sleep it off and see whether it would stop: it didn’t.

As far as I know, the Mac mini itself doesn’t use its own content caching for downloading content for it (which I thought was strange), so I ruled it out as the source of all downloading.

On the third morning of constantly serving 12GB/hour (from origin) and a total of 336GB served, I got very worried and disabled SMB, SSH except VNC (or else I can’t access it), changed the admin password (previously it was neither an easy nor a short one), enabled the firewall (the default is off on a fresh install) and since then, the content caching has been serving 0 bytes for the last hours. Caching is still working for my 5 devices.

Apple says caching server doesn’t do networks other than local by default. My ISP’s upload speed is pretty much constant at 20mbps (2,5MB/s), so I think it’s not completely impossible for this data to have left my local network because it was being served at approximately 25~28mbps.

What could possibly have caused this? Could there be an unknown reason or nefarious client at work here? Could any data have been stolen?

I’ve never used content caching before, but Apple docs say data is encrypted. The log is huge and doesn’t seem to help me because, if I understand correctly, it doesn’t log IPs by default. I didn’t get any warnings for any of my Apple accounts, I have 2-factor auth for pretty much everything.

I also have pi-hole logs, but all network usage is merged into a single, virtual IP.

Thanks in advance and sorry for such a long question.

Solution 1:

Need more details to really say anything conclusively. When the data transfer is happening, you should run sudo nettop and look at the local addresses the AssetCache service is connecting to. This will reveal what device on the local network it's serving, or the lack thereof.

Some other quick thoughts:

• I'm fairly sure content caching DOES apply to the local machine. So updates downloaded by your Mac Mini will be counted and cached.
• Do system wide network usage stats show at least as much data transfer as the Content Caching stats? If not then it's probably just a bug in content caching statistics.
• Are you using iCloud? Content Caching can include iCloud content if you have that enabled.
• Do you have an iOS device plugged in over USB and have tethered caching enabled? That's a potential source.
• Try looking at the content caching log while this is happening: sudo log stream --predicate 'subsystem == "com.apple.AssetCache"' --info