Windows Event Forwarding (WEF) Large Scale Environment

We currently use Nxlog on all of our DCs and send that data to a central syslog-ng server. Due to dealing with the agent on each computer and the need for additional agents that only support reading the event viewer, we are debating about using WEF to forward all DC logs to a few servers so we have fewer agents to deal with. In theory this sounds fine, but as I started to read into it, I don't see any ability for HA or clustering. I could probably front end it with a load balance and round robin spray the events to the 5 or so servers on the back end but not sure if that would work the way I want it to.

Does anyone have experience with using WEF in a fairly large environment? We receive around 200 million Windows event logs a day and need to increase the logging level. Also, we have the need for the logs to be as near real time as possible so with this scale, has anyone ran into performance issue on either the DC forwarding logs or latency of the collectors receiving them?

Thanks for your help and input.

Solution 1:

I would highly recommend switching all your agents to elastic beats. I have used nxlog in the past and it simply does not do everything as nice as elastic beats have.

Plus they are written in GO so no dependancies needed.

Syslog-NG is great too, but I've since switched to logstash here as well, it supports clustering, failover, queues, and many exports different (like to graylog or splunk).

Lastly, we deploy our beats to windows and linux with Ansible.