How do I collect SNMP readings from intermittently-connected sites?

I am collecting SNMP data on-site for a number of systems, currently using Cacti. These systems are spread on a number of sites that aren't always connected to internet, but I also need to centralize the data on a single system (datacenter housed server) and get graphs out of it.

If I directly poll remote systems with a centralized Cacti I'd lose data when a site is not connected to internet. I should record data on-site (I have a server at each site and I can run whatever I want on it) and then 'sync' everything to the central system.

One hack could be a cacti or directly an rrdtool on site and then periodically rsync RRD data to the central Cacti system, but that doesn't sound like a 'clean' solution: every RRD would have to be defined at both places and rsync scripts setup with the specific file names.

Can you suggest a better solution? Cacti is not a requirement but I'd like to use something like that on the central system. On-site systems need only to collect data I don't need to graph it there or manage users rights to view data and stuff like that, users will only access the centralized system.

Solution 1:

I would setup a remote collection system (a simple snmp poller) that stores data remotely and can be contacted via http or ssh ... the central server would try to contact the remote service service at regular intervals. When it manages connect, it gets the updates that have been collected in the mean time ...

Since rrdtool works well with timestamped updates, you can then re-play the updates into the rrds at the central site ...

As a bonus you may even see quite good performance since running multiple updates into an rrd in a single call is quite a bit more effective than doing it step by step.

Solution 2:

You need some mechanism to store and forward. RRD has a rrdcached so maybe you could cache and then at certain times flush to a NFS mounted disk.

You could also dump, send the files and restore. This may be safer than moving the entire file.