Protocol choice for Machine-to-Machine communication - level:n00b
Have you thought of using SMTP (email) ? The Raspberry has two processes on it: one reads data and append them to a file, another one (possibly in crontab) moves the files (possibly aggregating some) and send them by email to the destination machine. This way, you have no coupling between the frequency of sampling and the frquency of logging, if it's okay with you (by sending an email every 20s or every minute for example).
The mail can be compressed, encrypted (using SMTP TLS). Moreover it's resilient: in case you lost your uplink, the data we'll be sent when the link is restored. From your logging process point of view, the link is always "up".
On your couchdb server (or another machine connected to the couchdb), you create a dedicated user and put a script in its .forward that unzip the message and feed it to couchdb.
If you want to authenticate, you can go with many schemes from a shared secret to PGP signing !
We do this to feed our data warehouse, because we don't want any kind of incoming connection on it (https or ssh), admittedly not at a 1s sampling rate (but with bigger data).
Last but not least, you can test every component individually (logger, sender, receiver and db-feeder) without needing all the infrastructure running.
Generally, everything depends on what you do on the other side. For example: if you need this data to be forwarded to the Zabbix - you use Zabbix agent, for SNMP - you use snmpd, for web application processing - HTTP etc.
Rsyslog could be a good option for transport, because it can already solve many problems you may encounter while developing custom solution (i.e. when working with REST, generic HTTP):
- It's reliable with RELP
- Has on-disk queuing option, so you can deliver messages after connection is re-established
- Supports gzip message compression
- Supports TLS
- Supports multiple input modules
- And output modules
- Filtering and modification on both sides
TLS will have huge overhead - try to not use it at all.
You will need to setup same system on the other side, then you can do tricks with messages as you want i.e. insert it to the mysql or/and postgresql database, or/and any other with DBI, or/and forward to file, or/and to named pipe, or/and as snmp trap, or/and as email, and/or to custom application etc.
I'm using similar system that you have (Arduino+Raspberry), but I'm running proprietary application with it's specific queue and SOAP transport. For simple messaging, I think I'd use rsyslog.
You could also look into RabbitMQ Since it's a message broker, which is kind of designed for this problem.
RabbitMQ is a message broker. The principal idea is pretty simple: it accepts and forwards messages. You can think about it as a post office: when you send mail to the post box you're pretty sure that Mr. Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman.
The major difference between RabbitMQ and the post office is the fact that it doesn't deal with paper, instead it accepts, stores and forwards binary blobs of data ‒ messages.
Here's a post describing how to use it on an Rpi.