Container storage solutions
- ClusterHQ Flocker. Note that ClusterHQ, the company behind Flocker, shut down operations in 12/2016 and the status of Flocker, as an open source project, is ATM unknown.
- EMC REX-ray
- Amazon S3 (managed)
- Azure Storage (managed)
- Google Cloud Storage (managed)
- LeoFS is an unstructured, HA object/data storage system incl. S3 and NFS support.
- minio is an open source alternative to S3 that you can deploy where and how you want. See also the Minio client mc.
Time series datastores
- Geras supports some IoT formats/protocols such as MQTT out-of-the-box. The (free) on-line version can be used without limitations, however comes without any SLAs and an on-premise version is available.
- InfluxDB: Written in Go this open-source distributed time series database has no external dependencies. Has a powerful SQL-like query language, leveraging regexp and allowing continuos queries, and supporting fan-out as well as a number of aggregate functions such as
PERCENTILE. It comes with a Raft-based cluster mode.
- KairosDB: A distributed scalable time series database written on top of Cassandra.
- kdb+: Comes with own query language offering temporal primitives and execution control and R integration.
- OpenTSDB: A mature and established player implemented on top of Apache HBase. Scales to billions of data points and ingest rates up to the 100,000 million data points per second.
- Prometheus: An open-source service monitoring system and time series database by the Soundcloud folks. Prometheus works well for recording any purely numeric time series. It fits both machine-centric monitoring as well as monitoring of highly dynamic service-oriented architectures.
- boto, a Python interface to Amazon Web Services
- lsyncd synchronizes local directories with remote targets
- rsync to sync across (remote) filesystems
- rclone is a command line program to sync files and directories to and from pretty much everything under the sun.
- scp to copy to/from remote filesystems
You might also want to check out How to transfer large amounts of data via network, providing some good advices around efficiency, latency, and security.