Container storage solutions

  • ClusterHQ Flocker. Note that ClusterHQ, the company behind Flocker, shut down operations in 12/2016 and the status of Flocker, as an open source project, is ATM unknown.
  • EMC REX-ray
  • libStorage

Message queues

See queues.io.

Object storage

  • Amazon S3 (managed)
  • Azure Storage (managed)
  • Google Cloud Storage (managed)
  • LeoFS is an unstructured, HA object/data storage system incl. S3 and NFS support.
  • minio is an open source alternative to S3 that you can deploy where and how you want. See also the Minio client mc.

Time series datastores

  • Geras supports some IoT formats/protocols such as MQTT out-of-the-box. The (free) on-line version can be used without limitations, however comes without any SLAs and an on-premise version is available.
  • InfluxDB: Written in Go this open-source distributed time series database has no external dependencies. Has a powerful SQL-like query language, leveraging regexp and allowing continuos queries, and supporting fan-out as well as a number of aggregate functions such as PERCENTILE. It comes with a Raft-based cluster mode.
  • KairosDB: A distributed scalable time series database written on top of Cassandra.
  • kdb+: Comes with own query language offering temporal primitives and execution control and R integration.
  • OpenTSDB: A mature and established player implemented on top of Apache HBase. Scales to billions of data points and ingest rates up to the 100,000 million data points per second.
  • Prometheus: An open-source service monitoring system and time series database by the Soundcloud folks. Prometheus works well for recording any purely numeric time series. It fits both machine-centric monitoring as well as monitoring of highly dynamic service-oriented architectures.

Tooling

  • boto, a Python interface to Amazon Web Services
  • lsyncd synchronizes local directories with remote targets
  • rsync to sync across (remote) filesystems
  • rclone is a command line program to sync files and directories to and from pretty much everything under the sun.
  • scp to copy to/from remote filesystems

You might also want to check out How to transfer large amounts of data via network, providing some good advices around efficiency, latency, and security.