SeaweedFS is a simple and highly-scalable distributed file system that is an independent Apache-licensed open source project.
SeaweedFS started as an Object Store to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages file volumes, and it lets these volume servers manage files and their metadata. The project can transparently integrate with the cloud.
“With hot data on local cluster, and warm data on the cloud with O(1) access time, SeaweedFS can achieve both fast local access time and elastic cloud storage capacity, without any client side changes,” the working group behind the project wrote in a post.
SeaweedFS started by implementing Facebook’s Haystack design paper and implements erasure coding with ideas from f4: Facebook’s Warm BLOB Storage System.
Also, the Filer can support directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata stores, MySQL, Postgres, MongoDB, Redis, Cassandra, Elastic Search, LevelDB, MemSQL, TiDB, etcd, CockroachDB, and more.
Additional features include the ability to choose no replication or different replication levels, rack and data center aware, as well as automatic master server failover, automatic GZip compression depending on file mime type, and other automated features and support.
