Research distributed file storage systems on linux on Azure
€30-250 EUR
Closed
Posted over 3 years ago
€30-250 EUR
Paid on delivery
I want to research a good option for (almost) infinitely scalable file storage system that is fast to use by many concurrent nodes. Nodes are running Ubuntu 20.04. File system needs to be highly fault tolerant and it should handle automatic backups by configuration.
System needs to support basic POSIX file permissions.
There are two options:
1. Use managed (remote) service such as Azure Files (SMB). Even within Azure network the latencies are too high for high performance applications. This option would need a local RAM cache, which I believe is supported already by cachefilesd.
2. Manage the cluster by yourself using open-souce software such as:
- MooseFS (free version has a single point of failure)
- BeeGFS
- LizardFS
- Ceph
- Quobyte
Requirements are:
- POSIX file permissions
- High availability
- Fault tolerant
- Built-in backups via fault tolerance or a tool to use
- High amount of concurrent reads, almost none concurrent writes
What I need to research the best option based on which is easier to setup/manager, performance and especially high availability and fault tolerance. Cluster should be able to handle random node crashes.
I want benchmarks for each good option from you. I also want benchmarks against standard HDD and SDD to get a baseline.
Hi,
I've read your requirements and I have proficiency in building Ceph cluster on RBD, CephFS and Ceph RGW use case.
From your case, I think you need CephFS and I can help you set it up. Hit me up on PM and let's discuss.
Regards.