fbox is an easy to use and maintain distributed filesystem written in Go using Reed Solomon Erasure with FUSE and S3 interfaces and a modern Web UI https://files.mills.io/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
James Mills edb713b615
Fix link to Go Reference in README
2 weeks ago
.chglog Fix import paths 1 year ago
.github Refactor shard mapping (#4) 1 year ago
blob Fix import paths 1 year ago
data Add /status and /cluster endpoints (#15) 1 year ago
docs Update screenshot 1 year ago
store Fix bad log.Error call 1 year ago
tools Add some project infra borrowed from other projects 1 year ago
ui Cursor and Close (#14) 1 year ago
.dockerignore Add Dockerfile to build Docker images 1 year ago
.drone.yml Add CI triggers 1 year ago
.gitignore Refactor shard mapping (#4) 1 year ago
.goreleaser.yml Add some project infra borrowed from other projects 1 year ago
AUTHORS Updated README, Added Roadmap, AUTHORS and updated Contributing guideline 1 year ago
CHANGELOG.md Fix import paths 1 year ago
CODE_OF_CONDUCT.md Add contributing guidelines and code of conduct 1 year ago
CONTRIBUTING.md Updated README, Added Roadmap, AUTHORS and updated Contributing guideline 1 year ago
Dockerfile Use local images for Docker builds 1 year ago
LICENSE Add a README and LICENSE 1 year ago
Makefile Fix embedding of version and commit hash 1 year ago
README.md Fix link to Go Reference in README 2 weeks ago
Roadmap.md Updated README, Added Roadmap, AUTHORS and updated Contributing guideline 1 year ago
_config.yml Set theme jekyll-theme-architect 1 year ago
commands.go Add endpoint for deleting files (shards and metdata) as well as a sub-command (#8) 1 year ago
docker-compose.yml Add a working docker-compose.yml 1 year ago
fbox.yml Auto update the deploy docker stack services 1 year ago
go.mod Upgrade to bitcask v1.0.0 1 year ago
go.sum Upgrade to bitcask v1.0.0 1 year ago
handlers.go Fix import paths 1 year ago
main.go Use git.mills.io/prologic/useragent fork 1 year ago
metadata.go Fix import paths 1 year ago
node.go Add endpoint for deleting files (shards and metdata) as well as a sub-command (#8) 1 year ago
start-cluster.sh Fixed dev UI dev workflow 1 year ago
test.sh Optimze getShards() 1 year ago
utils.go Optimze getShards() 1 year ago
version.go Initial Commit 1 year ago

README.md

fbox - Files in a Box

Build Status Go Report Card Go Reference

fbox is a distributed file system written in Go that has the the following features:

Current Features:

  • Single portable binary
  • Simple to setup and maintain
  • A Web Interface with Drag 'n Drop
  • Data redundancy with erasure coding and sharding

Planned Features:

  • POSIX compatible FUSE interface
  • S3 Object Storage interface
  • Docker Volume Driver

There is also a publicly (freely) available demo instance available at:

NOTE: I, James Mills, run this instance on pretty cheap hardware on a limited budget. Please use it fairly so everyone can enjoy using it equally!

Sponsor this project to support the development of new features, improving existings ones and fix bugs!

Screenshot

Table of Contents

Created by gh-md-toc

Why?

fbox was written primarily and firstly as en education exercise in distributed file systems and a lot of time add effort went into understanding how many distributed file systems in the wild are designed.

Most other distributed file systems are either complex and hard to set up and maintain or come with really expensive license feeds. Therefore fbox is also an attempt at designing and implementing a distributed file system that is free and open source with the reliability, scale and durability of other distributed file systems.

Getting Started

Install from Releases

You can install fbox by simply downloading the latest version from the Release page for your platform and placing the binary in your $PATH.

Install from Homebrew

On macOS you can install fbox using Homebrew:

brew tap prologic/fbox https://git.mills.io/prologic/homebrew-fbox.git
brew install fbox

Install from Source

To install fbox from source you can run go get directly if you have a Go environment setup:

go get git.mills.io/prologic/fbox

NOTE: Be sure to have $GOBIN (if not empty) or your $GOPATH/bin in your $PATH. See Compile and install packages and dependencies

Or grab the source code and build:

git clone https://git.mills.io/prologic/fbox.git
cd fbox
make build

And optionally run make install to place the binary fbox in your $GOBIN or $GOPATH/bin (again see note above).

Usage

Prepare storage

mkdir data1 data2 data3

Start a master node

fbox -a 127.0.0.1:8000 -b :8000 -d ./data1

Start some data nodes

fbox -a 127.0.0.1:8001 -b :8001 -d ./data2 -m http://127.0.0.1:8000
fbox -a 127.0.0.1:8002 -b :8002 -d ./data3 -m http://127.0.0.1:8000

This will set up a 3-node cluster on your local machine (for testing only).

You can open the Web UI by navigating to:

http://127.0.0.1:8000/ui/

You can also use the command-line client fbox itself; See usage:

$ ./fbox --help
Usage: fbox [options] [command [arguments]]

fbox is a simple distributed file system...

Valid commands:
 - cat <name> -- Downloads the given file given by <name> to stdout
 - put <name> -- Upload the given file given by <name>

Valid options:
  -a, --advertise-addr string   [interface]:port to advertise
  -b, --bind string             [interface]:port to bind to (default "0.0.0.0:8000")
  -d, --data string             path to store data in (default "./data")
  -s, --data-shards int         no. of data shards (default 3)
  -D, --debug                   enable debug logging
  -m, --master string           address:port of master to join (empty if master)
  -p, --parity-shards int       no. of parity shards (default 1)
  -u, --store string            blob store uri (default "disk://data")
  -V, --version                 display version information and exit
pflag: help requested

For example to store a file:

echo "Hello World" > hello.txt
fbox -m http://127.0.0.1:8000 put hello.txt

And to retrieve the file:

fbox -m http://127.0.0.1:8000 cat hello.txt

Production Deployments

Docker Swarm

You can deploy fbox to a Docker Swarm cluster by utilising the provided fbox.yml Docker Stack. This also depends on and uses the Traefik ingress load balancer so you must also have that configured and running in your cluster appropriately.

export DOMAIN=files.yourdomain.tld
docker stack deploy -c fbox.yml fbox

In the News

Sponsor

Support the ongoing development of fbox!

Sponsor

Contributing

Interested in contributing to this project? You are welcome! Here are some ways you can contribute:

  • File an Issue -- For a bug, or interesting idea you have for a new feature or just general questions.
  • Submit a Pull-Request or two! We welcome all PR(s) that improve the project!

Please see the Contributing Guidelines and checkout the Roadmap for inspiration, ideas and the general vision and goals.

Contributors

Thank you to all those that have contributed to this project, battle-tested it, used it in their own projects or products, fixed bugs, improved performance and even fix tiny typos in documentation! Thank you and keep contributing!

You can find an AUTHORS file where we keep a list of contributors to the project. If you contribute a PR please consider adding your name there. There is also Github's own Contributors statistics.

Stargazers over time

Stargazers over time

Attributions and Prior Art

  • nicolagi/dino -- Some code was borrowed and much inspiration for metadata and FUSE interface design with performance in mind.
  • seaweedfs -- Some inspiration taken from design and access interfaces (e.g: S3).
  • GlusterFS, Portworx, Storidge, StorageOS, MooseFS -- All distributed filesystems evalauted, trialed in anger and some used to drive design goals.

License

fbox is licensed under the terms of the MIT License