fbox is an easy to use and maintain distributed filesystem written in Go using Reed Solomon Erasure with FUSE and S3 interfaces and a modern Web UI https://files.mills.io/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
James Mills 407f1a6228
Upgrade to bitcask v1.0.0
3 months ago
.chglog Fix import paths 3 months ago
.github Refactor shard mapping (#4) 4 months ago
blob Fix import paths 3 months ago
data Add /status and /cluster endpoints (#15) 3 months ago
docs Update screenshot 4 months ago
store Fix bad log.Error call 3 months ago
tools Add some project infra borrowed from other projects 4 months ago
ui Cursor and Close (#14) 4 months ago
.dockerignore Add Dockerfile to build Docker images 4 months ago
.drone.yml Add CI triggers 3 months ago
.gitignore Refactor shard mapping (#4) 4 months ago
.goreleaser.yml Add some project infra borrowed from other projects 4 months ago
AUTHORS Updated README, Added Roadmap, AUTHORS and updated Contributing guideline 4 months ago
CHANGELOG.md Fix import paths 3 months ago
CODE_OF_CONDUCT.md Add contributing guidelines and code of conduct 4 months ago
CONTRIBUTING.md Updated README, Added Roadmap, AUTHORS and updated Contributing guideline 4 months ago
Dockerfile Use local images for Docker builds 3 months ago
LICENSE Add a README and LICENSE 4 months ago
Makefile Fix embedding of version and commit hash 4 months ago
README.md Fix Homebrew isntructions 3 months ago
Roadmap.md Updated README, Added Roadmap, AUTHORS and updated Contributing guideline 4 months ago
_config.yml Set theme jekyll-theme-architect 4 months ago
commands.go Add endpoint for deleting files (shards and metdata) as well as a sub-command (#8) 4 months ago
docker-compose.yml Add a working docker-compose.yml 4 months ago
fbox.yml Auto update the deploy docker stack services 4 months ago
go.mod Upgrade to bitcask v1.0.0 3 months ago
go.sum Upgrade to bitcask v1.0.0 3 months ago
handlers.go Fix import paths 3 months ago
main.go Use git.mills.io/prologic/useragent fork 3 months ago
metadata.go Fix import paths 3 months ago
node.go Add endpoint for deleting files (shards and metdata) as well as a sub-command (#8) 4 months ago
start-cluster.sh Fixed dev UI dev workflow 4 months ago
test.sh Optimze getShards() 4 months ago
utils.go Optimze getShards() 4 months ago
version.go Initial Commit 5 months ago

README.md

fbox - Files in a Box

Build Status Go Report Card GoDoc

fbox is a distributed file system written in Go that has the the following features:

Current Features:

  • Single portable binary
  • Simple to setup and maintain
  • A Web Interface with Drag 'n Drop
  • Data redundancy with erasure coding and sharding

Planned Features:

  • POSIX compatible FUSE interface
  • S3 Object Storage interface
  • Docker Volume Driver

There is also a publicly (freely) available demo instance available at:

NOTE: I, James Mills, run this instance on pretty cheap hardware on a limited budget. Please use it fairly so everyone can enjoy using it equally!

Sponsor this project to support the development of new features, improving existings ones and fix bugs!

Screenshot

Table of Contents

Created by gh-md-toc

Why?

fbox was written primarily and firstly as en education exercise in distributed file systems and a lot of time add effort went into understanding how many distributed file systems in the wild are designed.

Most other distributed file systems are either complex and hard to set up and maintain or come with really expensive license feeds. Therefore fbox is also an attempt at designing and implementing a distributed file system that is free and open source with the reliability, scale and durability of other distributed file systems.

Getting Started

Install from Releases

You can install fbox by simply downloading the latest version from the Release page for your platform and placing the binary in your $PATH.

Install from Homebrew

On macOS you can install fbox using Homebrew:

brew tap prologic/fbox https://git.mills.io/prologic/homebrew-fbox.git
brew install fbox

Install from Source

To install fbox from source you can run go get directly if you have a Go environment setup:

go get git.mills.io/prologic/fbox

NOTE: Be sure to have $GOBIN (if not empty) or your $GOPATH/bin in your $PATH. See Compile and install packages and dependencies

Or grab the source code and build:

git clone https://git.mills.io/prologic/fbox.git
cd fbox
make build

And optionally run make install to place the binary fbox in your $GOBIN or $GOPATH/bin (again see note above).

Usage

Prepare storage

mkdir data1 data2 data3

Start a master node

fbox -a 127.0.0.1:8000 -b :8000 -d ./data1

Start some data nodes

fbox -a 127.0.0.1:8001 -b :8001 -d ./data2 -m http://127.0.0.1:8000
fbox -a 127.0.0.1:8002 -b :8002 -d ./data3 -m http://127.0.0.1:8000

This will set up a 3-node cluster on your local machine (for testing only).

You can open the Web UI by navigating to:

http://127.0.0.1:8000/ui/

You can also use the command-line client fbox itself; See usage:

$ ./fbox --help
Usage: fbox [options] [command [arguments]]

fbox is a simple distributed file system...

Valid commands:
 - cat <name> -- Downloads the given file given by <name> to stdout
 - put <name> -- Upload the given file given by <name>

Valid options:
  -a, --advertise-addr string   [interface]:port to advertise
  -b, --bind string             [interface]:port to bind to (default "0.0.0.0:8000")
  -d, --data string             path to store data in (default "./data")
  -s, --data-shards int         no. of data shards (default 3)
  -D, --debug                   enable debug logging
  -m, --master string           address:port of master to join (empty if master)
  -p, --parity-shards int       no. of parity shards (default 1)
  -u, --store string            blob store uri (default "disk://data")
  -V, --version                 display version information and exit
pflag: help requested

For example to store a file:

echo "Hello World" > hello.txt
fbox -m http://127.0.0.1:8000 put hello.txt

And to retrieve the file:

fbox -m http://127.0.0.1:8000 cat hello.txt

Production Deployments

Docker Swarm

You can deploy fbox to a Docker Swarm cluster by utilising the provided fbox.yml Docker Stack. This also depends on and uses the Traefik ingress load balancer so you must also have that configured and running in your cluster appropriately.

export DOMAIN=files.yourdomain.tld
docker stack deploy -c fbox.yml fbox

In the News

Sponsor

Support the ongoing development of fbox!

Sponsor

Contributing

Interested in contributing to this project? You are welcome! Here are some ways you can contribute:

  • File an Issue -- For a bug, or interesting idea you have for a new feature or just general questions.
  • Submit a Pull-Request or two! We welcome all PR(s) that improve the project!

Please see the Contributing Guidelines and checkout the Roadmap for inspiration, ideas and the general vision and goals.

Contributors

Thank you to all those that have contributed to this project, battle-tested it, used it in their own projects or products, fixed bugs, improved performance and even fix tiny typos in documentation! Thank you and keep contributing!

You can find an AUTHORS file where we keep a list of contributors to the project. If you contribute a PR please consider adding your name there. There is also Github's own Contributors statistics.

Stargazers over time

Stargazers over time

Attributions and Prior Art

  • nicolagi/dino -- Some code was borrowed and much inspiration for metadata and FUSE interface design with performance in mind.
  • seaweedfs -- Some inspiration taken from design and access interfaces (e.g: S3).
  • GlusterFS, Portworx, Storidge, StorageOS, MooseFS -- All distributed filesystems evalauted, trialed in anger and some used to drive design goals.

License

fbox is licensed under the terms of the MIT License