A high-level Reed Solomon erasure encoding library and set of command-line tools written in Go.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
James Mills 2d547770d2
Fix import paths
5 months ago
cmd Fix import paths 5 months ago
testdata Initial Commit 11 months ago
.gitignore Add .DS_Store to .gitignore 11 months ago
LICENSE Add README and LICENSE 11 months ago
README.md Fix import paths 5 months ago
_config.yml Set theme jekyll-theme-architect 11 months ago
encoder.go Add missing encoder 6 months ago
go.mod Fix import paths 5 months ago
go.sum Add missing encoder 6 months ago
version.go Initial Commit 11 months ago

README.md

reedlib

A high-level Reed Solomon erasure encoding library and set of command-line tools written in Go.

Quick Start

Installing the library

Either run:

go get git.mills.io/prologic/reedlib

Or just import the library in your project:

import "git.mills.io/prologic/reedlib"

NOTE: There isn't much of a library yet here at the time of writing this (2020-12-27). Coming soon...

Installing the command-line tools

Install reed-encode:

go get git.mills.io/prologic/reedlib/cmd/reed-encode/....

Install reed-decode:

go get git.mills.io/prologic/reedlib/cmd/reed-decode/....

Usage

Encoding a file (command-line)

To encode a file using Reed-Solomon erasure encoding using the default Data and Parity Shards (3 + 1 respectively):

reed-encode ./testdata/IMG_7895.JPG

This will result in a number of output files from the result of splitting up and encoding the original input file using the specified number of data and parity shards:

$ ls -lah ./testdata/
total 6.4M
drwxr-xr-x  8 prologic staff  256 Dec 27 09:10 .
drwxr-xr-x 17 prologic staff  544 Dec 27 09:12 ..
-rw-r--r--  1 prologic staff 6.1K Dec 27 09:10 .DS_Store
-rw-r--r--  1 prologic staff 2.7M Dec 27 09:10 IMG_7895.JPG
-rw-r--r--  1 prologic staff 916K Dec 27 09:10 IMG_7895.JPG.0
-rw-r--r--  1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.1
-rw-r--r--  1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.2
-rw-r--r--  1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.3

Keep the pieces (data and parity shards) on different storage devices or storage nodes which can then later be used to reconstruct the original file, even if one of the shards is lost or corrupt (parity of one).

Decoding shards (command-line)

To decode a number of shards from a previous encoding using the default Data and Parity Shards (3 + 1 respectively):

First remove the original input file to demonstrate recovery:

rm -f ./testdata/IMG_7895.JPG

Now reconstruct the original input file using the shards:

reed-decode ./testdata/IMG_7895.JPG

You should now have the original file recovered and intact:

$ ls -lah ./testdata/
total 6.3M
drwxr-xr-x  7 prologic staff  224 Dec 27 09:33 .
drwxr-xr-x 15 prologic staff  480 Dec 27 09:33 ..
-rw-r--r--  1 prologic staff 2.7M Dec 27 09:33 IMG_7895.JPG
-rw-r--r--  1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.0
-rw-r--r--  1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.1
-rw-r--r--  1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.2
-rw-r--r--  1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.3

You can even remove the original input file and either remove or corrupt one of the shards and the original input file is still recoverable from the remaining shards.

Notes

This is a high-level wrapper library and set of command-line tools that uses @klauspost's reedsolomon library which itself is a Go port of the JavaReedSolomon library released by Backblaze, with some additional optimizations.

For an introduction on erasure coding, see the post on the Backblaze blog.

Some interesting properties to note:

  • The number of data + parity shards MUST NOT exceed 256 as the default bit-field size is 8 for Reed Solomon erasure codes.
  • You can detect errors or recover from errors from up to parity shards. For example, with a data/parity of 3+1 you can recover from 1 lost or corrupt shard. With a data/parity of 3+2 you can recover from up to two shard failures.

License

reedlib is licensed under the terms of the MIT License