custom high-performance encoder implementation #52

Merged
jsign merged 1 commits from newcodec into master 4 years ago
jsign commented 4 years ago (Migrated from github.com)

This is a continuation of #49

Now rebased to #51 (current master), here are the results now:

benchmark               old ns/op     new ns/op     delta
BenchmarkGet/128B-8     1342          935           -30.33%
BenchmarkGet/256B-8     1241          916           -26.19%
BenchmarkGet/512B-8     1502          1094          -27.16%
BenchmarkGet/1K-8       1856          1312          -29.31%
BenchmarkGet/2K-8       2554          1694          -33.67%
BenchmarkGet/4K-8       4084          2555          -37.44%
BenchmarkGet/8K-8       6694          3888          -41.92%
BenchmarkGet/16K-8      11409         6833          -40.11%
BenchmarkGet/32K-8      23304         13686         -41.27%
BenchmarkPut/128B-8     4509          4215          -6.52%
BenchmarkPut/256B-8     5815          5398          -7.17%
BenchmarkPut/1K-8       12526         11751         -6.19%
BenchmarkPut/2K-8       20990         19896         -5.21%
BenchmarkPut/4K-8       50118         36880         -26.41%
BenchmarkPut/8K-8       74180         93285         +25.75%
BenchmarkPut/16K-8      143040        149835        +4.75%
BenchmarkPut/32K-8      275379        274837        -0.20%
BenchmarkScan-8         2923          2896          -0.92%

benchmark               old MB/s     new MB/s     speedup
BenchmarkGet/128B-8     95.36        136.81       1.43x
BenchmarkGet/256B-8     206.13       279.46       1.36x
BenchmarkGet/512B-8     340.82       467.87       1.37x
BenchmarkGet/1K-8       551.65       780.14       1.41x
BenchmarkGet/2K-8       801.67       1208.85      1.51x
BenchmarkGet/4K-8       1002.76      1603.09      1.60x
BenchmarkGet/8K-8       1223.72      2106.49      1.72x
BenchmarkGet/16K-8      1435.96      2397.63      1.67x
BenchmarkGet/32K-8      1406.06      2394.13      1.70x
BenchmarkPut/128B-8     28.38        30.37        1.07x
BenchmarkPut/256B-8     44.02        47.42        1.08x
BenchmarkPut/1K-8       81.74        87.14        1.07x
BenchmarkPut/2K-8       97.57        102.93       1.05x
BenchmarkPut/4K-8       81.73        111.06       1.36x
BenchmarkPut/8K-8       110.43       87.82        0.80x
BenchmarkPut/16K-8      114.54       109.35       0.95x
BenchmarkPut/32K-8      118.99       119.23       1.00x

benchmark               old allocs     new allocs     delta
BenchmarkGet/128B-8     5              2              -60.00%
BenchmarkGet/256B-8     5              2              -60.00%
BenchmarkGet/512B-8     5              2              -60.00%
BenchmarkGet/1K-8       5              2              -60.00%
BenchmarkGet/2K-8       5              2              -60.00%
BenchmarkGet/4K-8       5              2              -60.00%
BenchmarkGet/8K-8       5              2              -60.00%
BenchmarkGet/16K-8      5              2              -60.00%
BenchmarkGet/32K-8      5              2              -60.00%
BenchmarkPut/128B-8     8              7              -12.50%
BenchmarkPut/256B-8     8              7              -12.50%
BenchmarkPut/1K-8       8              7              -12.50%
BenchmarkPut/2K-8       8              7              -12.50%
BenchmarkPut/4K-8       8              7              -12.50%
BenchmarkPut/8K-8       8              7              -12.50%
BenchmarkPut/16K-8      8              7              -12.50%
BenchmarkPut/32K-8      9              8              -11.11%
BenchmarkScan-8         35             35             +0.00%

benchmark               old bytes     new bytes     delta
BenchmarkGet/128B-8     400           168           -58.00%
BenchmarkGet/256B-8     656           296           -54.88%
BenchmarkGet/512B-8     1200          584           -51.33%
BenchmarkGet/1K-8       2288          1160          -49.30%
BenchmarkGet/2K-8       4464          2312          -48.21%
BenchmarkGet/4K-8       9072          4872          -46.30%
BenchmarkGet/8K-8       17776         9480          -46.67%
BenchmarkGet/16K-8      34928         18440         -47.21%
BenchmarkGet/32K-8      73840         40968         -44.52%
BenchmarkPut/128B-8     457           209           -54.27%
BenchmarkPut/256B-8     587           210           -64.22%
BenchmarkPut/1K-8       1459          218           -85.06%
BenchmarkPut/2K-8       2622          229           -91.27%
BenchmarkPut/4K-8       5203          249           -95.21%
BenchmarkPut/8K-8       9859          296           -97.00%
BenchmarkPut/16K-8      18897         372           -98.03%
BenchmarkPut/32K-8      41596         537           -98.71%
BenchmarkScan-8         424           424           +0.00%
This is a continuation of #49 Now rebased to #51 (current `master`), here are the results now: ``` benchmark old ns/op new ns/op delta BenchmarkGet/128B-8 1342 935 -30.33% BenchmarkGet/256B-8 1241 916 -26.19% BenchmarkGet/512B-8 1502 1094 -27.16% BenchmarkGet/1K-8 1856 1312 -29.31% BenchmarkGet/2K-8 2554 1694 -33.67% BenchmarkGet/4K-8 4084 2555 -37.44% BenchmarkGet/8K-8 6694 3888 -41.92% BenchmarkGet/16K-8 11409 6833 -40.11% BenchmarkGet/32K-8 23304 13686 -41.27% BenchmarkPut/128B-8 4509 4215 -6.52% BenchmarkPut/256B-8 5815 5398 -7.17% BenchmarkPut/1K-8 12526 11751 -6.19% BenchmarkPut/2K-8 20990 19896 -5.21% BenchmarkPut/4K-8 50118 36880 -26.41% BenchmarkPut/8K-8 74180 93285 +25.75% BenchmarkPut/16K-8 143040 149835 +4.75% BenchmarkPut/32K-8 275379 274837 -0.20% BenchmarkScan-8 2923 2896 -0.92% benchmark old MB/s new MB/s speedup BenchmarkGet/128B-8 95.36 136.81 1.43x BenchmarkGet/256B-8 206.13 279.46 1.36x BenchmarkGet/512B-8 340.82 467.87 1.37x BenchmarkGet/1K-8 551.65 780.14 1.41x BenchmarkGet/2K-8 801.67 1208.85 1.51x BenchmarkGet/4K-8 1002.76 1603.09 1.60x BenchmarkGet/8K-8 1223.72 2106.49 1.72x BenchmarkGet/16K-8 1435.96 2397.63 1.67x BenchmarkGet/32K-8 1406.06 2394.13 1.70x BenchmarkPut/128B-8 28.38 30.37 1.07x BenchmarkPut/256B-8 44.02 47.42 1.08x BenchmarkPut/1K-8 81.74 87.14 1.07x BenchmarkPut/2K-8 97.57 102.93 1.05x BenchmarkPut/4K-8 81.73 111.06 1.36x BenchmarkPut/8K-8 110.43 87.82 0.80x BenchmarkPut/16K-8 114.54 109.35 0.95x BenchmarkPut/32K-8 118.99 119.23 1.00x benchmark old allocs new allocs delta BenchmarkGet/128B-8 5 2 -60.00% BenchmarkGet/256B-8 5 2 -60.00% BenchmarkGet/512B-8 5 2 -60.00% BenchmarkGet/1K-8 5 2 -60.00% BenchmarkGet/2K-8 5 2 -60.00% BenchmarkGet/4K-8 5 2 -60.00% BenchmarkGet/8K-8 5 2 -60.00% BenchmarkGet/16K-8 5 2 -60.00% BenchmarkGet/32K-8 5 2 -60.00% BenchmarkPut/128B-8 8 7 -12.50% BenchmarkPut/256B-8 8 7 -12.50% BenchmarkPut/1K-8 8 7 -12.50% BenchmarkPut/2K-8 8 7 -12.50% BenchmarkPut/4K-8 8 7 -12.50% BenchmarkPut/8K-8 8 7 -12.50% BenchmarkPut/16K-8 8 7 -12.50% BenchmarkPut/32K-8 9 8 -11.11% BenchmarkScan-8 35 35 +0.00% benchmark old bytes new bytes delta BenchmarkGet/128B-8 400 168 -58.00% BenchmarkGet/256B-8 656 296 -54.88% BenchmarkGet/512B-8 1200 584 -51.33% BenchmarkGet/1K-8 2288 1160 -49.30% BenchmarkGet/2K-8 4464 2312 -48.21% BenchmarkGet/4K-8 9072 4872 -46.30% BenchmarkGet/8K-8 17776 9480 -46.67% BenchmarkGet/16K-8 34928 18440 -47.21% BenchmarkGet/32K-8 73840 40968 -44.52% BenchmarkPut/128B-8 457 209 -54.27% BenchmarkPut/256B-8 587 210 -64.22% BenchmarkPut/1K-8 1459 218 -85.06% BenchmarkPut/2K-8 2622 229 -91.27% BenchmarkPut/4K-8 5203 249 -95.21% BenchmarkPut/8K-8 9859 296 -97.00% BenchmarkPut/16K-8 18897 372 -98.03% BenchmarkPut/32K-8 41596 537 -98.71% BenchmarkScan-8 424 424 +0.00% ```
codecov[bot] commented 4 years ago (Migrated from github.com)

Codecov Report

Merging #52 into master will increase coverage by 1.25%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #52      +/-   ##
==========================================
+ Coverage   63.08%   64.33%   +1.25%     
==========================================
  Files           2        2              
  Lines         279      272       -7     
==========================================
- Hits          176      175       -1     
+ Misses         66       60       -6     
  Partials       37       37
Impacted Files Coverage Δ
options.go 95.34% <ø> (+11.67%) ⬆️
bitcask.go 58.51% <100%> (-0.19%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 755b187...77e9ee2. Read the comment docs.

# [Codecov](https://codecov.io/gh/prologic/bitcask/pull/52?src=pr&el=h1) Report > Merging [#52](https://codecov.io/gh/prologic/bitcask/pull/52?src=pr&el=desc) into [master](https://codecov.io/gh/prologic/bitcask/commit/755b1879b51b742672d541b0940fe8ccc0f9f7d3?src=pr&el=desc) will **increase** coverage by `1.25%`. > The diff coverage is `100%`. [![Impacted file tree graph](https://codecov.io/gh/prologic/bitcask/pull/52/graphs/tree.svg?width=650&token=6texGIVC5l&height=150&src=pr)](https://codecov.io/gh/prologic/bitcask/pull/52?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #52 +/- ## ========================================== + Coverage 63.08% 64.33% +1.25% ========================================== Files 2 2 Lines 279 272 -7 ========================================== - Hits 176 175 -1 + Misses 66 60 -6 Partials 37 37 ``` | [Impacted Files](https://codecov.io/gh/prologic/bitcask/pull/52?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [options.go](https://codecov.io/gh/prologic/bitcask/pull/52/diff?src=pr&el=tree#diff-b3B0aW9ucy5nbw==) | `95.34% <ø> (+11.67%)` | :arrow_up: | | [bitcask.go](https://codecov.io/gh/prologic/bitcask/pull/52/diff?src=pr&el=tree#diff-Yml0Y2Fzay5nbw==) | `58.51% <100%> (-0.19%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/prologic/bitcask/pull/52?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/prologic/bitcask/pull/52?src=pr&el=footer). Last update [755b187...77e9ee2](https://codecov.io/gh/prologic/bitcask/pull/52?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Merging since I already approved the previous version of this PR :)

Merging since I already approved the previous version of this PR :)

Just for comparison; on my work Macbook Pro 13":

goos: darwin
goarch: amd64
pkg: github.com/prologic/bitcask
BenchmarkGet/128B-4         	 1000000	      1623 ns/op	  78.86 MB/s	     168 B/op	       2 allocs/op
BenchmarkGet/256B-4         	 1000000	      1641 ns/op	 155.98 MB/s	     296 B/op	       2 allocs/op
BenchmarkGet/512B-4         	 1000000	      1930 ns/op	 265.27 MB/s	     584 B/op	       2 allocs/op
BenchmarkGet/1K-4           	  500000	      2090 ns/op	 489.88 MB/s	    1160 B/op	       2 allocs/op
BenchmarkGet/2K-4           	  500000	      2422 ns/op	 845.53 MB/s	    2312 B/op	       2 allocs/op
BenchmarkGet/4K-4           	  500000	      2892 ns/op	1416.17 MB/s	    4872 B/op	       2 allocs/op
BenchmarkGet/8K-4           	  300000	      3823 ns/op	2142.43 MB/s	    9480 B/op	       2 allocs/op
BenchmarkGet/16K-4          	  200000	      6210 ns/op	2638.03 MB/s	   18440 B/op	       2 allocs/op
BenchmarkGet/32K-4          	  200000	     10883 ns/op	3010.88 MB/s	   40968 B/op	       2 allocs/op
BenchmarkPut/128B-4         	  200000	      8291 ns/op	  15.44 MB/s	     209 B/op	       7 allocs/op
BenchmarkPut/256B-4         	  200000	     10002 ns/op	  25.59 MB/s	     210 B/op	       7 allocs/op
BenchmarkPut/1K-4           	  100000	     21394 ns/op	  47.86 MB/s	     218 B/op	       7 allocs/op
BenchmarkPut/2K-4           	   50000	     34374 ns/op	  59.58 MB/s	     229 B/op	       7 allocs/op
BenchmarkPut/4K-4           	   20000	     66137 ns/op	  61.93 MB/s	     252 B/op	       7 allocs/op
BenchmarkPut/8K-4           	   10000	    125546 ns/op	  65.25 MB/s	     289 B/op	       7 allocs/op
BenchmarkPut/16K-4          	   10000	    212448 ns/op	  77.12 MB/s	     373 B/op	       7 allocs/op
BenchmarkPut/32K-4          	    5000	    358053 ns/op	  91.52 MB/s	     538 B/op	       8 allocs/op
BenchmarkScan-4             	 1000000	      2341 ns/op	     424 B/op	      35 allocs/op
Just for comparison; on my work Macbook Pro 13": ``` goos: darwin goarch: amd64 pkg: github.com/prologic/bitcask BenchmarkGet/128B-4 1000000 1623 ns/op 78.86 MB/s 168 B/op 2 allocs/op BenchmarkGet/256B-4 1000000 1641 ns/op 155.98 MB/s 296 B/op 2 allocs/op BenchmarkGet/512B-4 1000000 1930 ns/op 265.27 MB/s 584 B/op 2 allocs/op BenchmarkGet/1K-4 500000 2090 ns/op 489.88 MB/s 1160 B/op 2 allocs/op BenchmarkGet/2K-4 500000 2422 ns/op 845.53 MB/s 2312 B/op 2 allocs/op BenchmarkGet/4K-4 500000 2892 ns/op 1416.17 MB/s 4872 B/op 2 allocs/op BenchmarkGet/8K-4 300000 3823 ns/op 2142.43 MB/s 9480 B/op 2 allocs/op BenchmarkGet/16K-4 200000 6210 ns/op 2638.03 MB/s 18440 B/op 2 allocs/op BenchmarkGet/32K-4 200000 10883 ns/op 3010.88 MB/s 40968 B/op 2 allocs/op BenchmarkPut/128B-4 200000 8291 ns/op 15.44 MB/s 209 B/op 7 allocs/op BenchmarkPut/256B-4 200000 10002 ns/op 25.59 MB/s 210 B/op 7 allocs/op BenchmarkPut/1K-4 100000 21394 ns/op 47.86 MB/s 218 B/op 7 allocs/op BenchmarkPut/2K-4 50000 34374 ns/op 59.58 MB/s 229 B/op 7 allocs/op BenchmarkPut/4K-4 20000 66137 ns/op 61.93 MB/s 252 B/op 7 allocs/op BenchmarkPut/8K-4 10000 125546 ns/op 65.25 MB/s 289 B/op 7 allocs/op BenchmarkPut/16K-4 10000 212448 ns/op 77.12 MB/s 373 B/op 7 allocs/op BenchmarkPut/32K-4 5000 358053 ns/op 91.52 MB/s 538 B/op 8 allocs/op BenchmarkScan-4 1000000 2341 ns/op 424 B/op 35 allocs/op ```

That's 600k/120k read/write per second right? 🤣

That's 600k/120k read/write per second right? 🤣
The pull request has been merged as fd179b4a86.
Sign in to join this conversation.
Loading…
There is no content yet.