Commit graph

11 commits

Author SHA1 Message Date
Moinak Ghosh
fa78621cbf Cleanup pointer casting in code to use macros. 2013-09-22 20:11:15 +05:30
Moinak Ghosh
12a2b8ed63 Additional error checks in RLE encoding for bsdiff extra data.
Add a buffer overflow check in RLE encoder.
Avoid calling RLE encoding if extra data length is zero.
Make 2KB block size default for non-global deduplication.
Update test cases for new 2KB block size support.
2013-08-30 19:51:43 +05:30
Moinak Ghosh
f8f23e5200 Major License text cleanup. 2013-03-07 20:26:48 +05:30
Moinak Ghosh
3888c8d316 Many optimization tweaks
Optimize Rabin Deduplication and Bsdiff
Vectorize XXHash using SE4
2013-01-20 22:02:26 +05:30
Moinak Ghosh
d49a088eea Fixes and performance improvements for Dedupe Delta Compression
Avoid using fingerprints in minhash computation and fix write amplification
Modify min-heap to use 64bit values
Improve bsdiff performance
Fix pointer comparison in bsdiff
Use 32bit offsets in bsdiff to reduce memory usage
Improve Zero RLE Encoder performance
Add more buffer overflow checks in Zero RLE Decoder
2013-01-13 22:04:59 +05:30
Moinak Ghosh
26a4f42506 Introduce strict compiler flags and fix scores of warnings/issues.
Avoid different optimization flags for Dedupe sources.
Fix liberal mixing of uint64_t and int64_t (should all be uint64_t).
Fix corner case crash when decompressing.
2012-12-27 23:06:48 +05:30
Moinak Ghosh
f2ffcad2fd Compute and compare Mean sketch cksum to improve similarity comparison
Fix optflags settings in Makefile
Small optimization in zero RLE encoder to avoid scanning during lookahead
Some minor fixes
2012-08-09 23:57:24 +05:30
Moinak Ghosh
400d0bfa72 Bias fingerprint value with occurrence counts for a better sketch
Fix latent bug when calling algo deinit in decompression code
Reduce diff threshold for slightly greater delta encoding
Limit similar buffer size difference for less wasted diffing
Change zlib compression wrapper to use faster deflateReset mechanism
Reduce optimization level for Dedupe code, it goes faster
2012-08-08 22:40:58 +05:30
Moinak Ghosh
94563a7ecd Fix buffer size computation when allocating Rabin block array.
Reduce memory usage of Rabin block array.
Add an SSE optimization for bsdiff.
Move integer hashing function to utils file.
More updates to README.
2012-07-28 23:55:24 +05:30
Moinak Ghosh
fd7c7e9a65 Use 4-byte ints for header values instead of 8-byte size_t.
Use RLE on control data if it reduces the size.
Update some comments.
Use scratch space at end of data chunk, if available.
2012-07-20 20:53:46 +05:30
Moinak Ghosh
e788eb43b8 Implement Delta Encoding based on modified bsdiff.
Change to more accurate Sketch value computation approach.
2012-07-19 21:41:07 +05:30