Commit graph

18 commits

Author SHA1 Message Date
Moinak Ghosh
5f41057f9c Add config script to generate Makefile based on flags.
Add install target and installation readme file.
A few comment changes.
2012-08-26 14:09:24 +05:30
Moinak Ghosh
bf149e880d Add LZP Pre-Compression support ported from libbsc.
Add generic pre-processing wrappers for future support of other pre-processors.
Clean up computation of Rabin block sizes.
Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 22:58:44 +05:30
Moinak Ghosh
9eac774eb1 Add multithreaded LZMA port from p7zip
Compute balanced thread count between chunk threads and algo threads
Generic way to handle querying algorithm parameters
Clean up unnecessary includes
2012-08-18 10:20:52 +05:30
Moinak Ghosh
2dadf411fa Reduce memory consumption and improve performance in dedupe
Re-introduce crc64 for dedup blocks to avoid wasted memcpy-s
Restructe block array to be an array of pointers allocated on demand
Fix a corner case issue when splitting chunks at a dedup boundary
2012-08-17 11:03:02 +05:30
Moinak Ghosh
55d0485d34 Improve Rabin computations using an irreducible polynomial
Slight improvement to similarity computation
A simple mechanism to include DEBUG mode stats
Include stdint for common int types
2012-08-15 20:13:40 +05:30
Moinak Ghosh
f2ffcad2fd Compute and compare Mean sketch cksum to improve similarity comparison
Fix optflags settings in Makefile
Small optimization in zero RLE encoder to avoid scanning during lookahead
Some minor fixes
2012-08-09 23:57:24 +05:30
Moinak Ghosh
400d0bfa72 Bias fingerprint value with occurrence counts for a better sketch
Fix latent bug when calling algo deinit in decompression code
Reduce diff threshold for slightly greater delta encoding
Limit similar buffer size difference for less wasted diffing
Change zlib compression wrapper to use faster deflateReset mechanism
Reduce optimization level for Dedupe code, it goes faster
2012-08-08 22:40:58 +05:30
Moinak Ghosh
a4311f2ede Fix handling of incompressible chunks.
Fix handling of various dedup failures.
Add NULL compression option for dedup only compression.
2012-08-05 22:35:51 +05:30
Moinak Ghosh
296e2ab6b2 Add support for LZ4 compression including multi-pass LZ4.
Add missing Read_Adjusted() declaration, was causing a crash with 2GB chunks.
Fix minor cut-paste issues in comments.
2012-07-25 21:07:36 +05:30
Moinak Ghosh
8cfd54fe34 Add LZFX Compression support, a very fast lightweight compressor.
Avoid a branch in the rabin loop.
2012-07-23 00:15:08 +05:30
Moinak Ghosh
e788eb43b8 Implement Delta Encoding based on modified bsdiff.
Change to more accurate Sketch value computation approach.
2012-07-19 21:41:07 +05:30
Moinak Ghosh
d3f5287ee5 Update License info to LGPLv3. 2012-07-07 22:18:29 +05:30
Moinak Ghosh
a1825a2305 Implement Parallel deduplication support.
Restructure compression functions to take chunk flag as argument.
Add missing error flag printing in LZMA.
Only create enough threads as needed by chunk size and file size.
Minor cleanups and variable name changes.
2012-07-01 21:44:02 +05:30
Moinak Ghosh
cbf9728278 Implement Deduplication based on Rabin Fingerprinting: work in progress.
Fix bug that prevented pipe mode from being used.
Allow building without specialized allocator.
Use basic optimize flag in debuig build.
2012-06-29 18:23:55 +05:30
Moinak Ghosh
8f5f531967 Add license and other minor fixes. 2012-06-21 20:40:43 +05:30
Moinak Ghosh
733923cbf2 Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression.
Remove unnecessary checks in compression loop.
2012-06-21 20:27:05 +05:30
Moinak Ghosh
2eaf151ca0 New capability in allocator to add slab caches with user-specified size.
Use specific slab caches in main program and Lzma to slightly reduce memory footprint.
Fix missing hashtable counter update in allocator.
Slight cleanup of repeated computation with macro in Lzma.
2012-05-31 13:06:40 +05:30
Moinak Ghosh
07dfed7769 Initial Commit. 2012-05-28 20:19:29 +05:30