greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
71 commits 3 branches 21 tags 7.6 MiB
Commit graph

22 commits

Author SHA1 Message Date
Moinak Ghosh
eda312ce1e Add support for Skein512 and Skein256 checksums 
Import Skein code from NIST CD submission
Make checksum algorithms pluggable
Fix handling of huge buffers (>2GB) in LZP
Cleanup of some buffer sizing code
Speed up CRC64 calculation in dedupe chunking
 2012-08-31 22:36:06 +05:30
Moinak Ghosh
f03834278a Bump version to 0.8. 
Update Changelog for 0.8 beta release.
 2012-08-27 22:53:39 +05:30
Moinak Ghosh
d75535bc7e Add support for libbsc a high-performance block sorting compressor. 
Enable external algorithm threading for single chunk compressed files.
Update docs.
 2012-08-27 21:51:55 +05:30
Moinak Ghosh
3b83bc2d4e Bump file version. 2012-08-26 15:01:18 +05:30
Moinak Ghosh
bf149e880d Add LZP Pre-Compression support ported from libbsc. 
Add generic pre-processing wrappers for future support of other pre-processors.
Clean up computation of Rabin block sizes.
Compute Rabin scratch space accurately to avoid RAM wastage.
 2012-08-23 22:58:44 +05:30
Moinak Ghosh
3851c9c6cc Delay allocation of per-thread chunks for performance and memory efficiency. 
Avoid allocating double-buffer for single-chunk files.
Introduce lzmaMt option to indicate multithreaded LZMA.
Update README.
 2012-08-18 22:00:14 +05:30
Moinak Ghosh
9eac774eb1 Add multithreaded LZMA port from p7zip 
Compute balanced thread count between chunk threads and algo threads
Generic way to handle querying algorithm parameters
Clean up unnecessary includes
 2012-08-18 10:20:52 +05:30
Moinak Ghosh
7b7007d6c5 Update version to 0.7 (0.6 was alpha). 2012-08-10 11:01:42 +05:30
Moinak Ghosh
bde917c8e9 Fix handling of compression flags in adaptive mode 
Fix error handling when chunk size is too small for dedupe
Bump version to 0.6
 2012-08-10 10:47:11 +05:30
Moinak Ghosh
400d0bfa72 Bias fingerprint value with occurrence counts for a better sketch 
Fix latent bug when calling algo deinit in decompression code
Reduce diff threshold for slightly greater delta encoding
Limit similar buffer size difference for less wasted diffing
Change zlib compression wrapper to use faster deflateReset mechanism
Reduce optimization level for Dedupe code, it goes faster
 2012-08-08 22:40:58 +05:30
Moinak Ghosh
a4311f2ede Fix handling of incompressible chunks. 
Fix handling of various dedup failures.
Add NULL compression option for dedup only compression.
 2012-08-05 22:35:51 +05:30
Moinak Ghosh
2cbcb0c9e4 Fix buffer sizing for LZ4. 
Fix exit condition checks in LZ4 decompression wrapper.
 2012-08-04 17:55:20 +05:30
Moinak Ghosh
bc71caffc3 Display release version in usage text. 2012-07-27 22:07:56 +05:30
Moinak Ghosh
c7cc7b469c Update chunk size computation to reduce memory usage. 
Implement runtime bypass of custom allocator.
Update README.
 2012-07-27 22:03:24 +05:30
Moinak Ghosh
296e2ab6b2 Add support for LZ4 compression including multi-pass LZ4. 
Add missing Read_Adjusted() declaration, was causing a crash with 2GB chunks.
Fix minor cut-paste issues in comments.
 2012-07-25 21:07:36 +05:30
Moinak Ghosh
53d4311534 Make LZFX Hash size dynamic. 
Use smaller min rabin block when using fast compression algos.
Add missing check for algo init function return value.
 2012-07-23 21:43:12 +05:30
Moinak Ghosh
8cfd54fe34 Add LZFX Compression support, a very fast lightweight compressor. 
Avoid a branch in the rabin loop.
 2012-07-23 00:15:08 +05:30
Moinak Ghosh
d3f5287ee5 Update License info to LGPLv3. 2012-07-07 22:18:29 +05:30
Moinak Ghosh
a1825a2305 Implement Parallel deduplication support. 
Restructure compression functions to take chunk flag as argument.
Add missing error flag printing in LZMA.
Only create enough threads as needed by chunk size and file size.
Minor cleanups and variable name changes.
 2012-07-01 21:44:02 +05:30
Moinak Ghosh
cbf9728278 Implement Deduplication based on Rabin Fingerprinting: work in progress. 
Fix bug that prevented pipe mode from being used.
Allow building without specialized allocator.
Use basic optimize flag in debuig build.
 2012-06-29 18:23:55 +05:30
Moinak Ghosh
e46d3d10b3 Add basic compression statistics and new option to display them. 2012-05-31 21:36:33 +05:30
Moinak Ghosh
07dfed7769 Initial Commit. 2012-05-28 20:19:29 +05:30