greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
152 commits 3 branches 21 tags 7.6 MiB
Commit graph

77 commits

Author SHA1 Message Date
Moinak Ghosh
063624d0d3 Fix Delta2 decode handling when not compressing. 
More debug statements and other minor changes.
 2012-12-24 23:21:02 +05:30
Moinak Ghosh
d597f0f05c Major enhancements to Delta2 encoding. 
Avoid transposing below-threshold spans. Reduces compression ratio.
Use little-endian storage format for numbers to optimize for x86.
Improve embedded table detection.
Reduce header sizes.
Get rid of Gcc's LTO flag. Causes a performance drop.
Fix preprocessing behavior when LZP does not compress but Delta2 works.
 2012-12-23 23:50:45 +05:30
Moinak Ghosh
a43fdd7d2c Improve Delta2 scanning speed and effectiveness. 
Add destination buffer overflow check in Delta2.
Add rough speed computation.
 2012-12-23 00:44:56 +05:30
Moinak Ghosh
c2df666daf Major improvement to Delta2 encoding algorithm. 2012-12-18 23:32:55 +05:30
Moinak Ghosh
fb30b5c295 Enable building with alternate Zlib and Bzlib. 
Update README and comments.
Fix correct setting of output size when using Delta2 without LZP.
 2012-12-16 23:17:04 +05:30
Moinak Ghosh
ef0191729e Make Delta2 encoding independent of LZP. 
Tweak Delta2 parameters.
Update README and test cases.
 2012-12-15 22:03:23 +05:30
Moinak Ghosh
a98778d62f Fine tune transpose parameters. 
Fix minor nits.
 2012-12-14 19:12:48 +05:30
Moinak Ghosh
b0f41c2888 Add matrix transpose to Delta2 encoding. 
Change confusing structure member name.
 2012-12-13 21:18:16 +05:30
Moinak Ghosh
375ebefa0d Add Matrix Transpose of Dedupe index to compress it better. 
Fix handling of Dedupe index compression failure.
 2012-12-13 00:00:47 +05:30
Moinak Ghosh
03840b31c5 Update to latest LZ4. 
Update a couple of comments.
 2012-12-11 11:38:42 +05:30
Moinak Ghosh
224fb529e9 Get rid of size_t in places where 64-bitness is assumed. 2012-12-09 10:15:06 +05:30
Moinak Ghosh
75c81b5e9c Add support for 64-bit Keccak implementation. 
Sanitize error message from tests.
Add more tests.
Improve platform detection in config script.
 2012-12-08 14:19:01 +05:30
Moinak Ghosh
29b0d8fd7b Implement Adaptive Delta Encoding. 2012-12-05 00:09:47 +05:30
Moinak Ghosh
51249c858d Work in progress Adaptive Delta Encoding. 2012-12-03 23:45:41 +05:30
Moinak Ghosh
deecbadac4 Fix thread error reporting. 
Update error condition tests to not truncate archive.
 2012-11-25 15:12:45 +05:30
Moinak Ghosh
d250322490 Fix issues with error handling. 
Add new tests for out of range values and corrupted file.
 2012-11-24 23:53:07 +05:30
Moinak Ghosh
bdf16c4cb9 Avoid CRC32 when decompressing older version archives. 2012-11-24 00:00:05 +05:30
Moinak Ghosh
817e16cce6 Use HMAC on header and encrypted data, avoid regular digest when encrypting. 2012-11-23 22:27:14 +05:30
Moinak Ghosh
6e9ca97bdf Add CRC32 header checksums in non-cryptographic mode. 2012-11-23 00:00:39 +05:30
Moinak Ghosh
d054e0f713 Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. 
Change some function signatures to improve algo init function behavior.
Fix corner case dedupe bug in error handling flow.
Bump archive version signature.
 2012-11-22 21:02:50 +05:30
Moinak Ghosh
5b1a13480b Fix chunk flag setup when compression fails in adaptive mode. 
Prevent display of non-fatal errors during compression.
 2012-11-09 21:07:23 +05:30
Moinak Ghosh
77db54e712 Add buffer overflow check in Ppmd compression routines. 
Fix pipe mode encryption check.
Change file difference check in tests.
Add more tests.
 2012-11-09 20:05:13 +05:30
Moinak Ghosh
da9083ae12 Fix encryption with adaptive modes. 
Add missing zero-out of algorithm data field.
 2012-11-09 18:53:48 +05:30
Moinak Ghosh
315bb50264 Change chunk HMAC to only check header, not full data. Digest in hdr int turn validates data. 2012-10-19 21:51:27 +05:30
Moinak Ghosh
1eb8c94fed Implement HMAC verification for chunk data and header. 
Do not ask for decryption passwd twice.
 2012-10-18 22:55:41 +05:30
Moinak Ghosh
d13aad830e Implement HMAC Functionality. 
Refactor crypto utility functions in a separate file.
Add HMAC verification for file header.
Add a few extra input validation checks.
Include space in chunk header for HMAC.
 2012-10-17 23:32:35 +05:30
Moinak Ghosh
f2d7bea902 Support for chunk-level encryption using AES and Scrypt based PBE. 
Couple of minor fixes.
 2012-10-15 12:10:00 +05:30
Moinak Ghosh
2578676b1b Add support for SHA256 and SHA512 digests from OpenSSL library. 2012-10-03 22:43:02 +05:30
Moinak Ghosh
8f8af7ed6b Update adaptive mode heuristic based on algorithms. 
Remove incorrect check in PPMd decompression code.
More refactoring of variable names.
 2012-09-27 22:29:08 +05:30
Moinak Ghosh
3544a8c708 Fix polynomial table computation. 
Change hashing and length bias to reduce hashtable bucket collisions.
Add support for user-selectable 60% or 40% similarity for Delta Compression.
Overall slight speedup.
 2012-09-24 22:20:27 +05:30
Moinak Ghosh
8386e72566 Rewrite core dedupe logic to simplify code and improve performance. 
Hashtable based chunk-level deduplication instead of Quicksort.
Fix a corner case bug in Dedupe decompression.
 2012-09-23 14:57:09 +05:30
Moinak Ghosh
99a8e4cd98 Speed up Hash computation for dedupe blocks. 
Add missing initialization of sliding window.
Update help text.
 2012-09-19 20:29:44 +05:30
Moinak Ghosh
e3befd9e16 Add support for Fixed-Block deduplication. 
More refactoring of symbol names.
 2012-09-16 11:12:58 +05:30
Moinak Ghosh
b9355a5dcc Reduce dedupe loop checks for slight speed edge. 
Beginnings of Fixed-block dedupe.
Update variable name for clarity.
 2012-09-15 11:14:58 +05:30
Moinak Ghosh
fb0aef0bd6 Fix return code handling in LZP pre-compression, crashed adaptive modes. 2012-09-07 19:31:35 +05:30
Moinak Ghosh
e6f042aaf8 Allow user-specified minimum Dedupe block size. 
Compute similarity sketch only if Delta Compression enabled.
 2012-09-05 22:43:54 +05:30
Moinak Ghosh
262566b59a Add xxHash for Rabin block checksums, slightly faster than CRC64. 
Fix missing initialization of character counts table.
Some file reorganization.
 2012-09-02 20:40:32 +05:30
Moinak Ghosh
4ba840b255 Add ASM version of Skein for x64 platforms with auto-detection 
Error checking for checksum flag when decompressing
Update comments and READMEs
 2012-09-01 14:40:15 +05:30
Moinak Ghosh
eda312ce1e Add support for Skein512 and Skein256 checksums 
Import Skein code from NIST CD submission
Make checksum algorithms pluggable
Fix handling of huge buffers (>2GB) in LZP
Cleanup of some buffer sizing code
Speed up CRC64 calculation in dedupe chunking
 2012-08-31 22:36:06 +05:30
Moinak Ghosh
a222772940 Fix single chunk flag handling during decompression. 
Update docs.
 2012-08-27 22:24:23 +05:30
Moinak Ghosh
d75535bc7e Add support for libbsc a high-performance block sorting compressor. 
Enable external algorithm threading for single chunk compressed files.
Update docs.
 2012-08-27 21:51:55 +05:30
Moinak Ghosh
3b83bc2d4e Bump file version. 2012-08-26 15:01:18 +05:30
Moinak Ghosh
d4e9cd0140 Improve memory efficiency when total file size < total size of chunks. 
Fix freeing of Zlib structures.
 2012-08-24 20:16:21 +05:30
Moinak Ghosh
bf149e880d Add LZP Pre-Compression support ported from libbsc. 
Add generic pre-processing wrappers for future support of other pre-processors.
Clean up computation of Rabin block sizes.
Compute Rabin scratch space accurately to avoid RAM wastage.
 2012-08-23 22:58:44 +05:30
Moinak Ghosh
3851c9c6cc Delay allocation of per-thread chunks for performance and memory efficiency. 
Avoid allocating double-buffer for single-chunk files.
Introduce lzmaMt option to indicate multithreaded LZMA.
Update README.
 2012-08-18 22:00:14 +05:30
Moinak Ghosh
9eac774eb1 Add multithreaded LZMA port from p7zip 
Compute balanced thread count between chunk threads and algo threads
Generic way to handle querying algorithm parameters
Clean up unnecessary includes
 2012-08-18 10:20:52 +05:30
Moinak Ghosh
bde917c8e9 Fix handling of compression flags in adaptive mode 
Fix error handling when chunk size is too small for dedupe
Bump version to 0.6
 2012-08-10 10:47:11 +05:30
Moinak Ghosh
400d0bfa72 Bias fingerprint value with occurrence counts for a better sketch 
Fix latent bug when calling algo deinit in decompression code
Reduce diff threshold for slightly greater delta encoding
Limit similar buffer size difference for less wasted diffing
Change zlib compression wrapper to use faster deflateReset mechanism
Reduce optimization level for Dedupe code, it goes faster
 2012-08-08 22:40:58 +05:30
Moinak Ghosh
a4311f2ede Fix handling of incompressible chunks. 
Fix handling of various dedup failures.
Add NULL compression option for dedup only compression.
 2012-08-05 22:35:51 +05:30
Moinak Ghosh
2cbcb0c9e4 Fix buffer sizing for LZ4. 
Fix exit condition checks in LZ4 decompression wrapper.
 2012-08-04 17:55:20 +05:30