greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
175 commits 3 branches 21 tags 7.6 MiB
Commit graph

87 commits

Author SHA1 Message Date
Moinak Ghosh
26bb137257 Changes for generalized runtime SSE/AVX/XOP detection. 
Multi instruction set XXhash build with runtime selection.
Extend CPUID code to detect more instruction sets.
Add options for BLAKE2 hash.
Move GCC builtins into utils header.
Bump file format version number due to extended digest flags.
Add descriptions to digest list.
 2013-01-25 00:10:12 +05:30
Moinak Ghosh
3888c8d316 Many optimization tweaks 
Optimize Rabin Deduplication and Bsdiff
Vectorize XXHash using SE4
 2013-01-20 22:02:26 +05:30
Moinak Ghosh
49ec3a054d Add SSE2 improvements to CTR mode AES. 
Add debug print of encryption and HMAC throughput.
Fix error message for invalid option.
 2013-01-16 19:52:46 +05:30
Moinak Ghosh
39dbc4be43 Implement algo-specific minimum distance match for Delta Compression. 2013-01-14 13:20:07 +05:30
Moinak Ghosh
a8fd60fb06 Fix issue #1 and issue #2. 
Enable building with older openssl (at least 0.9.8e).
Add variants of missing functions in older openssl versions.
Allow proper linking with libraries in alternate locations and setting RUNPATH.
Increase hash rounds for nonce generation.
 2013-01-05 00:16:15 +05:30
Moinak Ghosh
16b1d9e7a3 Bump version, update Changelog and documentation for 1.2 release. 2013-01-03 23:40:21 +05:30
Moinak Ghosh
47ebd5b752 Fix calculation of extra scratch space for Dedupe. 
Add missing return value check in small buffer Delta2.
Update test failure detection in test driver.
 2013-01-03 22:25:14 +05:30
Moinak Ghosh
d9eb82e0e8 Fix numeric parsing. 
Fix dedupe bug introduced in last commit.
Reset valid flag when resetting dedupe context.
Cleanup test suites.
Do not abort test suite on failure of a test case.
 2013-01-03 00:27:18 +05:30
Moinak Ghosh
6b756eb165 Fix Delta2 handling for small buffers. 
Fix LZP handling during preprocessing.
Fix type flag handling during preprocessing.
Update test cases to use configurable list of test corpus files.
Update some more int64_t datatypes to uint64_t
Add a gitignore.
 2013-01-02 22:56:21 +05:30
Moinak Ghosh
26a4f42506 Introduce strict compiler flags and fix scores of warnings/issues. 
Avoid different optimization flags for Dedupe sources.
Fix liberal mixing of uint64_t and int64_t (should all be uint64_t).
Fix corner case crash when decompressing.
 2012-12-27 23:06:48 +05:30
Moinak Ghosh
063624d0d3 Fix Delta2 decode handling when not compressing. 
More debug statements and other minor changes.
 2012-12-24 23:21:02 +05:30
Moinak Ghosh
d597f0f05c Major enhancements to Delta2 encoding. 
Avoid transposing below-threshold spans. Reduces compression ratio.
Use little-endian storage format for numbers to optimize for x86.
Improve embedded table detection.
Reduce header sizes.
Get rid of Gcc's LTO flag. Causes a performance drop.
Fix preprocessing behavior when LZP does not compress but Delta2 works.
 2012-12-23 23:50:45 +05:30
Moinak Ghosh
a43fdd7d2c Improve Delta2 scanning speed and effectiveness. 
Add destination buffer overflow check in Delta2.
Add rough speed computation.
 2012-12-23 00:44:56 +05:30
Moinak Ghosh
c2df666daf Major improvement to Delta2 encoding algorithm. 2012-12-18 23:32:55 +05:30
Moinak Ghosh
fb30b5c295 Enable building with alternate Zlib and Bzlib. 
Update README and comments.
Fix correct setting of output size when using Delta2 without LZP.
 2012-12-16 23:17:04 +05:30
Moinak Ghosh
ef0191729e Make Delta2 encoding independent of LZP. 
Tweak Delta2 parameters.
Update README and test cases.
 2012-12-15 22:03:23 +05:30
Moinak Ghosh
a98778d62f Fine tune transpose parameters. 
Fix minor nits.
 2012-12-14 19:12:48 +05:30
Moinak Ghosh
b0f41c2888 Add matrix transpose to Delta2 encoding. 
Change confusing structure member name.
 2012-12-13 21:18:16 +05:30
Moinak Ghosh
375ebefa0d Add Matrix Transpose of Dedupe index to compress it better. 
Fix handling of Dedupe index compression failure.
 2012-12-13 00:00:47 +05:30
Moinak Ghosh
03840b31c5 Update to latest LZ4. 
Update a couple of comments.
 2012-12-11 11:38:42 +05:30
Moinak Ghosh
224fb529e9 Get rid of size_t in places where 64-bitness is assumed. 2012-12-09 10:15:06 +05:30
Moinak Ghosh
75c81b5e9c Add support for 64-bit Keccak implementation. 
Sanitize error message from tests.
Add more tests.
Improve platform detection in config script.
 2012-12-08 14:19:01 +05:30
Moinak Ghosh
29b0d8fd7b Implement Adaptive Delta Encoding. 2012-12-05 00:09:47 +05:30
Moinak Ghosh
51249c858d Work in progress Adaptive Delta Encoding. 2012-12-03 23:45:41 +05:30
Moinak Ghosh
deecbadac4 Fix thread error reporting. 
Update error condition tests to not truncate archive.
 2012-11-25 15:12:45 +05:30
Moinak Ghosh
d250322490 Fix issues with error handling. 
Add new tests for out of range values and corrupted file.
 2012-11-24 23:53:07 +05:30
Moinak Ghosh
bdf16c4cb9 Avoid CRC32 when decompressing older version archives. 2012-11-24 00:00:05 +05:30
Moinak Ghosh
817e16cce6 Use HMAC on header and encrypted data, avoid regular digest when encrypting. 2012-11-23 22:27:14 +05:30
Moinak Ghosh
6e9ca97bdf Add CRC32 header checksums in non-cryptographic mode. 2012-11-23 00:00:39 +05:30
Moinak Ghosh
d054e0f713 Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. 
Change some function signatures to improve algo init function behavior.
Fix corner case dedupe bug in error handling flow.
Bump archive version signature.
 2012-11-22 21:02:50 +05:30
Moinak Ghosh
5b1a13480b Fix chunk flag setup when compression fails in adaptive mode. 
Prevent display of non-fatal errors during compression.
 2012-11-09 21:07:23 +05:30
Moinak Ghosh
77db54e712 Add buffer overflow check in Ppmd compression routines. 
Fix pipe mode encryption check.
Change file difference check in tests.
Add more tests.
 2012-11-09 20:05:13 +05:30
Moinak Ghosh
da9083ae12 Fix encryption with adaptive modes. 
Add missing zero-out of algorithm data field.
 2012-11-09 18:53:48 +05:30
Moinak Ghosh
315bb50264 Change chunk HMAC to only check header, not full data. Digest in hdr int turn validates data. 2012-10-19 21:51:27 +05:30
Moinak Ghosh
1eb8c94fed Implement HMAC verification for chunk data and header. 
Do not ask for decryption passwd twice.
 2012-10-18 22:55:41 +05:30
Moinak Ghosh
d13aad830e Implement HMAC Functionality. 
Refactor crypto utility functions in a separate file.
Add HMAC verification for file header.
Add a few extra input validation checks.
Include space in chunk header for HMAC.
 2012-10-17 23:32:35 +05:30
Moinak Ghosh
f2d7bea902 Support for chunk-level encryption using AES and Scrypt based PBE. 
Couple of minor fixes.
 2012-10-15 12:10:00 +05:30
Moinak Ghosh
2578676b1b Add support for SHA256 and SHA512 digests from OpenSSL library. 2012-10-03 22:43:02 +05:30
Moinak Ghosh
8f8af7ed6b Update adaptive mode heuristic based on algorithms. 
Remove incorrect check in PPMd decompression code.
More refactoring of variable names.
 2012-09-27 22:29:08 +05:30
Moinak Ghosh
3544a8c708 Fix polynomial table computation. 
Change hashing and length bias to reduce hashtable bucket collisions.
Add support for user-selectable 60% or 40% similarity for Delta Compression.
Overall slight speedup.
 2012-09-24 22:20:27 +05:30
Moinak Ghosh
8386e72566 Rewrite core dedupe logic to simplify code and improve performance. 
Hashtable based chunk-level deduplication instead of Quicksort.
Fix a corner case bug in Dedupe decompression.
 2012-09-23 14:57:09 +05:30
Moinak Ghosh
99a8e4cd98 Speed up Hash computation for dedupe blocks. 
Add missing initialization of sliding window.
Update help text.
 2012-09-19 20:29:44 +05:30
Moinak Ghosh
e3befd9e16 Add support for Fixed-Block deduplication. 
More refactoring of symbol names.
 2012-09-16 11:12:58 +05:30
Moinak Ghosh
b9355a5dcc Reduce dedupe loop checks for slight speed edge. 
Beginnings of Fixed-block dedupe.
Update variable name for clarity.
 2012-09-15 11:14:58 +05:30
Moinak Ghosh
fb0aef0bd6 Fix return code handling in LZP pre-compression, crashed adaptive modes. 2012-09-07 19:31:35 +05:30
Moinak Ghosh
e6f042aaf8 Allow user-specified minimum Dedupe block size. 
Compute similarity sketch only if Delta Compression enabled.
 2012-09-05 22:43:54 +05:30
Moinak Ghosh
262566b59a Add xxHash for Rabin block checksums, slightly faster than CRC64. 
Fix missing initialization of character counts table.
Some file reorganization.
 2012-09-02 20:40:32 +05:30
Moinak Ghosh
4ba840b255 Add ASM version of Skein for x64 platforms with auto-detection 
Error checking for checksum flag when decompressing
Update comments and READMEs
 2012-09-01 14:40:15 +05:30
Moinak Ghosh
eda312ce1e Add support for Skein512 and Skein256 checksums 
Import Skein code from NIST CD submission
Make checksum algorithms pluggable
Fix handling of huge buffers (>2GB) in LZP
Cleanup of some buffer sizing code
Speed up CRC64 calculation in dedupe chunking
 2012-08-31 22:36:06 +05:30
Moinak Ghosh
a222772940 Fix single chunk flag handling during decompression. 
Update docs.
 2012-08-27 22:24:23 +05:30