greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
245 commits 3 branches 21 tags 7.6 MiB
Commit graph

245 commits

This branch
This branch
All branches
Author SHA1 Message Date
Moinak Ghosh
efe5232cdc Add compatibility to decode old-format parallel hashes created with version 1.2. 
Bump archive version to 7 as parallel hashes are now merkle style hashes.
 2013-02-24 20:05:16 +05:30
Moinak Ghosh
5f6217bb1f Add lookup and insert functionality for global index. 
Make global dedupe code buildable.
 2013-02-21 23:07:07 +05:30
Moinak Ghosh
cb853821c7 Use PPMd fallback for adapt2 if BSC is not enabled. 2013-02-17 22:01:29 +05:30
Moinak Ghosh
f41ea40bb9 Improve XML detection in adaptive mode. 2013-02-17 21:36:20 +05:30
Moinak Ghosh
6badbcaea7 Make global dedupe bits buildable and fix errors. 
Rename Adaptive compression type constants to avoid conflict with global constants.
 2013-02-17 21:05:40 +05:30
Moinak Ghosh
7386f82a4f Work in progress global dedupe. 2013-02-16 23:33:06 +05:30
Moinak Ghosh
f89473d29c Fixes for issues/warnings reported in issue #4. 2013-02-15 22:53:17 +05:30
Moinak Ghosh
24d62bfde9 Global dedupe work in progress. 2013-02-14 23:10:53 +05:30
Moinak Ghosh
1eae57c8a2 Starting changes for single-file global dedupe. 2013-02-12 21:53:04 +05:30
Moinak Ghosh
3e1737b4ab Use OpenMP parallelism when computing xxHashes for chunks. 2013-02-02 09:27:58 +05:30
Moinak Ghosh
6bfd044311 Use 2-stage Merkle Tree hashing for parallel hashes for better crypto properties. 
Update xxhash comment.
 2013-02-01 22:07:28 +05:30
Moinak Ghosh
af4c6e1d84 Reduce dedupe hash table collisions by half. 2013-01-31 00:38:41 +05:30
Moinak Ghosh
3d8f3ada1c Improve Deduplication performance by another 95%. 
Start sliding window scanning near minimum chunk size boundaries to avoid scanning whole chunk.
 2013-01-30 22:41:13 +05:30
Moinak Ghosh
9983d79e62 Bump version and update changelog for 1.3.0 release. 
Fix issue #3.
 2013-01-29 21:42:54 +05:30
Moinak Ghosh
a03e3ba41b Fix return from parallel versions of Keccak_Hash() function. 2013-01-28 00:51:47 +05:30
Moinak Ghosh
468044d816 Add parallel versions of various checksums for single-segment, single-thread compression. 2013-01-27 23:47:55 +05:30
Moinak Ghosh
2da0d0950b Use BLAKE2 parallel version for single-chunk archives (whole file in one chunk). 
Set decompression threads correctly for single-chunk archives.
 2013-01-26 18:28:13 +05:30
Moinak Ghosh
68f60ba50b Update default checksum to BLAKE256. 2013-01-26 17:40:23 +05:30
Moinak Ghosh
d08b5ea399 Add optimized BLAKE2 implementations with runtime detection of CPU capability (SSE/AVX). 
Minor cleanups.
 2013-01-26 15:39:10 +05:30
Moinak Ghosh
43af97042a Major changes to use Intel's optimized SHA512 code for SHA512 and SHA512/256. 
Remove earlier SHA256 code which is slower than SHA512/256 (on 64-bit CPU).
Use HMAC from Alan Saddi's implementation for cleaner, faster code.
 2013-01-25 22:55:55 +05:30
Moinak Ghosh
26bb137257 Changes for generalized runtime SSE/AVX/XOP detection. 
Multi instruction set XXhash build with runtime selection.
Extend CPUID code to detect more instruction sets.
Add options for BLAKE2 hash.
Move GCC builtins into utils header.
Bump file format version number due to extended digest flags.
Add descriptions to digest list.
 2013-01-25 00:10:12 +05:30
Moinak Ghosh
7b7c85dab4 Rationalize XXHash implementation to deal with 32-byte blocks instead of 16-byte. 
Fix XXHash performance degradation for small keys.
Modify a data analysis loop in adaptive compress to make it auto-vectorizable.
 2013-01-23 20:58:39 +05:30
Moinak Ghosh
5c8704c5bb Improve Deduplication throughtput by 90%. 
Use SSE4 register as sliding window for default 16-byte window size.
Use local variable for sliding window position to avoid spurios memory access in non-SIMD case.
Avoid computing breakpoint check value if processed length < minimum block length.
 2013-01-22 15:54:42 +05:30
Moinak Ghosh
e9e3e1e632 Improve SSE version detection. 
Add SSE4 detection.
Fix setting of some opt flags in Makefile.in.
 2013-01-20 22:53:36 +05:30
Moinak Ghosh
3888c8d316 Many optimization tweaks 
Optimize Rabin Deduplication and Bsdiff
Vectorize XXHash using SE4
 2013-01-20 22:02:26 +05:30
Moinak Ghosh
455c8107d5 Use pre-increment for shorter instruction length and slight speed. 2013-01-17 22:54:30 +05:30
Moinak Ghosh
49ec3a054d Add SSE2 improvements to CTR mode AES. 
Add debug print of encryption and HMAC throughput.
Fix error message for invalid option.
 2013-01-16 19:52:46 +05:30
Moinak Ghosh
39dbc4be43 Implement algo-specific minimum distance match for Delta Compression. 2013-01-14 13:20:07 +05:30
Moinak Ghosh
d49a088eea Fixes and performance improvements for Dedupe Delta Compression 
Avoid using fingerprints in minhash computation and fix write amplification
Modify min-heap to use 64bit values
Improve bsdiff performance
Fix pointer comparison in bsdiff
Use 32bit offsets in bsdiff to reduce memory usage
Improve Zero RLE Encoder performance
Add more buffer overflow checks in Zero RLE Decoder
 2013-01-13 22:04:59 +05:30
Moinak Ghosh
87aa12206e Use SSE3 lddqu in the matchfinder if SSE3 is enabled. 2013-01-06 19:28:40 +05:30
Moinak Ghosh
976a12afbe Remove outdated LZP note. 2013-01-05 19:59:17 +05:30
Moinak Ghosh
254f03a7a2 Update Changelog for last fixes. 2013-01-05 17:03:41 +05:30
Moinak Ghosh
a8fd60fb06 Fix issue #1 and issue #2. 
Enable building with older openssl (at least 0.9.8e).
Add variants of missing functions in older openssl versions.
Allow proper linking with libraries in alternate locations and setting RUNPATH.
Increase hash rounds for nonce generation.
 2013-01-05 00:16:15 +05:30
Moinak Ghosh
16b1d9e7a3 Bump version, update Changelog and documentation for 1.2 release. 2013-01-03 23:40:21 +05:30
Moinak Ghosh
47ebd5b752 Fix calculation of extra scratch space for Dedupe. 
Add missing return value check in small buffer Delta2.
Update test failure detection in test driver.
 2013-01-03 22:25:14 +05:30
Moinak Ghosh
d9eb82e0e8 Fix numeric parsing. 
Fix dedupe bug introduced in last commit.
Reset valid flag when resetting dedupe context.
Cleanup test suites.
Do not abort test suite on failure of a test case.
 2013-01-03 00:27:18 +05:30
Moinak Ghosh
6b756eb165 Fix Delta2 handling for small buffers. 
Fix LZP handling during preprocessing.
Fix type flag handling during preprocessing.
Update test cases to use configurable list of test corpus files.
Update some more int64_t datatypes to uint64_t
Add a gitignore.
 2013-01-02 22:56:21 +05:30
Moinak Ghosh
43f5acfa2d Add more comments to code. 2012-12-31 23:27:31 +05:30
Moinak Ghosh
13d9378acd Update to latest XXHash version. 2012-12-31 11:53:47 +05:30
Moinak Ghosh
8bfa49fc66 Fix Keccak invocation and reset default checksum back to SKEIN 256. 2012-12-30 01:13:55 +05:30
Moinak Ghosh
28224d29d3 Improve Dedupe performance. 
Add more debug timing stats.
Change default checksum to Keccak 256 (SIMD version 4x faster than Skein).
Fix compiler warning in allocator code.
 2012-12-29 23:43:41 +05:30
Moinak Ghosh
36d95276ee Further improvements to Delta2 performance. 
Fix the byteswap macros.
Start adding assertions.
 2012-12-28 22:12:38 +05:30
Moinak Ghosh
26a4f42506 Introduce strict compiler flags and fix scores of warnings/issues. 
Avoid different optimization flags for Dedupe sources.
Fix liberal mixing of uint64_t and int64_t (should all be uint64_t).
Fix corner case crash when decompressing.
 2012-12-27 23:06:48 +05:30
Moinak Ghosh
063624d0d3 Fix Delta2 decode handling when not compressing. 
More debug statements and other minor changes.
 2012-12-24 23:21:02 +05:30
Moinak Ghosh
d597f0f05c Major enhancements to Delta2 encoding. 
Avoid transposing below-threshold spans. Reduces compression ratio.
Use little-endian storage format for numbers to optimize for x86.
Improve embedded table detection.
Reduce header sizes.
Get rid of Gcc's LTO flag. Causes a performance drop.
Fix preprocessing behavior when LZP does not compress but Delta2 works.
 2012-12-23 23:50:45 +05:30
Moinak Ghosh
a43fdd7d2c Improve Delta2 scanning speed and effectiveness. 
Add destination buffer overflow check in Delta2.
Add rough speed computation.
 2012-12-23 00:44:56 +05:30
Moinak Ghosh
765b014018 More tweaks to Delta2 implementation. 2012-12-21 22:58:30 +05:30
Moinak Ghosh
9bd8146e88 Improve Delta2 performance. 2012-12-20 23:49:44 +05:30
Moinak Ghosh
e9c3b34ff5 Add debug build verbose messages to delta code. 2012-12-19 22:42:55 +05:30
Moinak Ghosh
c2df666daf Major improvement to Delta2 encoding algorithm. 2012-12-18 23:32:55 +05:30