Moinak Ghosh
3e1737b4ab
Use OpenMP parallelism when computing xxHashes for chunks.
2013-02-02 09:27:58 +05:30
Moinak Ghosh
6bfd044311
Use 2-stage Merkle Tree hashing for parallel hashes for better crypto properties.
...
Update xxhash comment.
2013-02-01 22:07:28 +05:30
Moinak Ghosh
af4c6e1d84
Reduce dedupe hash table collisions by half.
2013-01-31 00:38:41 +05:30
Moinak Ghosh
3d8f3ada1c
Improve Deduplication performance by another 95%.
...
Start sliding window scanning near minimum chunk size boundaries to avoid scanning whole chunk.
2013-01-30 22:41:13 +05:30
Moinak Ghosh
9983d79e62
Bump version and update changelog for 1.3.0 release.
...
Fix issue #3 .
2013-01-29 21:42:54 +05:30
Moinak Ghosh
a03e3ba41b
Fix return from parallel versions of Keccak_Hash() function.
2013-01-28 00:51:47 +05:30
Moinak Ghosh
468044d816
Add parallel versions of various checksums for single-segment, single-thread compression.
2013-01-27 23:47:55 +05:30
Moinak Ghosh
2da0d0950b
Use BLAKE2 parallel version for single-chunk archives (whole file in one chunk).
...
Set decompression threads correctly for single-chunk archives.
2013-01-26 18:28:13 +05:30
Moinak Ghosh
68f60ba50b
Update default checksum to BLAKE256.
2013-01-26 17:40:23 +05:30
Moinak Ghosh
d08b5ea399
Add optimized BLAKE2 implementations with runtime detection of CPU capability (SSE/AVX).
...
Minor cleanups.
2013-01-26 15:39:10 +05:30
Moinak Ghosh
43af97042a
Major changes to use Intel's optimized SHA512 code for SHA512 and SHA512/256.
...
Remove earlier SHA256 code which is slower than SHA512/256 (on 64-bit CPU).
Use HMAC from Alan Saddi's implementation for cleaner, faster code.
2013-01-25 22:55:55 +05:30
Moinak Ghosh
26bb137257
Changes for generalized runtime SSE/AVX/XOP detection.
...
Multi instruction set XXhash build with runtime selection.
Extend CPUID code to detect more instruction sets.
Add options for BLAKE2 hash.
Move GCC builtins into utils header.
Bump file format version number due to extended digest flags.
Add descriptions to digest list.
2013-01-25 00:10:12 +05:30
Moinak Ghosh
7b7c85dab4
Rationalize XXHash implementation to deal with 32-byte blocks instead of 16-byte.
...
Fix XXHash performance degradation for small keys.
Modify a data analysis loop in adaptive compress to make it auto-vectorizable.
2013-01-23 20:58:39 +05:30
Moinak Ghosh
5c8704c5bb
Improve Deduplication throughtput by 90%.
...
Use SSE4 register as sliding window for default 16-byte window size.
Use local variable for sliding window position to avoid spurios memory access in non-SIMD case.
Avoid computing breakpoint check value if processed length < minimum block length.
2013-01-22 15:54:42 +05:30
Moinak Ghosh
e9e3e1e632
Improve SSE version detection.
...
Add SSE4 detection.
Fix setting of some opt flags in Makefile.in.
2013-01-20 22:53:36 +05:30
Moinak Ghosh
3888c8d316
Many optimization tweaks
...
Optimize Rabin Deduplication and Bsdiff
Vectorize XXHash using SE4
2013-01-20 22:02:26 +05:30
Moinak Ghosh
455c8107d5
Use pre-increment for shorter instruction length and slight speed.
2013-01-17 22:54:30 +05:30
Moinak Ghosh
49ec3a054d
Add SSE2 improvements to CTR mode AES.
...
Add debug print of encryption and HMAC throughput.
Fix error message for invalid option.
2013-01-16 19:52:46 +05:30
Moinak Ghosh
39dbc4be43
Implement algo-specific minimum distance match for Delta Compression.
2013-01-14 13:20:07 +05:30
Moinak Ghosh
d49a088eea
Fixes and performance improvements for Dedupe Delta Compression
...
Avoid using fingerprints in minhash computation and fix write amplification
Modify min-heap to use 64bit values
Improve bsdiff performance
Fix pointer comparison in bsdiff
Use 32bit offsets in bsdiff to reduce memory usage
Improve Zero RLE Encoder performance
Add more buffer overflow checks in Zero RLE Decoder
2013-01-13 22:04:59 +05:30
Moinak Ghosh
87aa12206e
Use SSE3 lddqu in the matchfinder if SSE3 is enabled.
2013-01-06 19:28:40 +05:30
Moinak Ghosh
976a12afbe
Remove outdated LZP note.
2013-01-05 19:59:17 +05:30
Moinak Ghosh
254f03a7a2
Update Changelog for last fixes.
2013-01-05 17:03:41 +05:30
Moinak Ghosh
a8fd60fb06
Fix issue #1 and issue #2 .
...
Enable building with older openssl (at least 0.9.8e).
Add variants of missing functions in older openssl versions.
Allow proper linking with libraries in alternate locations and setting RUNPATH.
Increase hash rounds for nonce generation.
2013-01-05 00:16:15 +05:30
Moinak Ghosh
16b1d9e7a3
Bump version, update Changelog and documentation for 1.2 release.
2013-01-03 23:40:21 +05:30
Moinak Ghosh
47ebd5b752
Fix calculation of extra scratch space for Dedupe.
...
Add missing return value check in small buffer Delta2.
Update test failure detection in test driver.
2013-01-03 22:25:14 +05:30
Moinak Ghosh
d9eb82e0e8
Fix numeric parsing.
...
Fix dedupe bug introduced in last commit.
Reset valid flag when resetting dedupe context.
Cleanup test suites.
Do not abort test suite on failure of a test case.
2013-01-03 00:27:18 +05:30
Moinak Ghosh
6b756eb165
Fix Delta2 handling for small buffers.
...
Fix LZP handling during preprocessing.
Fix type flag handling during preprocessing.
Update test cases to use configurable list of test corpus files.
Update some more int64_t datatypes to uint64_t
Add a gitignore.
2013-01-02 22:56:21 +05:30
Moinak Ghosh
43f5acfa2d
Add more comments to code.
2012-12-31 23:27:31 +05:30
Moinak Ghosh
13d9378acd
Update to latest XXHash version.
2012-12-31 11:53:47 +05:30
Moinak Ghosh
8bfa49fc66
Fix Keccak invocation and reset default checksum back to SKEIN 256.
2012-12-30 01:13:55 +05:30
Moinak Ghosh
28224d29d3
Improve Dedupe performance.
...
Add more debug timing stats.
Change default checksum to Keccak 256 (SIMD version 4x faster than Skein).
Fix compiler warning in allocator code.
2012-12-29 23:43:41 +05:30
Moinak Ghosh
36d95276ee
Further improvements to Delta2 performance.
...
Fix the byteswap macros.
Start adding assertions.
2012-12-28 22:12:38 +05:30
Moinak Ghosh
26a4f42506
Introduce strict compiler flags and fix scores of warnings/issues.
...
Avoid different optimization flags for Dedupe sources.
Fix liberal mixing of uint64_t and int64_t (should all be uint64_t).
Fix corner case crash when decompressing.
2012-12-27 23:06:48 +05:30
Moinak Ghosh
063624d0d3
Fix Delta2 decode handling when not compressing.
...
More debug statements and other minor changes.
2012-12-24 23:21:02 +05:30
Moinak Ghosh
d597f0f05c
Major enhancements to Delta2 encoding.
...
Avoid transposing below-threshold spans. Reduces compression ratio.
Use little-endian storage format for numbers to optimize for x86.
Improve embedded table detection.
Reduce header sizes.
Get rid of Gcc's LTO flag. Causes a performance drop.
Fix preprocessing behavior when LZP does not compress but Delta2 works.
2012-12-23 23:50:45 +05:30
Moinak Ghosh
a43fdd7d2c
Improve Delta2 scanning speed and effectiveness.
...
Add destination buffer overflow check in Delta2.
Add rough speed computation.
2012-12-23 00:44:56 +05:30
Moinak Ghosh
765b014018
More tweaks to Delta2 implementation.
2012-12-21 22:58:30 +05:30
Moinak Ghosh
9bd8146e88
Improve Delta2 performance.
2012-12-20 23:49:44 +05:30
Moinak Ghosh
e9c3b34ff5
Add debug build verbose messages to delta code.
2012-12-19 22:42:55 +05:30
Moinak Ghosh
c2df666daf
Major improvement to Delta2 encoding algorithm.
2012-12-18 23:32:55 +05:30
Moinak Ghosh
1fe5157eec
Ability to run individual test suites from makefile.
2012-12-17 20:59:27 +05:30
Moinak Ghosh
00c8c534d0
Silence more Gcc warnings.
...
Update testing note.
2012-12-17 20:23:30 +05:30
Moinak Ghosh
25bb54e3bc
Improve 64-bit compiler and platform checks.
...
Ensure core dumps are enabled during testing.
2012-12-17 18:37:27 +05:30
Moinak Ghosh
fb30b5c295
Enable building with alternate Zlib and Bzlib.
...
Update README and comments.
Fix correct setting of output size when using Delta2 without LZP.
2012-12-16 23:17:04 +05:30
Moinak Ghosh
5ac47db6d5
README formatting.
2012-12-15 22:09:07 +05:30
Moinak Ghosh
ef0191729e
Make Delta2 encoding independent of LZP.
...
Tweak Delta2 parameters.
Update README and test cases.
2012-12-15 22:03:23 +05:30
Moinak Ghosh
b01d255f6c
Update README to align with current features/behavior.
2012-12-15 12:42:46 +05:30
Moinak Ghosh
a98778d62f
Fine tune transpose parameters.
...
Fix minor nits.
2012-12-14 19:12:48 +05:30
Moinak Ghosh
b0f41c2888
Add matrix transpose to Delta2 encoding.
...
Change confusing structure member name.
2012-12-13 21:18:16 +05:30