greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
217 commits 3 branches 21 tags 7.6 MiB
Commit graph

217 commits

This branch
This branch
All branches
Author SHA1 Message Date
Moinak Ghosh
3b1d6b55fe Work in progress global dedupe config loader. 2012-11-19 21:41:56 +05:30
Moinak Ghosh
2909a3abff Update README. 2012-11-18 23:19:22 +05:30
Moinak Ghosh
393ced991a A couple of minor cleanups. 2012-11-18 20:20:16 +05:30
Moinak Ghosh
b2cbf0699e Use fixed rolling-hash mask for better block size approximation. 2012-11-11 14:58:39 +05:30
Moinak Ghosh
4d2e8ba644 Update Changelog for 1.0 release. 
Update version string and install doc.
Add Fixed-Block Deduplication tests.
 2012-11-09 21:42:56 +05:30
Moinak Ghosh
f9a15972d6 Add some more crypto test cases. 2012-11-09 21:17:06 +05:30
Moinak Ghosh
5b1a13480b Fix chunk flag setup when compression fails in adaptive mode. 
Prevent display of non-fatal errors during compression.
 2012-11-09 21:07:23 +05:30
Moinak Ghosh
77db54e712 Add buffer overflow check in Ppmd compression routines. 
Fix pipe mode encryption check.
Change file difference check in tests.
Add more tests.
 2012-11-09 20:05:13 +05:30
Moinak Ghosh
da9083ae12 Fix encryption with adaptive modes. 
Add missing zero-out of algorithm data field.
 2012-11-09 18:53:48 +05:30
Moinak Ghosh
7debc60dd5 Add a functionality test suite. 2012-11-08 22:22:37 +05:30
Moinak Ghosh
eacbf207aa Tweak chunking parameters for better block size distribution and dedupe ratio. 2012-11-08 19:41:33 +05:30
Moinak Ghosh
e437390e53 Add some more debug mode info. 2012-11-07 22:55:14 +05:30
Moinak Ghosh
3aa33f5b94 Minor fix for adapt mode. 2012-11-04 21:46:04 +05:30
Moinak Ghosh
f0c7ba87a3 Use Libbsc for XML data in adapt2 mode. 2012-11-04 21:13:26 +05:30
Moinak Ghosh
ddda9e5320 Bump version to 0.9.1 
Update Changelog.
 2012-10-24 18:46:27 +05:30
Moinak Ghosh
c5ebe1f30a Portability to Debian based distros. 
Enable SSE4/AVX detection for AMD platforms (Bulldozer has both).
Portable long long int print formatting to silence gcc 4.6 warnings.
 2012-10-21 21:03:07 +05:30
Moinak Ghosh
45487469d1 Add another example invocation. 2012-10-21 12:34:34 +05:30
Moinak Ghosh
189d32ea52 Bump version to 0.9.0. 
Update Changelog.
 2012-10-20 23:10:05 +05:30
Moinak Ghosh
315bb50264 Change chunk HMAC to only check header, not full data. Digest in hdr int turn validates data. 2012-10-19 21:51:27 +05:30
Moinak Ghosh
1eb8c94fed Implement HMAC verification for chunk data and header. 
Do not ask for decryption passwd twice.
 2012-10-18 22:55:41 +05:30
Moinak Ghosh
d13aad830e Implement HMAC Functionality. 
Refactor crypto utility functions in a separate file.
Add HMAC verification for file header.
Add a few extra input validation checks.
Include space in chunk header for HMAC.
 2012-10-17 23:32:35 +05:30
Moinak Ghosh
f2d7bea902 Support for chunk-level encryption using AES and Scrypt based PBE. 
Couple of minor fixes.
 2012-10-15 12:10:00 +05:30
Moinak Ghosh
21cbef6d60 Incorporate SSE/AVX optimized Intel SHA-256 implementation. 
Add support for runtime cpuid detection.
 2012-10-06 20:51:23 +05:30
Moinak Ghosh
c880b73d26 Fix typo. 2012-10-03 22:47:16 +05:30
Moinak Ghosh
2578676b1b Add support for SHA256 and SHA512 digests from OpenSSL library. 2012-10-03 22:43:02 +05:30
Moinak Ghosh
3492949aff Bump version to 0.8.6 
Update Changelog
 2012-10-02 10:24:32 +05:30
Moinak Ghosh
9475ccc3d6 Fix polynomial computation. 
Fix incorrect block length when doing fixed-block dedupe.
 2012-10-02 00:20:44 +05:30
Moinak Ghosh
0019efbadb Adjust break pattern check mask for closer approximation to average block size. 
Remove unused structure member.
 2012-09-29 23:31:45 +05:30
Moinak Ghosh
24e6f4e629 Switch to multiplicative rolling hash for good distribution properties. 2012-09-29 00:09:49 +05:30
Moinak Ghosh
d94be4e314 Update Changelog and bump version to 0.8.5. 2012-09-27 23:16:13 +05:30
Moinak Ghosh
8f8af7ed6b Update adaptive mode heuristic based on algorithms. 
Remove incorrect check in PPMd decompression code.
More refactoring of variable names.
 2012-09-27 22:29:08 +05:30
Moinak Ghosh
449dc35675 Speed up adaptive modes by using heuristics to select compression algorithm. 
Select similarity percentage based on dedupe block size for effectiveness.
 2012-09-26 19:47:32 +05:30
Moinak Ghosh
333b7b011e Fix check for size reduction in dedupe. 
Tweak debug message for clarity.
 2012-09-25 23:24:33 +05:30
Moinak Ghosh
3544a8c708 Fix polynomial table computation. 
Change hashing and length bias to reduce hashtable bucket collisions.
Add support for user-selectable 60% or 40% similarity for Delta Compression.
Overall slight speedup.
 2012-09-24 22:20:27 +05:30
Moinak Ghosh
8386e72566 Rewrite core dedupe logic to simplify code and improve performance. 
Hashtable based chunk-level deduplication instead of Quicksort.
Fix a corner case bug in Dedupe decompression.
 2012-09-23 14:57:09 +05:30
Moinak Ghosh
99a8e4cd98 Speed up Hash computation for dedupe blocks. 
Add missing initialization of sliding window.
Update help text.
 2012-09-19 20:29:44 +05:30
Moinak Ghosh
e3befd9e16 Add support for Fixed-Block deduplication. 
More refactoring of symbol names.
 2012-09-16 11:12:58 +05:30
Moinak Ghosh
b9355a5dcc Reduce dedupe loop checks for slight speed edge. 
Beginnings of Fixed-block dedupe.
Update variable name for clarity.
 2012-09-15 11:14:58 +05:30
Moinak Ghosh
a6b3719d89 Fix conditional in heapq function. 2012-09-11 21:59:08 +05:30
Moinak Ghosh
f3f472b860 Implement K-min-values Sketch for Similarity detection. 2012-09-11 20:26:36 +05:30
Moinak Ghosh
117382c141 Update README to reflect current features. 2012-09-07 21:32:20 +05:30
Moinak Ghosh
05a010a9dd Bump version to 0.8.1 
Update Changelog for 0.8.1 release.
 2012-09-07 19:38:36 +05:30
Moinak Ghosh
fb0aef0bd6 Fix return code handling in LZP pre-compression, crashed adaptive modes. 2012-09-07 19:31:35 +05:30
Moinak Ghosh
e6f042aaf8 Allow user-specified minimum Dedupe block size. 
Compute similarity sketch only if Delta Compression enabled.
 2012-09-05 22:43:54 +05:30
Moinak Ghosh
560fa85aab Fix secondary sketch computation, some more accuracy in diff detection. 2012-09-04 23:28:02 +05:30
Moinak Ghosh
262566b59a Add xxHash for Rabin block checksums, slightly faster than CRC64. 
Fix missing initialization of character counts table.
Some file reorganization.
 2012-09-02 20:40:32 +05:30
Moinak Ghosh
4ba840b255 Add ASM version of Skein for x64 platforms with auto-detection 
Error checking for checksum flag when decompressing
Update comments and READMEs
 2012-09-01 14:40:15 +05:30
Moinak Ghosh
eda312ce1e Add support for Skein512 and Skein256 checksums 
Import Skein code from NIST CD submission
Make checksum algorithms pluggable
Fix handling of huge buffers (>2GB) in LZP
Cleanup of some buffer sizing code
Speed up CRC64 calculation in dedupe chunking
 2012-08-31 22:36:06 +05:30
Moinak Ghosh
f03834278a Bump version to 0.8. 
Update Changelog for 0.8 beta release.
 2012-08-27 22:53:39 +05:30
Moinak Ghosh
a222772940 Fix single chunk flag handling during decompression. 
Update docs.
 2012-08-27 22:24:23 +05:30