greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
210 commits 3 branches 21 tags 7.6 MiB
Commit graph

52 commits

Author SHA1 Message Date
Moinak Ghosh
b7fdeb08bc Work in progress global dedupe changes. 2013-03-20 22:47:03 +05:30
Moinak Ghosh
f2806d4ffa Work in progress global dedupe changes. 2013-03-19 20:13:44 +05:30
Moinak Ghosh
f8f23e5200 Major License text cleanup. 2013-03-07 20:26:48 +05:30
Moinak Ghosh
370e84f2be Update Changelog and bump version for 1.4 release. 
Add license text to Salsa20 files.
 2013-03-06 17:15:03 +05:30
Moinak Ghosh
efe5232cdc Add compatibility to decode old-format parallel hashes created with version 1.2. 
Bump archive version to 7 as parallel hashes are now merkle style hashes.
 2013-02-24 20:05:16 +05:30
Moinak Ghosh
6badbcaea7 Make global dedupe bits buildable and fix errors. 
Rename Adaptive compression type constants to avoid conflict with global constants.
 2013-02-17 21:05:40 +05:30
Moinak Ghosh
9983d79e62 Bump version and update changelog for 1.3.0 release. 
Fix issue #3.
 2013-01-29 21:42:54 +05:30
Moinak Ghosh
2da0d0950b Use BLAKE2 parallel version for single-chunk archives (whole file in one chunk). 
Set decompression threads correctly for single-chunk archives.
 2013-01-26 18:28:13 +05:30
Moinak Ghosh
26bb137257 Changes for generalized runtime SSE/AVX/XOP detection. 
Multi instruction set XXhash build with runtime selection.
Extend CPUID code to detect more instruction sets.
Add options for BLAKE2 hash.
Move GCC builtins into utils header.
Bump file format version number due to extended digest flags.
Add descriptions to digest list.
 2013-01-25 00:10:12 +05:30
Moinak Ghosh
16b1d9e7a3 Bump version, update Changelog and documentation for 1.2 release. 2013-01-03 23:40:21 +05:30
Moinak Ghosh
26a4f42506 Introduce strict compiler flags and fix scores of warnings/issues. 
Avoid different optimization flags for Dedupe sources.
Fix liberal mixing of uint64_t and int64_t (should all be uint64_t).
Fix corner case crash when decompressing.
 2012-12-27 23:06:48 +05:30
Moinak Ghosh
c2df666daf Major improvement to Delta2 encoding algorithm. 2012-12-18 23:32:55 +05:30
Moinak Ghosh
224fb529e9 Get rid of size_t in places where 64-bitness is assumed. 2012-12-09 10:15:06 +05:30
Moinak Ghosh
29b0d8fd7b Implement Adaptive Delta Encoding. 2012-12-05 00:09:47 +05:30
Moinak Ghosh
1f0c237495 Update Change and bump version for 1.1 release. 2012-11-26 20:27:09 +05:30
Moinak Ghosh
d054e0f713 Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. 
Change some function signatures to improve algo init function behavior.
Fix corner case dedupe bug in error handling flow.
Bump archive version signature.
 2012-11-22 21:02:50 +05:30
Moinak Ghosh
4d2e8ba644 Update Changelog for 1.0 release. 
Update version string and install doc.
Add Fixed-Block Deduplication tests.
 2012-11-09 21:42:56 +05:30
Moinak Ghosh
5b1a13480b Fix chunk flag setup when compression fails in adaptive mode. 
Prevent display of non-fatal errors during compression.
 2012-11-09 21:07:23 +05:30
Moinak Ghosh
f0c7ba87a3 Use Libbsc for XML data in adapt2 mode. 2012-11-04 21:13:26 +05:30
Moinak Ghosh
ddda9e5320 Bump version to 0.9.1 
Update Changelog.
 2012-10-24 18:46:27 +05:30
Moinak Ghosh
189d32ea52 Bump version to 0.9.0. 
Update Changelog.
 2012-10-20 23:10:05 +05:30
Moinak Ghosh
1eb8c94fed Implement HMAC verification for chunk data and header. 
Do not ask for decryption passwd twice.
 2012-10-18 22:55:41 +05:30
Moinak Ghosh
d13aad830e Implement HMAC Functionality. 
Refactor crypto utility functions in a separate file.
Add HMAC verification for file header.
Add a few extra input validation checks.
Include space in chunk header for HMAC.
 2012-10-17 23:32:35 +05:30
Moinak Ghosh
f2d7bea902 Support for chunk-level encryption using AES and Scrypt based PBE. 
Couple of minor fixes.
 2012-10-15 12:10:00 +05:30
Moinak Ghosh
3492949aff Bump version to 0.8.6 
Update Changelog
 2012-10-02 10:24:32 +05:30
Moinak Ghosh
d94be4e314 Update Changelog and bump version to 0.8.5. 2012-09-27 23:16:13 +05:30
Moinak Ghosh
e3befd9e16 Add support for Fixed-Block deduplication. 
More refactoring of symbol names.
 2012-09-16 11:12:58 +05:30
Moinak Ghosh
b9355a5dcc Reduce dedupe loop checks for slight speed edge. 
Beginnings of Fixed-block dedupe.
Update variable name for clarity.
 2012-09-15 11:14:58 +05:30
Moinak Ghosh
05a010a9dd Bump version to 0.8.1 
Update Changelog for 0.8.1 release.
 2012-09-07 19:38:36 +05:30
Moinak Ghosh
262566b59a Add xxHash for Rabin block checksums, slightly faster than CRC64. 
Fix missing initialization of character counts table.
Some file reorganization.
 2012-09-02 20:40:32 +05:30
Moinak Ghosh
eda312ce1e Add support for Skein512 and Skein256 checksums 
Import Skein code from NIST CD submission
Make checksum algorithms pluggable
Fix handling of huge buffers (>2GB) in LZP
Cleanup of some buffer sizing code
Speed up CRC64 calculation in dedupe chunking
 2012-08-31 22:36:06 +05:30
Moinak Ghosh
f03834278a Bump version to 0.8. 
Update Changelog for 0.8 beta release.
 2012-08-27 22:53:39 +05:30
Moinak Ghosh
d75535bc7e Add support for libbsc a high-performance block sorting compressor. 
Enable external algorithm threading for single chunk compressed files.
Update docs.
 2012-08-27 21:51:55 +05:30
Moinak Ghosh
3b83bc2d4e Bump file version. 2012-08-26 15:01:18 +05:30
Moinak Ghosh
bf149e880d Add LZP Pre-Compression support ported from libbsc. 
Add generic pre-processing wrappers for future support of other pre-processors.
Clean up computation of Rabin block sizes.
Compute Rabin scratch space accurately to avoid RAM wastage.
 2012-08-23 22:58:44 +05:30
Moinak Ghosh
3851c9c6cc Delay allocation of per-thread chunks for performance and memory efficiency. 
Avoid allocating double-buffer for single-chunk files.
Introduce lzmaMt option to indicate multithreaded LZMA.
Update README.
 2012-08-18 22:00:14 +05:30
Moinak Ghosh
9eac774eb1 Add multithreaded LZMA port from p7zip 
Compute balanced thread count between chunk threads and algo threads
Generic way to handle querying algorithm parameters
Clean up unnecessary includes
 2012-08-18 10:20:52 +05:30
Moinak Ghosh
7b7007d6c5 Update version to 0.7 (0.6 was alpha). 2012-08-10 11:01:42 +05:30
Moinak Ghosh
bde917c8e9 Fix handling of compression flags in adaptive mode 
Fix error handling when chunk size is too small for dedupe
Bump version to 0.6
 2012-08-10 10:47:11 +05:30
Moinak Ghosh
400d0bfa72 Bias fingerprint value with occurrence counts for a better sketch 
Fix latent bug when calling algo deinit in decompression code
Reduce diff threshold for slightly greater delta encoding
Limit similar buffer size difference for less wasted diffing
Change zlib compression wrapper to use faster deflateReset mechanism
Reduce optimization level for Dedupe code, it goes faster
 2012-08-08 22:40:58 +05:30
Moinak Ghosh
a4311f2ede Fix handling of incompressible chunks. 
Fix handling of various dedup failures.
Add NULL compression option for dedup only compression.
 2012-08-05 22:35:51 +05:30
Moinak Ghosh
2cbcb0c9e4 Fix buffer sizing for LZ4. 
Fix exit condition checks in LZ4 decompression wrapper.
 2012-08-04 17:55:20 +05:30
Moinak Ghosh
bc71caffc3 Display release version in usage text. 2012-07-27 22:07:56 +05:30
Moinak Ghosh
c7cc7b469c Update chunk size computation to reduce memory usage. 
Implement runtime bypass of custom allocator.
Update README.
 2012-07-27 22:03:24 +05:30
Moinak Ghosh
296e2ab6b2 Add support for LZ4 compression including multi-pass LZ4. 
Add missing Read_Adjusted() declaration, was causing a crash with 2GB chunks.
Fix minor cut-paste issues in comments.
 2012-07-25 21:07:36 +05:30
Moinak Ghosh
53d4311534 Make LZFX Hash size dynamic. 
Use smaller min rabin block when using fast compression algos.
Add missing check for algo init function return value.
 2012-07-23 21:43:12 +05:30
Moinak Ghosh
8cfd54fe34 Add LZFX Compression support, a very fast lightweight compressor. 
Avoid a branch in the rabin loop.
 2012-07-23 00:15:08 +05:30
Moinak Ghosh
d3f5287ee5 Update License info to LGPLv3. 2012-07-07 22:18:29 +05:30
Moinak Ghosh
a1825a2305 Implement Parallel deduplication support. 
Restructure compression functions to take chunk flag as argument.
Add missing error flag printing in LZMA.
Only create enough threads as needed by chunk size and file size.
Minor cleanups and variable name changes.
 2012-07-01 21:44:02 +05:30
Moinak Ghosh
cbf9728278 Implement Deduplication based on Rabin Fingerprinting: work in progress. 
Fix bug that prevented pipe mode from being used.
Allow building without specialized allocator.
Use basic optimize flag in debuig build.
 2012-06-29 18:23:55 +05:30