greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
375 commits 3 branches 21 tags 7.6 MiB
Commit graph

55 commits

Author SHA1 Message Date
Moinak Ghosh
935717373b Capability to list offset and length of each block when deduplication for external use. 2014-03-30 17:35:21 +05:30
Moinak Ghosh
62568e9066 Basic capability to list contents of an archive without extracting to disk. 2014-01-12 20:38:20 +05:30
Moinak Ghosh
3ddaf6d45f Bump version and update command help text. 2014-01-04 21:45:23 +05:30
Moinak Ghosh
16da0b0339 Fix handling of some options. 
Update README with additional option details.
 2014-01-03 22:51:02 +05:30
Moinak Ghosh
ea345a902a Overhaul documentation part #1 
Detect and handle uncompressed PDF files using libbsc.
Force binary/text data detection for tar archives.
Get rid of unnecessary CLI option.
Add full pipeline mode check when archiving.
 2013-12-30 23:24:37 +05:30
Moinak Ghosh
5a49252bb9 Remove external Libbsc option. 2013-11-30 22:43:31 +05:30
Moinak Ghosh
bc451aba36 Archiving support using Libarchive: Work in progress changes #2. 2013-10-22 23:41:51 +05:30
Moinak Ghosh
7f81869874 Archiving support using Libarchive: Work in progress changes. 
Change all perror() calls to use logger.
Make the config script a little verbose.
 2013-10-20 23:54:27 +05:30
Moinak Ghosh
28fd9848f9 Ability to specify output compressed pathname. 
Fix log level handling.
Trim commented code.
 2013-10-10 21:19:44 +05:30
Moinak Ghosh
b236638e72 Remove confusing option with little practical utility. 
Update test cases and documentation.
 2013-09-01 15:02:28 +05:30
Moinak Ghosh
12a2b8ed63 Additional error checks in RLE encoding for bsdiff extra data. 
Add a buffer overflow check in RLE encoder.
Avoid calling RLE encoding if extra data length is zero.
Make 2KB block size default for non-global deduplication.
Update test cases for new 2KB block size support.
 2013-08-30 19:51:43 +05:30
Moinak Ghosh
9a7a8e84fe Add more example usage. 2013-08-28 21:01:25 +05:30
Moinak Ghosh
3db5188445 Support for deduplication using 2KB block size. 2013-08-19 13:38:52 +05:30
Moinak Ghosh
6432c76b4b Update README formatting yet again - ugh. 2013-06-16 21:12:04 +05:30
Moinak Ghosh
52723cbbac Update README formatting. 2013-06-16 21:09:04 +05:30
Moinak Ghosh
92be5a17f0 Update README with pointers to relevant analysis and documentation. 2013-06-16 20:46:17 +05:30
Moinak Ghosh
8b3761ee81 Update Changelog, docs and bump version for 2.1 release. 2013-05-09 18:53:11 +05:30
Moinak Ghosh
969e242b31 Update README with details of Global Dedupe block hash selection. 2013-05-06 23:50:56 +05:30
Moinak Ghosh
120877348c Use SHA256 for Global Dedupe chunk lookup hash by default. 
Allow changing Global Dedupe chunk lookup hash via env variable.
 2013-05-02 00:05:05 +05:30
Moinak Ghosh
b23b5789fb Fix bugs and improve accuracy in Segmented Dedupe. 
Fix segment hashlist size computation.
Remove unnecessary sync of segment hashlist file writes.
Pass correct number of threads to index creation routine.
Add more error checks.
Handle correct positioning of segment hashlist file offset on write error.
Add missing semaphore signaling at dedupe abort points with global dedupe.
Use closer min-values sampling for improved segmented dedupe accuracy.
Update proper checksum info in README.
 2013-04-30 19:35:18 +05:30
Moinak Ghosh
eb964b0bde Update README. 2013-04-26 18:46:14 +05:30
Moinak Ghosh
aed69b2d53 Add test cases for Global Deduplication. 
Update documentation and code comments.
Remove tempfile pathname after creation to ensure clean removal after process exit.
 2013-04-26 18:32:00 +05:30
Moinak Ghosh
79a6e7f770 Capability to output data to stdout when compressing. 
Always use segmented similarity bases dedupe when using -G option in pipe mode.
Standardize on average 8MB segment size for segmented dedupe.
Fix hashtable sizing.
Some miscellaneous cleanups.
Update README with details of new features.
 2013-04-24 23:03:58 +05:30
Moinak Ghosh
e41f156beb Update README and test cases with new crypto options. 
Update usage text.
 2013-03-05 21:07:54 +05:30
Moinak Ghosh
7a29c7be1e Change default encryption key length to 256 bits. 
Add optional ability to change key length at runtime via cli option.
Include key length property in archive header.
Fix header HMAC to include salt, nonce and key length properties.
Retain backward compatibility to handle older format archives.
Fix compilation of AES ASM code.
 2013-03-03 20:02:14 +05:30
Moinak Ghosh
976a12afbe Remove outdated LZP note. 2013-01-05 19:59:17 +05:30
Moinak Ghosh
16b1d9e7a3 Bump version, update Changelog and documentation for 1.2 release. 2013-01-03 23:40:21 +05:30
Moinak Ghosh
fb30b5c295 Enable building with alternate Zlib and Bzlib. 
Update README and comments.
Fix correct setting of output size when using Delta2 without LZP.
 2012-12-16 23:17:04 +05:30
Moinak Ghosh
5ac47db6d5 README formatting. 2012-12-15 22:09:07 +05:30
Moinak Ghosh
ef0191729e Make Delta2 encoding independent of LZP. 
Tweak Delta2 parameters.
Update README and test cases.
 2012-12-15 22:03:23 +05:30
Moinak Ghosh
b01d255f6c Update README to align with current features/behavior. 2012-12-15 12:42:46 +05:30
Moinak Ghosh
24d5368f7f Update README. 2012-12-05 00:12:29 +05:30
Moinak Ghosh
2909a3abff Update README. 2012-11-18 23:19:22 +05:30
Moinak Ghosh
45487469d1 Add another example invocation. 2012-10-21 12:34:34 +05:30
Moinak Ghosh
f2d7bea902 Support for chunk-level encryption using AES and Scrypt based PBE. 
Couple of minor fixes.
 2012-10-15 12:10:00 +05:30
Moinak Ghosh
2578676b1b Add support for SHA256 and SHA512 digests from OpenSSL library. 2012-10-03 22:43:02 +05:30
Moinak Ghosh
0019efbadb Adjust break pattern check mask for closer approximation to average block size. 
Remove unused structure member.
 2012-09-29 23:31:45 +05:30
Moinak Ghosh
8f8af7ed6b Update adaptive mode heuristic based on algorithms. 
Remove incorrect check in PPMd decompression code.
More refactoring of variable names.
 2012-09-27 22:29:08 +05:30
Moinak Ghosh
449dc35675 Speed up adaptive modes by using heuristics to select compression algorithm. 
Select similarity percentage based on dedupe block size for effectiveness.
 2012-09-26 19:47:32 +05:30
Moinak Ghosh
3544a8c708 Fix polynomial table computation. 
Change hashing and length bias to reduce hashtable bucket collisions.
Add support for user-selectable 60% or 40% similarity for Delta Compression.
Overall slight speedup.
 2012-09-24 22:20:27 +05:30
Moinak Ghosh
e3befd9e16 Add support for Fixed-Block deduplication. 
More refactoring of symbol names.
 2012-09-16 11:12:58 +05:30
Moinak Ghosh
b9355a5dcc Reduce dedupe loop checks for slight speed edge. 
Beginnings of Fixed-block dedupe.
Update variable name for clarity.
 2012-09-15 11:14:58 +05:30
Moinak Ghosh
117382c141 Update README to reflect current features. 2012-09-07 21:32:20 +05:30
Moinak Ghosh
4ba840b255 Add ASM version of Skein for x64 platforms with auto-detection 
Error checking for checksum flag when decompressing
Update comments and READMEs
 2012-09-01 14:40:15 +05:30
Moinak Ghosh
a222772940 Fix single chunk flag handling during decompression. 
Update docs.
 2012-08-27 22:24:23 +05:30
Moinak Ghosh
d75535bc7e Add support for libbsc a high-performance block sorting compressor. 
Enable external algorithm threading for single chunk compressed files.
Update docs.
 2012-08-27 21:51:55 +05:30
Moinak Ghosh
3b83bc2d4e Bump file version. 2012-08-26 15:01:18 +05:30
Moinak Ghosh
bf149e880d Add LZP Pre-Compression support ported from libbsc. 
Add generic pre-processing wrappers for future support of other pre-processors.
Clean up computation of Rabin block sizes.
Compute Rabin scratch space accurately to avoid RAM wastage.
 2012-08-23 22:58:44 +05:30
Moinak Ghosh
3851c9c6cc Delay allocation of per-thread chunks for performance and memory efficiency. 
Avoid allocating double-buffer for single-chunk files.
Introduce lzmaMt option to indicate multithreaded LZMA.
Update README.
 2012-08-18 22:00:14 +05:30
Moinak Ghosh
3150bdbed7 Implement secondary sketch based on character counts to refine similarity checksum. 
Proper checksum update for last block.
Update comments.
 2012-08-12 13:06:49 +05:30