greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Commit graph

  • 26a4f42506 Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing. Moinak Ghosh 2012-12-27 23:06:48 +0530
  • 063624d0d3 Fix Delta2 decode handling when not compressing. More debug statements and other minor changes. Moinak Ghosh 2012-12-24 23:21:02 +0530
  • d597f0f05c Major enhancements to Delta2 encoding. Avoid transposing below-threshold spans. Reduces compression ratio. Use little-endian storage format for numbers to optimize for x86. Improve embedded table detection. Reduce header sizes. Get rid of Gcc's LTO flag. Causes a performance drop. Fix preprocessing behavior when LZP does not compress but Delta2 works. Moinak Ghosh 2012-12-23 23:50:45 +0530
  • a43fdd7d2c Improve Delta2 scanning speed and effectiveness. Add destination buffer overflow check in Delta2. Add rough speed computation. Moinak Ghosh 2012-12-23 00:44:56 +0530
  • 765b014018 More tweaks to Delta2 implementation. Moinak Ghosh 2012-12-21 22:58:30 +0530
  • 9bd8146e88 Improve Delta2 performance. Moinak Ghosh 2012-12-20 23:49:44 +0530
  • e9c3b34ff5 Add debug build verbose messages to delta code. Moinak Ghosh 2012-12-19 22:42:55 +0530
  • c2df666daf Major improvement to Delta2 encoding algorithm. Moinak Ghosh 2012-12-18 23:32:55 +0530
  • 1fe5157eec Ability to run individual test suites from makefile. Moinak Ghosh 2012-12-17 20:59:27 +0530
  • 00c8c534d0 Silence more Gcc warnings. Update testing note. Moinak Ghosh 2012-12-17 20:23:30 +0530
  • 25bb54e3bc Improve 64-bit compiler and platform checks. Ensure core dumps are enabled during testing. Moinak Ghosh 2012-12-17 18:37:27 +0530
  • fb30b5c295 Enable building with alternate Zlib and Bzlib. Update README and comments. Fix correct setting of output size when using Delta2 without LZP. Moinak Ghosh 2012-12-16 23:17:04 +0530
  • 5ac47db6d5 README formatting. Moinak Ghosh 2012-12-15 22:09:07 +0530
  • ef0191729e Make Delta2 encoding independent of LZP. Tweak Delta2 parameters. Update README and test cases. Moinak Ghosh 2012-12-15 22:03:23 +0530
  • b01d255f6c Update README to align with current features/behavior. Moinak Ghosh 2012-12-15 12:42:46 +0530
  • a98778d62f Fine tune transpose parameters. Fix minor nits. Moinak Ghosh 2012-12-14 19:12:48 +0530
  • b0f41c2888 Add matrix transpose to Delta2 encoding. Change confusing structure member name. Moinak Ghosh 2012-12-13 21:18:16 +0530
  • 375ebefa0d Add Matrix Transpose of Dedupe index to compress it better. Fix handling of Dedupe index compression failure. Moinak Ghosh 2012-12-13 00:00:47 +0530
  • c7b960e72c Enable use of Link Time Optimizations in GCC. Moinak Ghosh 2012-12-11 17:58:23 +0530
  • 42858c10a2 Ensure intermediate file cleanup in tests. Moinak Ghosh 2012-12-11 16:23:35 +0530
  • 03840b31c5 Update to latest LZ4. Update a couple of comments. Moinak Ghosh 2012-12-11 11:38:42 +0530
  • 224fb529e9 Get rid of size_t in places where 64-bitness is assumed. Moinak Ghosh 2012-12-09 10:15:06 +0530
  • 75c81b5e9c Add support for 64-bit Keccak implementation. Sanitize error message from tests. Add more tests. Improve platform detection in config script. Moinak Ghosh 2012-12-08 14:19:01 +0530
  • 970d8163a2 Add adaptive delta encoding test. Moinak Ghosh 2012-12-05 22:03:09 +0530
  • 24d5368f7f Update README. Moinak Ghosh 2012-12-05 00:12:29 +0530
  • 29b0d8fd7b Implement Adaptive Delta Encoding. Moinak Ghosh 2012-12-05 00:09:47 +0530
  • 51249c858d Work in progress Adaptive Delta Encoding. Moinak Ghosh 2012-12-03 23:45:41 +0530
  • 6c3173f929 Work in progress global dedupe config setup. Moinak Ghosh 2012-11-29 22:28:50 +0530
  • 8583818b8a Create gh-pages branch via GitHub Moinak Ghosh 2012-11-28 08:32:19 -0800
  • 02196bc61e Create gh-pages branch via GitHub Moinak Ghosh 2012-11-27 06:30:44 -0800
  • 139381231a Create gh-pages branch via GitHub Moinak Ghosh 2012-11-26 09:10:41 -0800
  • 1f0c237495 Update Change and bump version for 1.1 release. 1.1Stable Moinak Ghosh 2012-11-26 20:27:09 +0530
  • 33c727e6e7 Fix building without Libbsc support. Add more tests for corrupted encrypted files. Moinak Ghosh 2012-11-26 20:21:03 +0530
  • 386d559f1e Create gh-pages branch via GitHub Moinak Ghosh 2012-11-26 06:20:08 -0800
  • 250c51ca9e Create gh-pages branch via GitHub Moinak Ghosh 2012-11-25 01:56:17 -0800
  • deecbadac4 Fix thread error reporting. Update error condition tests to not truncate archive. Moinak Ghosh 2012-11-25 15:12:45 +0530
  • d250322490 Fix issues with error handling. Add new tests for out of range values and corrupted file. Moinak Ghosh 2012-11-24 23:53:07 +0530
  • bdf16c4cb9 Avoid CRC32 when decompressing older version archives. Moinak Ghosh 2012-11-24 00:00:05 +0530
  • 817e16cce6 Use HMAC on header and encrypted data, avoid regular digest when encrypting. Moinak Ghosh 2012-11-23 22:27:14 +0530
  • 6e9ca97bdf Add CRC32 header checksums in non-cryptographic mode. Moinak Ghosh 2012-11-23 00:00:39 +0530
  • d054e0f713 Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature. Moinak Ghosh 2012-11-22 21:02:50 +0530
  • 3b1d6b55fe Work in progress global dedupe config loader. Moinak Ghosh 2012-11-19 21:41:56 +0530
  • 2909a3abff Update README. Moinak Ghosh 2012-11-18 23:19:22 +0530
  • a114da2aa4 Create gh-pages branch via GitHub Moinak Ghosh 2012-11-18 09:37:54 -0800
  • 393ced991a A couple of minor cleanups. Moinak Ghosh 2012-11-18 20:20:16 +0530
  • a19e7182b3 Create gh-pages branch via GitHub Moinak Ghosh 2012-11-18 04:40:55 -0800
  • b2cbf0699e Use fixed rolling-hash mask for better block size approximation. Moinak Ghosh 2012-11-11 14:58:39 +0530
  • 4d2e8ba644 Update Changelog for 1.0 release. Update version string and install doc. Add Fixed-Block Deduplication tests. 1.0Stable Moinak Ghosh 2012-11-09 21:42:56 +0530
  • f9a15972d6 Add some more crypto test cases. Moinak Ghosh 2012-11-09 21:17:06 +0530
  • 5b1a13480b Fix chunk flag setup when compression fails in adaptive mode. Prevent display of non-fatal errors during compression. Moinak Ghosh 2012-11-09 21:07:23 +0530
  • 77db54e712 Add buffer overflow check in Ppmd compression routines. Fix pipe mode encryption check. Change file difference check in tests. Add more tests. Moinak Ghosh 2012-11-09 20:05:13 +0530
  • da9083ae12 Fix encryption with adaptive modes. Add missing zero-out of algorithm data field. Moinak Ghosh 2012-11-09 18:53:48 +0530
  • 7debc60dd5 Add a functionality test suite. Moinak Ghosh 2012-11-08 22:22:37 +0530
  • eacbf207aa Tweak chunking parameters for better block size distribution and dedupe ratio. Moinak Ghosh 2012-11-08 19:41:33 +0530
  • e437390e53 Add some more debug mode info. Moinak Ghosh 2012-11-07 22:55:14 +0530
  • 3aa33f5b94 Minor fix for adapt mode. Moinak Ghosh 2012-11-04 21:46:04 +0530
  • f0c7ba87a3 Use Libbsc for XML data in adapt2 mode. Moinak Ghosh 2012-11-04 21:13:26 +0530
  • ddda9e5320 Bump version to 0.9.1 Update Changelog. 0.9.1Beta Moinak Ghosh 2012-10-24 18:46:27 +0530
  • c5ebe1f30a Portability to Debian based distros. Enable SSE4/AVX detection for AMD platforms (Bulldozer has both). Portable long long int print formatting to silence gcc 4.6 warnings. Moinak Ghosh 2012-10-21 21:03:07 +0530
  • 45487469d1 Add another example invocation. Moinak Ghosh 2012-10-21 12:34:34 +0530
  • 189d32ea52 Bump version to 0.9.0. Update Changelog. 0.9.0Beta Moinak Ghosh 2012-10-20 23:10:05 +0530
  • 315bb50264 Change chunk HMAC to only check header, not full data. Digest in hdr int turn validates data. Moinak Ghosh 2012-10-19 21:51:27 +0530
  • 1eb8c94fed Implement HMAC verification for chunk data and header. Do not ask for decryption passwd twice. Moinak Ghosh 2012-10-18 22:55:41 +0530
  • d13aad830e Implement HMAC Functionality. Refactor crypto utility functions in a separate file. Add HMAC verification for file header. Add a few extra input validation checks. Include space in chunk header for HMAC. Moinak Ghosh 2012-10-17 23:32:35 +0530
  • f2d7bea902 Support for chunk-level encryption using AES and Scrypt based PBE. Couple of minor fixes. Moinak Ghosh 2012-10-15 12:10:00 +0530
  • 21cbef6d60 Incorporate SSE/AVX optimized Intel SHA-256 implementation. Add support for runtime cpuid detection. Moinak Ghosh 2012-10-06 20:51:23 +0530
  • c880b73d26 Fix typo. Moinak Ghosh 2012-10-03 22:47:16 +0530
  • 2578676b1b Add support for SHA256 and SHA512 digests from OpenSSL library. Moinak Ghosh 2012-10-03 22:43:02 +0530
  • 3492949aff Bump version to 0.8.6 Update Changelog 0.8.6Beta Moinak Ghosh 2012-10-02 10:24:32 +0530
  • 9475ccc3d6 Fix polynomial computation. Fix incorrect block length when doing fixed-block dedupe. Moinak Ghosh 2012-10-02 00:20:44 +0530
  • 0019efbadb Adjust break pattern check mask for closer approximation to average block size. Remove unused structure member. Moinak Ghosh 2012-09-29 23:31:45 +0530
  • 24e6f4e629 Switch to multiplicative rolling hash for good distribution properties. Moinak Ghosh 2012-09-29 00:09:49 +0530
  • d94be4e314 Update Changelog and bump version to 0.8.5. 0.8.5Beta Moinak Ghosh 2012-09-27 23:16:13 +0530
  • 8f8af7ed6b Update adaptive mode heuristic based on algorithms. Remove incorrect check in PPMd decompression code. More refactoring of variable names. Moinak Ghosh 2012-09-27 22:29:08 +0530
  • 449dc35675 Speed up adaptive modes by using heuristics to select compression algorithm. Select similarity percentage based on dedupe block size for effectiveness. Moinak Ghosh 2012-09-26 19:47:32 +0530
  • 333b7b011e Fix check for size reduction in dedupe. Tweak debug message for clarity. Moinak Ghosh 2012-09-25 23:24:33 +0530
  • 3544a8c708 Fix polynomial table computation. Change hashing and length bias to reduce hashtable bucket collisions. Add support for user-selectable 60% or 40% similarity for Delta Compression. Overall slight speedup. Moinak Ghosh 2012-09-24 22:20:27 +0530
  • 8386e72566 Rewrite core dedupe logic to simplify code and improve performance. Hashtable based chunk-level deduplication instead of Quicksort. Fix a corner case bug in Dedupe decompression. Moinak Ghosh 2012-09-23 14:57:09 +0530
  • 99a8e4cd98 Speed up Hash computation for dedupe blocks. Add missing initialization of sliding window. Update help text. Moinak Ghosh 2012-09-19 20:29:44 +0530
  • e3befd9e16 Add support for Fixed-Block deduplication. More refactoring of symbol names. Moinak Ghosh 2012-09-16 11:12:58 +0530
  • b9355a5dcc Reduce dedupe loop checks for slight speed edge. Beginnings of Fixed-block dedupe. Update variable name for clarity. Moinak Ghosh 2012-09-15 11:14:58 +0530
  • a6b3719d89 Fix conditional in heapq function. Moinak Ghosh 2012-09-11 21:59:08 +0530
  • f3f472b860 Implement K-min-values Sketch for Similarity detection. Moinak Ghosh 2012-09-11 20:26:36 +0530
  • 117382c141 Update README to reflect current features. Moinak Ghosh 2012-09-07 21:32:20 +0530
  • 05a010a9dd Bump version to 0.8.1 Update Changelog for 0.8.1 release. 0.8.1Beta Moinak Ghosh 2012-09-07 19:38:36 +0530
  • fb0aef0bd6 Fix return code handling in LZP pre-compression, crashed adaptive modes. Moinak Ghosh 2012-09-07 19:31:35 +0530
  • e6f042aaf8 Allow user-specified minimum Dedupe block size. Compute similarity sketch only if Delta Compression enabled. Moinak Ghosh 2012-09-05 22:43:54 +0530
  • 560fa85aab Fix secondary sketch computation, some more accuracy in diff detection. Moinak Ghosh 2012-09-04 23:28:02 +0530
  • 262566b59a Add xxHash for Rabin block checksums, slightly faster than CRC64. Fix missing initialization of character counts table. Some file reorganization. Moinak Ghosh 2012-09-02 20:40:32 +0530
  • 4ba840b255 Add ASM version of Skein for x64 platforms with auto-detection Error checking for checksum flag when decompressing Update comments and READMEs Moinak Ghosh 2012-09-01 14:40:15 +0530
  • eda312ce1e Add support for Skein512 and Skein256 checksums Import Skein code from NIST CD submission Make checksum algorithms pluggable Fix handling of huge buffers (>2GB) in LZP Cleanup of some buffer sizing code Speed up CRC64 calculation in dedupe chunking Moinak Ghosh 2012-08-31 22:36:06 +0530
  • f03834278a Bump version to 0.8. Update Changelog for 0.8 beta release. 0.8Beta Moinak Ghosh 2012-08-27 22:53:39 +0530
  • a222772940 Fix single chunk flag handling during decompression. Update docs. Moinak Ghosh 2012-08-27 22:24:23 +0530
  • d75535bc7e Add support for libbsc a high-performance block sorting compressor. Enable external algorithm threading for single chunk compressed files. Update docs. Moinak Ghosh 2012-08-27 21:51:55 +0530
  • 3b83bc2d4e Bump file version. Moinak Ghosh 2012-08-26 15:01:18 +0530
  • 5f41057f9c Add config script to generate Makefile based on flags. Add install target and installation readme file. A few comment changes. Moinak Ghosh 2012-08-26 14:09:24 +0530
  • d4e9cd0140 Improve memory efficiency when total file size < total size of chunks. Fix freeing of Zlib structures. Moinak Ghosh 2012-08-24 20:16:21 +0530
  • bf149e880d Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage. Moinak Ghosh 2012-08-23 22:58:44 +0530
  • 3851c9c6cc Delay allocation of per-thread chunks for performance and memory efficiency. Avoid allocating double-buffer for single-chunk files. Introduce lzmaMt option to indicate multithreaded LZMA. Update README. Moinak Ghosh 2012-08-18 22:00:14 +0530
  • 9eac774eb1 Add multithreaded LZMA port from p7zip Compute balanced thread count between chunk threads and algo threads Generic way to handle querying algorithm parameters Clean up unnecessary includes Moinak Ghosh 2012-08-18 10:20:52 +0530