greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
369 commits 3 branches 21 tags 7.6 MiB
Commit graph

369 commits

This branch
This branch
All branches
Author SHA1 Message Date
Moinak Ghosh
7bf967b572 Fix PackJPG library usage. PackJPG interface doc is incomplete, ugh! 
Handle the case where PackJPG expands the file rather than compressing.
 2013-11-26 21:24:01 +05:30
Moinak Ghosh
4923551570 Fix Dispack decoding. 2013-11-24 22:44:35 +05:30
Moinak Ghosh
0192790c02 Add Dispack filter with auto-detection of x86 executables in archive mode. 
More elaborate magic header based detection of 32-bit and 64-bit x86 binaries.
Always use fast-mode LZ4 in Adaptive modes.
 2013-11-24 19:45:58 +05:30
Moinak Ghosh
1e2c3e479a Optimize preprocessed compression and avoid a bunch of memory copies. 
Fix a crash.
Add a few more file types.
More comments.
 2013-11-22 20:44:26 +05:30
Moinak Ghosh
664c8ef75b Fix fd leak. 2013-11-15 23:06:31 +05:30
Moinak Ghosh
c09a2b7b81 Fix issues when handling Jpegs where packJPG borks. 2013-11-15 23:02:09 +05:30
Moinak Ghosh
11584cab52 Add fast handling of totally incompressible data (like Jpegs) in adaptive modes. 
Add function to indicate totally incompressible data when archiving.
Reformat if statements in some places to reduce branching.
 2013-11-15 21:06:23 +05:30
Moinak Ghosh
c567a1d2f5 Enable auto-filtering of archive entries based on compression level. 
Miscellaneous fixes.
 2013-11-14 21:54:46 +05:30
Moinak Ghosh
e90c52e516 Work in progress changes for packJPG encoding and decoding. 
Enhance custom LibArchive filter functionlity.
 2013-11-13 23:28:01 +05:30
Moinak Ghosh
75dfa6a6fb Add basic framework for file type based filters during libarchive stage. 
Add packJPG filter for Jpeg files (not active yet).
Directory format changes for clarity.
 2013-11-10 23:09:42 +05:30
Moinak Ghosh
a5f1624a33 Add own implementation of archive entry extraction to allow custom filters. 
Fix magic number check for endianness.
 2013-11-09 21:55:18 +05:30
Moinak Ghosh
6aacd903ff Structured handling of file types. 
Handling of already compressed data based on compression algorithm.
Add a few more extension types.
 2013-11-09 16:46:19 +05:30
Moinak Ghosh
cae9de9b2e Leverage file type detection(archiver) to improve compression performance. 
Use detected file/data type(archiver) for Adaptive compression modes.
Update type flags and add more extensions.
 2013-11-08 23:50:28 +05:30
Moinak Ghosh
b7facc929e Add file type detection based on magic values. 
Add more comments.
Add more extensions.
 2013-11-07 23:57:15 +05:30
Moinak Ghosh
991482403b Add extension based file type detection and setting segment data type. 
Use Bob Jenkins Minimal Perfect Hash to check for known extensions.
Use semaphore signaling and direct buffer copy for extraction.
Miscellaneous fixes.
 2013-11-07 21:48:54 +05:30
Moinak Ghosh
489b97cc79 Clear off private xattrs when extracting. 
Enable pathname sorting only for high compression levels.
 2013-11-04 18:35:22 +05:30
Moinak Ghosh
448890a014 Replace slow pipe with direct memory copy for archive extraction. 
Miscellaneous corrections and tweaks.
 2013-11-03 23:15:55 +05:30
Moinak Ghosh
7ed532133e Avoid using pipe during archive creation. Use semaphores and direct memory copy. 2013-11-02 23:43:59 +05:30
Moinak Ghosh
a374ca5909 Use mmap to read from the pathlist file for performance. 2013-11-02 12:14:46 +05:30
Moinak Ghosh
dcccffd7fa Archiving support using Libarchive: Fully functional archiving and extraction. 
Functionality to sort pathnames based on file extension and size.
 2013-11-01 23:15:40 +05:30
Moinak Ghosh
e09d8a485c Archiving support using Libarchive: Working archive extraction. 2013-10-31 00:15:17 +05:30
Moinak Ghosh
8e4b774c8c More changes for archiving. 
Allow multiple filenames on command line when archiving.
Remove unneded small block writes with libarchive.
 2013-10-27 20:36:48 +05:30
Moinak Ghosh
46b11def08 Archiving support using Libarchive: Work in progress changes #3. 
Make log_msg() add newline by default.
 2013-10-24 00:16:04 +05:30
Moinak Ghosh
bc451aba36 Archiving support using Libarchive: Work in progress changes #2. 2013-10-22 23:41:51 +05:30
Moinak Ghosh
7f81869874 Archiving support using Libarchive: Work in progress changes. 
Change all perror() calls to use logger.
Make the config script a little verbose.
 2013-10-20 23:54:27 +05:30
Moinak Ghosh
28fd9848f9 Ability to specify output compressed pathname. 
Fix log level handling.
Trim commented code.
 2013-10-10 21:19:44 +05:30
Moinak Ghosh
8c1f4ebe61 Add a simple log facility. 
Refactor all printfs to use log facility.
 2013-10-02 20:45:33 +05:30
Moinak Ghosh
fa78621cbf Cleanup pointer casting in code to use macros. 2013-09-22 20:11:15 +05:30
Moinak Ghosh
38c0869f5c Update Changelog and tweak free memory detection for 2.4 release. 
Add identifiers to error messages for clarity.
Fix init of dedupe block size.
Tweak free memory detection to include swap and shared memory consideration.
 2013-09-05 21:12:37 +05:30
Moinak Ghosh
a61fea75da Fix incorrect chunk size initialization from a previous commit. 2013-09-03 23:23:11 +05:30
Moinak Ghosh
b236638e72 Remove confusing option with little practical utility. 
Update test cases and documentation.
 2013-09-01 15:02:28 +05:30
Moinak Ghosh
12a2b8ed63 Additional error checks in RLE encoding for bsdiff extra data. 
Add a buffer overflow check in RLE encoder.
Avoid calling RLE encoding if extra data length is zero.
Make 2KB block size default for non-global deduplication.
Update test cases for new 2KB block size support.
 2013-08-30 19:51:43 +05:30
Moinak Ghosh
2e62be3c9c Truncate password file after zeroing. 2013-08-29 22:03:08 +05:30
Moinak Ghosh
9a7a8e84fe Add more example usage. 2013-08-28 21:01:25 +05:30
Moinak Ghosh
be1d0857a6 Avoid calling compression routine when dedupe reduces data size to zero. 2013-08-28 09:46:10 +05:30
Moinak Ghosh
cee8d88ded Bump version for upcoming release. 2013-08-27 21:41:16 +05:30
Moinak Ghosh
7685adefb2 Default compression level only when compressing. 2013-08-24 23:15:07 +05:30
Moinak Ghosh
fc65111bae Fix issue #11. 
Increase default chunk size to 8MB.
Use default compression level of 1 (fast mode) for LZ4.
 2013-08-24 22:58:50 +05:30
Moinak Ghosh
3db5188445 Support for deduplication using 2KB block size. 2013-08-19 13:38:52 +05:30
Moinak Ghosh
ef98422bd4 Add basic file format documentation. 
Reduce memory threshold for switching to Similarity based Deduplication.
 2013-08-18 20:11:20 +05:30
Moinak Ghosh
58f3113558 Avoid unnecessary re-hashing of 64-bit keys of the segment index. 2013-08-17 22:08:55 +05:30
Moinak Ghosh
d31c6433c2 Update free memory computation to include cached buffers. 
Fix a potential rare corner case.
 2013-08-17 11:31:44 +05:30
Moinak Ghosh
413a2a2fb1 Update Changelog and bump version for 2.3 release. 2013-08-10 10:25:41 +05:30
Moinak Ghosh
f35d0ff4ef Fix multiple crashes for some corner cases. 
Increase max block size for variable dedup block sizes greater than 16KB.
Update test cases and fix a test script bug.
 2013-08-09 21:55:06 +05:30
Moinak Ghosh
fe18afbcf4 Use wrapper script to set paths when launching pcompress from build directory. 
Use smaller max block size when doing global dedupe.
Fix init of executable name.
 2013-08-07 22:03:52 +05:30
Moinak Ghosh
f34cfb1aa6 Make data partitioning between threads more effective. 
Remove unnecessary computation to make Fixed block chunking faster.
 2013-07-21 09:31:59 +05:30
Moinak Ghosh
2a218e9da5 Fix Dedupe Mode initialization. 2013-07-12 18:21:49 +05:30
Moinak Ghosh
8b73303488 Some minor code cleanup. 2013-07-05 22:22:11 +05:30
Moinak Ghosh
e10a13ad94 Improve accuracy of the KMV sketch computation and speed it up. 2013-07-03 19:24:06 +05:30
Moinak Ghosh
6b67e98747 Reduce similarity indicators to reduce memory use with low impact on dedupe ratio. 2013-06-30 22:38:05 +05:30