greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
335 commits 3 branches 21 tags 7.6 MiB
Commit graph

46 commits

Author SHA1 Message Date
Moinak Ghosh
a851bac247 Check harder with more strides in Delta2 for extreme compression levels. 2013-12-13 19:53:14 +05:30
Moinak Ghosh
bb08b24989 Make LibArchive filter process buffer more generic. 
Include explicit CLI flags for PackJPG and Dispack.
Avoid auto-selection of filters if advanced options are specified.
 2013-12-12 00:22:15 +05:30
Moinak Ghosh
393fd790b0 Add more robust checks for Jpeg and packJPG format files in filter routine. 
Use case-insensitive checks for extension names.
Enable more features based on compression level, when archiving.
 2013-12-08 23:24:06 +05:30
Moinak Ghosh
36ed5d5a78 Use adapt2 as default compression in archive mode. 
Add more filter auto-selection by compression level in archive mode.
Replace odd stride lengths in Delta2 with standard numeric type lengths and improve performance.
 2013-12-05 22:20:01 +05:30
Moinak Ghosh
5e484f0694 Use libbsc for AVI and MP4 files. 2013-12-04 20:07:52 +05:30
Moinak Ghosh
c4c4b47138 Use Libbsc for DNA Sequence data instead of PPMD. Faster, better compression. 
Fix pz extension handling for real.
 2013-11-30 09:58:21 +05:30
Moinak Ghosh
dfeea8c19b Avoid Delta2,LZP for TIFF files. Negatively impacts compression. 2013-11-29 19:47:57 +05:30
Moinak Ghosh
306f145f22 Use libbsc/ppmd for BMP files. 
Fix extension based hashing.
Do not append .pz extension to filenames already having it.
Some code formatting changes.
 2013-11-28 22:42:51 +05:30
Moinak Ghosh
4923551570 Fix Dispack decoding. 2013-11-24 22:44:35 +05:30
Moinak Ghosh
0192790c02 Add Dispack filter with auto-detection of x86 executables in archive mode. 
More elaborate magic header based detection of 32-bit and 64-bit x86 binaries.
Always use fast-mode LZ4 in Adaptive modes.
 2013-11-24 19:45:58 +05:30
Moinak Ghosh
1e2c3e479a Optimize preprocessed compression and avoid a bunch of memory copies. 
Fix a crash.
Add a few more file types.
More comments.
 2013-11-22 20:44:26 +05:30
Moinak Ghosh
11584cab52 Add fast handling of totally incompressible data (like Jpegs) in adaptive modes. 
Add function to indicate totally incompressible data when archiving.
Reformat if statements in some places to reduce branching.
 2013-11-15 21:06:23 +05:30
Moinak Ghosh
c567a1d2f5 Enable auto-filtering of archive entries based on compression level. 
Miscellaneous fixes.
 2013-11-14 21:54:46 +05:30
Moinak Ghosh
e90c52e516 Work in progress changes for packJPG encoding and decoding. 
Enhance custom LibArchive filter functionlity.
 2013-11-13 23:28:01 +05:30
Moinak Ghosh
6aacd903ff Structured handling of file types. 
Handling of already compressed data based on compression algorithm.
Add a few more extension types.
 2013-11-09 16:46:19 +05:30
Moinak Ghosh
cae9de9b2e Leverage file type detection(archiver) to improve compression performance. 
Use detected file/data type(archiver) for Adaptive compression modes.
Update type flags and add more extensions.
 2013-11-08 23:50:28 +05:30
Moinak Ghosh
991482403b Add extension based file type detection and setting segment data type. 
Use Bob Jenkins Minimal Perfect Hash to check for known extensions.
Use semaphore signaling and direct buffer copy for extraction.
Miscellaneous fixes.
 2013-11-07 21:48:54 +05:30
Moinak Ghosh
489b97cc79 Clear off private xattrs when extracting. 
Enable pathname sorting only for high compression levels.
 2013-11-04 18:35:22 +05:30
Moinak Ghosh
448890a014 Replace slow pipe with direct memory copy for archive extraction. 
Miscellaneous corrections and tweaks.
 2013-11-03 23:15:55 +05:30
Moinak Ghosh
7ed532133e Avoid using pipe during archive creation. Use semaphores and direct memory copy. 2013-11-02 23:43:59 +05:30
Moinak Ghosh
a374ca5909 Use mmap to read from the pathlist file for performance. 2013-11-02 12:14:46 +05:30
Moinak Ghosh
dcccffd7fa Archiving support using Libarchive: Fully functional archiving and extraction. 
Functionality to sort pathnames based on file extension and size.
 2013-11-01 23:15:40 +05:30
Moinak Ghosh
e09d8a485c Archiving support using Libarchive: Working archive extraction. 2013-10-31 00:15:17 +05:30
Moinak Ghosh
8e4b774c8c More changes for archiving. 
Allow multiple filenames on command line when archiving.
Remove unneded small block writes with libarchive.
 2013-10-27 20:36:48 +05:30
Moinak Ghosh
46b11def08 Archiving support using Libarchive: Work in progress changes #3. 
Make log_msg() add newline by default.
 2013-10-24 00:16:04 +05:30
Moinak Ghosh
bc451aba36 Archiving support using Libarchive: Work in progress changes #2. 2013-10-22 23:41:51 +05:30
Moinak Ghosh
7f81869874 Archiving support using Libarchive: Work in progress changes. 
Change all perror() calls to use logger.
Make the config script a little verbose.
 2013-10-20 23:54:27 +05:30
Moinak Ghosh
28fd9848f9 Ability to specify output compressed pathname. 
Fix log level handling.
Trim commented code.
 2013-10-10 21:19:44 +05:30
Moinak Ghosh
8c1f4ebe61 Add a simple log facility. 
Refactor all printfs to use log facility.
 2013-10-02 20:45:33 +05:30
Moinak Ghosh
fa78621cbf Cleanup pointer casting in code to use macros. 2013-09-22 20:11:15 +05:30
Moinak Ghosh
38c0869f5c Update Changelog and tweak free memory detection for 2.4 release. 
Add identifiers to error messages for clarity.
Fix init of dedupe block size.
Tweak free memory detection to include swap and shared memory consideration.
 2013-09-05 21:12:37 +05:30
Moinak Ghosh
a61fea75da Fix incorrect chunk size initialization from a previous commit. 2013-09-03 23:23:11 +05:30
Moinak Ghosh
b236638e72 Remove confusing option with little practical utility. 
Update test cases and documentation.
 2013-09-01 15:02:28 +05:30
Moinak Ghosh
12a2b8ed63 Additional error checks in RLE encoding for bsdiff extra data. 
Add a buffer overflow check in RLE encoder.
Avoid calling RLE encoding if extra data length is zero.
Make 2KB block size default for non-global deduplication.
Update test cases for new 2KB block size support.
 2013-08-30 19:51:43 +05:30
Moinak Ghosh
2e62be3c9c Truncate password file after zeroing. 2013-08-29 22:03:08 +05:30
Moinak Ghosh
be1d0857a6 Avoid calling compression routine when dedupe reduces data size to zero. 2013-08-28 09:46:10 +05:30
Moinak Ghosh
7685adefb2 Default compression level only when compressing. 2013-08-24 23:15:07 +05:30
Moinak Ghosh
fc65111bae Fix issue #11. 
Increase default chunk size to 8MB.
Use default compression level of 1 (fast mode) for LZ4.
 2013-08-24 22:58:50 +05:30
Moinak Ghosh
3db5188445 Support for deduplication using 2KB block size. 2013-08-19 13:38:52 +05:30
Moinak Ghosh
f35d0ff4ef Fix multiple crashes for some corner cases. 
Increase max block size for variable dedup block sizes greater than 16KB.
Update test cases and fix a test script bug.
 2013-08-09 21:55:06 +05:30
Moinak Ghosh
fe18afbcf4 Use wrapper script to set paths when launching pcompress from build directory. 
Use smaller max block size when doing global dedupe.
Fix init of executable name.
 2013-08-07 22:03:52 +05:30
Moinak Ghosh
f34cfb1aa6 Make data partitioning between threads more effective. 
Remove unnecessary computation to make Fixed block chunking faster.
 2013-07-21 09:31:59 +05:30
Moinak Ghosh
de0695e2c5 Add missing init of rabin block size. 2013-06-29 19:13:22 +05:30
Moinak Ghosh
c0dd0102a5 A few minor fixes. 2013-06-14 22:25:01 +05:30
Moinak Ghosh
7743792018 Make default symbol visibility to hidden with explicit public visibility specified. 
Add missing static scope to a few more places.
 2013-06-03 20:51:00 +05:30
Moinak Ghosh
c859cf35d5 Make Pcompress functionality into a library - initial changes. 2013-06-02 20:54:33 +05:30