greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
428 commits 3 branches 21 tags 7.6 MiB
Commit graph

428 commits

This branch
This branch
All branches
Author SHA1 Message Date
Moinak Ghosh
5e484f0694 Use libbsc for AVI and MP4 files. 2013-12-04 20:07:52 +05:30
Moinak Ghosh
3f62cdf7d5 Use Libbsc for MP4 and FLAC files. 
Change some rare file type codes to indicate some common types.
 2013-12-03 21:56:07 +05:30
Moinak Ghosh
958bdf7edc Use Libbsc for TIFF images. 
Workaround for packJPG limitation.
 2013-12-02 21:50:19 +05:30
Moinak Ghosh
5a49252bb9 Remove external Libbsc option. 2013-11-30 22:43:31 +05:30
Moinak Ghosh
fb25e53b4f Add forked and optimized copy of LGPL version of Libbsc. 
Strip out Sort Transform from Libbsc copy.
Reduce Libbsc memory use.
Avoid redundant adler32 of data block in Libbsc.
 2013-11-30 22:13:33 +05:30
Moinak Ghosh
c4c4b47138 Use Libbsc for DNA Sequence data instead of PPMD. Faster, better compression. 
Fix pz extension handling for real.
 2013-11-30 09:58:21 +05:30
Moinak Ghosh
dfeea8c19b Avoid Delta2,LZP for TIFF files. Negatively impacts compression. 2013-11-29 19:47:57 +05:30
Moinak Ghosh
306f145f22 Use libbsc/ppmd for BMP files. 
Fix extension based hashing.
Do not append .pz extension to filenames already having it.
Some code formatting changes.
 2013-11-28 22:42:51 +05:30
Moinak Ghosh
bd530e3393 Get rid of nagging warning. 2013-11-26 21:43:17 +05:30
Moinak Ghosh
7bf967b572 Fix PackJPG library usage. PackJPG interface doc is incomplete, ugh! 
Handle the case where PackJPG expands the file rather than compressing.
 2013-11-26 21:24:01 +05:30
Moinak Ghosh
4923551570 Fix Dispack decoding. 2013-11-24 22:44:35 +05:30
Moinak Ghosh
0192790c02 Add Dispack filter with auto-detection of x86 executables in archive mode. 
More elaborate magic header based detection of 32-bit and 64-bit x86 binaries.
Always use fast-mode LZ4 in Adaptive modes.
 2013-11-24 19:45:58 +05:30
Moinak Ghosh
1e2c3e479a Optimize preprocessed compression and avoid a bunch of memory copies. 
Fix a crash.
Add a few more file types.
More comments.
 2013-11-22 20:44:26 +05:30
Moinak Ghosh
664c8ef75b Fix fd leak. 2013-11-15 23:06:31 +05:30
Moinak Ghosh
c09a2b7b81 Fix issues when handling Jpegs where packJPG borks. 2013-11-15 23:02:09 +05:30
Moinak Ghosh
11584cab52 Add fast handling of totally incompressible data (like Jpegs) in adaptive modes. 
Add function to indicate totally incompressible data when archiving.
Reformat if statements in some places to reduce branching.
 2013-11-15 21:06:23 +05:30
Moinak Ghosh
c567a1d2f5 Enable auto-filtering of archive entries based on compression level. 
Miscellaneous fixes.
 2013-11-14 21:54:46 +05:30
Moinak Ghosh
e90c52e516 Work in progress changes for packJPG encoding and decoding. 
Enhance custom LibArchive filter functionlity.
 2013-11-13 23:28:01 +05:30
Moinak Ghosh
75dfa6a6fb Add basic framework for file type based filters during libarchive stage. 
Add packJPG filter for Jpeg files (not active yet).
Directory format changes for clarity.
 2013-11-10 23:09:42 +05:30
Moinak Ghosh
a5f1624a33 Add own implementation of archive entry extraction to allow custom filters. 
Fix magic number check for endianness.
 2013-11-09 21:55:18 +05:30
Moinak Ghosh
6aacd903ff Structured handling of file types. 
Handling of already compressed data based on compression algorithm.
Add a few more extension types.
 2013-11-09 16:46:19 +05:30
Moinak Ghosh
cae9de9b2e Leverage file type detection(archiver) to improve compression performance. 
Use detected file/data type(archiver) for Adaptive compression modes.
Update type flags and add more extensions.
 2013-11-08 23:50:28 +05:30
Moinak Ghosh
b7facc929e Add file type detection based on magic values. 
Add more comments.
Add more extensions.
 2013-11-07 23:57:15 +05:30
Moinak Ghosh
991482403b Add extension based file type detection and setting segment data type. 
Use Bob Jenkins Minimal Perfect Hash to check for known extensions.
Use semaphore signaling and direct buffer copy for extraction.
Miscellaneous fixes.
 2013-11-07 21:48:54 +05:30
Moinak Ghosh
489b97cc79 Clear off private xattrs when extracting. 
Enable pathname sorting only for high compression levels.
 2013-11-04 18:35:22 +05:30
Moinak Ghosh
448890a014 Replace slow pipe with direct memory copy for archive extraction. 
Miscellaneous corrections and tweaks.
 2013-11-03 23:15:55 +05:30
Moinak Ghosh
7ed532133e Avoid using pipe during archive creation. Use semaphores and direct memory copy. 2013-11-02 23:43:59 +05:30
Moinak Ghosh
a374ca5909 Use mmap to read from the pathlist file for performance. 2013-11-02 12:14:46 +05:30
Moinak Ghosh
dcccffd7fa Archiving support using Libarchive: Fully functional archiving and extraction. 
Functionality to sort pathnames based on file extension and size.
 2013-11-01 23:15:40 +05:30
Moinak Ghosh
e09d8a485c Archiving support using Libarchive: Working archive extraction. 2013-10-31 00:15:17 +05:30
Moinak Ghosh
8e4b774c8c More changes for archiving. 
Allow multiple filenames on command line when archiving.
Remove unneded small block writes with libarchive.
 2013-10-27 20:36:48 +05:30
Moinak Ghosh
46b11def08 Archiving support using Libarchive: Work in progress changes #3. 
Make log_msg() add newline by default.
 2013-10-24 00:16:04 +05:30
Moinak Ghosh
bc451aba36 Archiving support using Libarchive: Work in progress changes #2. 2013-10-22 23:41:51 +05:30
Moinak Ghosh
7f81869874 Archiving support using Libarchive: Work in progress changes. 
Change all perror() calls to use logger.
Make the config script a little verbose.
 2013-10-20 23:54:27 +05:30
Moinak Ghosh
28fd9848f9 Ability to specify output compressed pathname. 
Fix log level handling.
Trim commented code.
 2013-10-10 21:19:44 +05:30
Moinak Ghosh
8c1f4ebe61 Add a simple log facility. 
Refactor all printfs to use log facility.
 2013-10-02 20:45:33 +05:30
Moinak Ghosh
fa78621cbf Cleanup pointer casting in code to use macros. 2013-09-22 20:11:15 +05:30
Moinak Ghosh
38c0869f5c Update Changelog and tweak free memory detection for 2.4 release. 
Add identifiers to error messages for clarity.
Fix init of dedupe block size.
Tweak free memory detection to include swap and shared memory consideration.
 2013-09-05 21:12:37 +05:30
Moinak Ghosh
a61fea75da Fix incorrect chunk size initialization from a previous commit. 2013-09-03 23:23:11 +05:30
Moinak Ghosh
b236638e72 Remove confusing option with little practical utility. 
Update test cases and documentation.
 2013-09-01 15:02:28 +05:30
Moinak Ghosh
12a2b8ed63 Additional error checks in RLE encoding for bsdiff extra data. 
Add a buffer overflow check in RLE encoder.
Avoid calling RLE encoding if extra data length is zero.
Make 2KB block size default for non-global deduplication.
Update test cases for new 2KB block size support.
 2013-08-30 19:51:43 +05:30
Moinak Ghosh
2e62be3c9c Truncate password file after zeroing. 2013-08-29 22:03:08 +05:30
Moinak Ghosh
9a7a8e84fe Add more example usage. 2013-08-28 21:01:25 +05:30
Moinak Ghosh
be1d0857a6 Avoid calling compression routine when dedupe reduces data size to zero. 2013-08-28 09:46:10 +05:30
Moinak Ghosh
cee8d88ded Bump version for upcoming release. 2013-08-27 21:41:16 +05:30
Moinak Ghosh
7685adefb2 Default compression level only when compressing. 2013-08-24 23:15:07 +05:30
Moinak Ghosh
fc65111bae Fix issue #11. 
Increase default chunk size to 8MB.
Use default compression level of 1 (fast mode) for LZ4.
 2013-08-24 22:58:50 +05:30
Moinak Ghosh
3db5188445 Support for deduplication using 2KB block size. 2013-08-19 13:38:52 +05:30
Moinak Ghosh
ef98422bd4 Add basic file format documentation. 
Reduce memory threshold for switching to Similarity based Deduplication.
 2013-08-18 20:11:20 +05:30
Moinak Ghosh
58f3113558 Avoid unnecessary re-hashing of 64-bit keys of the segment index. 2013-08-17 22:08:55 +05:30