greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
429 commits 3 branches 21 tags 7.6 MiB
Commit graph

429 commits

This branch
This branch
All branches
Author SHA1 Message Date
vales
9f69afaa2f Adds missing check for pctx->main_cancel 2015-07-23 11:36:55 +05:00
Moinak Ghosh
c52306fe6d Merge pull request #38 from szepeviktor/patch-1 
Updating options
 2015-06-14 16:13:00 +05:30
Viktor Szépe
3459026b36 Updating options 2015-04-29 13:19:24 +02:00
Moinak Ghosh
6a757ddb2c Multitue of tweaks and improvements. 
* Use BSC for PNM type and Markup containing binary data.
* Change thresholds in analyzer.
* Properly use double precision in analyzer for accuracy.
* Indicate BSC processing of packPNM output
* Bring back raw-block Dispack for file not processed by Dispack filter.
 2015-03-22 23:36:04 +05:30
Moinak Ghosh
4360c5581f Fix regression in handling of incompressible chunks. 
Attempt to properly signal exit on decompression error (not working yet).
 2015-02-09 23:00:06 -08:00
Moinak Ghosh
7ef20ec5be Specialized dictionary encoding for FASTA files. 2015-02-01 16:20:03 -08:00
Moinak Ghosh
30dee9a1a9 Improve check in E8E9 filter to avoid unnecessary encodes. 
Allow small increase in output chunk size for transform preprocessing (E8E9).
Ensure chunk allocations always include oerhead size.
 2015-01-25 23:17:23 -08:00
Moinak Ghosh
678a6a2da4 A few small fixes. 
Effect same compression algo for Jpeg and PackJPG output.
Fix compiler warning in PackPNM.
Allow unknown type (0) to be specified for Dispack output (for analyzer).
 2015-01-17 20:03:06 +05:30
Moinak Ghosh
d5e1d2cdef Some fixes in the Dictionary preprocessor. 
Fix checking of data type flags.
Allow file-level filters to change output data type.
Tweak analyzer threshold for markup type.
 2015-01-13 19:59:09 +05:30
Moinak Ghosh
077da83d5d A bunch of small fixes in Dict. 
Improve text analysis for markup tags.
Use Libbsc for plain text and PPMd for markup mixed text.
Change thresholds.
 2015-01-11 17:36:46 +05:30
Moinak Ghosh
66a482c968 A new Dictionary preprocessor for text files. 2015-01-09 22:13:24 +05:30
Moinak Ghosh
73307c3996 Multiple checks and balances in Dispack to avoid buffer overlfow. 
Allow filter variants to omit the standard header.
Use E8E9 in Dispack filter as a fallback.
Fix integer overflow for type value in thread data struct.
Do not inline functions in DEBUG build.
 2014-12-21 14:13:58 +05:30
Moinak Ghosh
1db822d866 Add Dispack file-level filter in the libarchive chain. 
Add new file type for Win32-PE executables (Dispack).
Reset file type flag after filter processing for better compression.
Fix array index handling for file type list.
 2014-12-20 11:24:09 +05:30
Moinak Ghosh
9a5361f010 Misclleanous cleanups. 
Fix some formatting.
Increase metadata chunk size slightly.
 2014-12-19 19:15:39 +05:30
Moinak Ghosh
2cd41ec257 Revamp Filter handling code. 
1) Really avoid adding filter xattr for non-processed files.
2) Clean up filter error handling.
3) Avoid libarchive data writes in filter callbacks.
4) Have libarchive data writes in a single place.
5) Properly handle skipping filter processing for a file.
6) Fix temporary file pathname handling.
 2014-12-14 23:37:40 +05:30
Moinak Ghosh
dfe18ef48f Fix missed archive entry record. 
Fix enabling of metadata stream feature.
Fix log message text.
Use macro for path separator.
 2014-12-11 23:16:26 +05:30
Moinak Ghosh
f970b41e34 A bunch of improvements and fixes. 
- Fix heap corruption in DICT Filter.
- Make default Dedup block size as 8KB.
- Revamp executable file handling: Part#1.
- Developed new E8E9 filter that works better than Dispack on raw data blocks.
- Remove block-based Dispack encoding. File-specific Dispack filter to be added.
- Improve file header based executable file detection.
- Introduce new sorting algorithm for filenames without extension.
 2014-12-11 19:15:36 +05:30
Moinak Ghosh
4c62e4db60 Fix buffer size handling in metadata chunks. 2014-11-29 13:18:01 +05:30
Moinak Ghosh
753360e479 Tweak some data type settings. 2014-11-19 20:19:16 +05:30
Moinak Ghosh
b257c83f33 Detect a few mozilla file signatures. 
Add missing option to suppress pathname sorting.
Fix chunk sizing to properly auto-enable deduplication.
Fix default dedupe block size to 8KB.
 2014-11-16 22:57:47 +05:30
Moinak Ghosh
29b5efc988 Add couple of mozilla file extensions. 
Check for files > INT64_T when sorting.
Makefile targets to help development.
 2014-11-15 19:17:33 +05:30
Moinak Ghosh
cfe2cea544 Add a couple of prefetch in LZMA for a 4%-18% improvement. 
Cleanup some whitespace.
 2014-11-09 22:04:34 +05:30
Moinak Ghosh
507e7c75d3 Centralise data analysis routine for optimum performance and leverage. 
Utilise buffer data analysis for preprocessing filters.
 2014-11-06 22:23:33 +05:30
Moinak Ghosh
848010fbb5 Tweak LZP and Dict to mostly avoid non-text files. 2014-11-05 22:05:19 +05:30
Moinak Ghosh
62c7590f26 Detiled listing of archive members (-i). 2014-11-04 00:36:18 +05:30
Moinak Ghosh
5a742b94f0 Add checksum verification of metadata chunks. 2014-11-03 23:32:12 +05:30
Moinak Ghosh
d63f848a24 Fix encryption of metadata chunks. 2014-11-03 23:22:22 +05:30
Moinak Ghosh
ffa7f9cff8 Evict libiconv for Linux plaforms. 2014-11-03 21:14:23 +05:30
Moinak Ghosh
b2ad225fbb iImplement fast TOC listing for metadata streams. 
Fix help text.
Removed redundant allocator code.
Actually free memory on exit.
 2014-11-03 20:20:05 +05:30
Moinak Ghosh
3259c7ced0 Fix sub-type count. 2014-10-27 21:23:03 +05:30
Moinak Ghosh
b7804a0caa Improve file sorting algorithm. 
Add more file extension names.
Fix data type mask size.
 2014-10-27 19:23:03 +05:30
Moinak Ghosh
cc68550670 Add metadata stream flag for archive. 
Change flag bit to not collide with checksum id.
Handle '-T' option properly.
 2014-10-25 22:57:31 +05:30
Moinak Ghosh
e7081eb5a3 Git commit - rehash. Incorrect earlier commit. 
Implement Separate metadata stream.
Fix blatant wrong check in Bzip2 compressor.
Implement E8E9 filter fallback in Dispack.
Improve dict buffer size checks.
Reduce thread count to control memory usage in archive mode.
 2014-10-24 23:30:40 +05:30
Moinak Ghosh
1a00613457 Changes for future feature (disabled now): Separate metadata stream. 
Fix blatant wrong check in Bzip2 compressor.
Implement E8E9 filter fallback in Dispack.
Improve dict buffer size checks.
Reduce thread count to control memory usage in archive mode.
 2014-10-24 21:12:48 +05:30
Moinak Ghosh
e3c32ed6d6 Remove unneeded archive writing function. 
Improve filter scratch buffer handling.
Improve memory accounting.
Remove delayed allocation when compressing. Allows better memory estimation.
Some cstyle fixes.
 2014-09-24 21:54:36 +05:30
Moinak Ghosh
6ba9c4b721 Change EXE code detection process to CSC's Analyzer. 2014-09-23 21:19:10 +05:30
Moinak Ghosh
4c3f3d5ee2 Libarchive tweaks to indicate data vs metadata reads and writes. 2014-09-22 21:21:52 +05:30
Moinak Ghosh
3debf1340c Fix missing newline in error message. 2014-09-21 22:08:10 +05:30
Moinak Ghosh
50643b1f93 Fix libarchive configure and makefiles for Linux build. 2014-09-21 19:57:28 +05:30
Moinak Ghosh
3fedef7f7f Add missing file for clean libarchive build. 2014-09-20 22:06:36 +05:30
Moinak Ghosh
2e5f2d8aab Make DICT filter useful. 
Improve data analysis in adaptive_compress.
 2014-09-20 21:49:06 +05:30
Moinak Ghosh
071a9e2b26 Update,simplify analyzer function to indicate text data for Dict filter. 
Fix archive header writing bug.
Strip ^M chars from dict filter files.
Include DICT preprocessing type.
Fix a bunch of bugs found by Xcode.
 2014-09-20 12:49:00 +05:30
Moinak Ghosh
4fedebc607 Dict filter work in progress. 2014-09-18 22:51:25 +05:30
Moinak Ghosh
f34962f8cc Set Wavpack compression mode based on compression level. 2014-09-17 21:43:00 +05:30
Moinak Ghosh
af39994a59 Working Wavpack filter for compressing WAV filies. 
Improved error handling of filter routines.
Improved verbose logging.
 2014-09-17 20:34:38 +05:30
Moinak Ghosh
fd087a8949 Step 0 of adding WavPack filter - does not work yet. 
WAV file detection.
Rename libarchive dir to be generic.
 2014-09-14 23:56:38 +05:30
Moinak Ghosh
5a875f3174 Regenerate extensions hash. 2014-09-12 17:00:36 +05:30
Moinak Ghosh
3e9a46a602 Add tagging of filter-processed entries with custom XATTR. 
Add magic number based detection of JPEG and PNM formats.
 2014-09-11 20:29:53 +05:30
Moinak Ghosh
08a30d270d Add caller-defined xattr handling features to private libarchive. 2014-09-11 19:17:02 +05:30
Moinak Ghosh
9ecbbbafd0 Pull in private copy of libarchive to add pcmpress-specific functionality. 
First step to add packPNM support.
 2014-09-11 18:34:43 +05:30