greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
399 commits 3 branches 21 tags 7.6 MiB
Commit graph

95 commits

Author SHA1 Message Date
Moinak Ghosh
b7804a0caa Improve file sorting algorithm. 
Add more file extension names.
Fix data type mask size.
 2014-10-27 19:23:03 +05:30
Moinak Ghosh
e3c32ed6d6 Remove unneeded archive writing function. 
Improve filter scratch buffer handling.
Improve memory accounting.
Remove delayed allocation when compressing. Allows better memory estimation.
Some cstyle fixes.
 2014-09-24 21:54:36 +05:30
Moinak Ghosh
071a9e2b26 Update,simplify analyzer function to indicate text data for Dict filter. 
Fix archive header writing bug.
Strip ^M chars from dict filter files.
Include DICT preprocessing type.
Fix a bunch of bugs found by Xcode.
 2014-09-20 12:49:00 +05:30
Moinak Ghosh
4fedebc607 Dict filter work in progress. 2014-09-18 22:51:25 +05:30
Moinak Ghosh
af39994a59 Working Wavpack filter for compressing WAV filies. 
Improved error handling of filter routines.
Improved verbose logging.
 2014-09-17 20:34:38 +05:30
Moinak Ghosh
fd087a8949 Step 0 of adding WavPack filter - does not work yet. 
WAV file detection.
Rename libarchive dir to be generic.
 2014-09-14 23:56:38 +05:30
Moinak Ghosh
5a875f3174 Regenerate extensions hash. 2014-09-12 17:00:36 +05:30
Moinak Ghosh
3e9a46a602 Add tagging of filter-processed entries with custom XATTR. 
Add magic number based detection of JPEG and PNM formats.
 2014-09-11 20:29:53 +05:30
Moinak Ghosh
9ecbbbafd0 Pull in private copy of libarchive to add pcmpress-specific functionality. 
First step to add packPNM support.
 2014-09-11 18:34:43 +05:30
Moinak Ghosh
d5ceda559e Update Licensing notes and build notes. 
More whitespace fixes.
 2014-07-26 15:28:40 +05:30
Moinak Ghosh
10f40e1c6f Part 1 changes to allow dual licensing to MPLV2. 
Make external LGPL code/features disabled in MPLV2 variant.
Nuke some unwanted whitespace (cstyle).
 2014-07-24 22:20:30 +05:30
Moinak Ghosh
0433452b37 Miscellaneous refactoring. 
Add some headers for OSX.
 2014-05-24 23:52:30 +05:30
Moinak Ghosh
125828b6b3 Update CPUID detection for AVX2. 2014-05-04 22:35:12 +05:30
Moinak Ghosh
7055a8fc71 Add AVX detection and usage of -mavx[2] flags. 
Force GCC to use Clang assembler always on MAC OS X for AVX support.
 2014-05-04 22:03:40 +05:30
Moinak Ghosh
63bef473cc Working MAC OS X port. 
Compatibility layer for semaphore handling.
 2014-05-04 21:11:31 +05:30
Moinak Ghosh
4b037f0ed7 More portability tweaks. 2014-05-04 11:54:33 +05:30
Moinak Ghosh
f2da433188 More portability tweaks. 
Handle platform specific yasm parameters.
Resolve namespace conflict on OSX.
Do not build Skein ASM code on OSX.
 2014-04-30 22:46:24 +05:30
Moinak Ghosh
6fba8aa8ac More OSX compatibility code. 
Fix new warnings with Gcc 4.8.
 2014-04-28 00:12:51 +05:30
Moinak Ghosh
8b52f7ccb8 Some initial tidbits for the start of an OSX/Windows port. 2014-04-26 23:05:14 +05:30
Moinak Ghosh
683c3e48b5 Detect some DICOM formats and use BSC for DICOM data. 2014-01-01 19:44:58 +05:30
Moinak Ghosh
ea345a902a Overhaul documentation part #1 
Detect and handle uncompressed PDF files using libbsc.
Force binary/text data detection for tar archives.
Get rid of unnecessary CLI option.
Add full pipeline mode check when archiving.
 2013-12-30 23:24:37 +05:30
Moinak Ghosh
5521955a94 Detect AR archives and set the type. 
Re-use a less common type code for AR.
Use Dispack generically for all executables and AR archives.
 2013-12-18 23:00:39 +05:30
Moinak Ghosh
a741f34f78 Move MSDOS COM single-byte magic number checks to last in the list. 
Move advanced options flag into context structure.
Include dtd files as text type.
 2013-12-18 00:09:32 +05:30
Moinak Ghosh
5e484f0694 Use libbsc for AVI and MP4 files. 2013-12-04 20:07:52 +05:30
Moinak Ghosh
3f62cdf7d5 Use Libbsc for MP4 and FLAC files. 
Change some rare file type codes to indicate some common types.
 2013-12-03 21:56:07 +05:30
Moinak Ghosh
c4c4b47138 Use Libbsc for DNA Sequence data instead of PPMD. Faster, better compression. 
Fix pz extension handling for real.
 2013-11-30 09:58:21 +05:30
Moinak Ghosh
dfeea8c19b Avoid Delta2,LZP for TIFF files. Negatively impacts compression. 2013-11-29 19:47:57 +05:30
Moinak Ghosh
306f145f22 Use libbsc/ppmd for BMP files. 
Fix extension based hashing.
Do not append .pz extension to filenames already having it.
Some code formatting changes.
 2013-11-28 22:42:51 +05:30
Moinak Ghosh
0192790c02 Add Dispack filter with auto-detection of x86 executables in archive mode. 
More elaborate magic header based detection of 32-bit and 64-bit x86 binaries.
Always use fast-mode LZ4 in Adaptive modes.
 2013-11-24 19:45:58 +05:30
Moinak Ghosh
1e2c3e479a Optimize preprocessed compression and avoid a bunch of memory copies. 
Fix a crash.
Add a few more file types.
More comments.
 2013-11-22 20:44:26 +05:30
Moinak Ghosh
11584cab52 Add fast handling of totally incompressible data (like Jpegs) in adaptive modes. 
Add function to indicate totally incompressible data when archiving.
Reformat if statements in some places to reduce branching.
 2013-11-15 21:06:23 +05:30
Moinak Ghosh
e90c52e516 Work in progress changes for packJPG encoding and decoding. 
Enhance custom LibArchive filter functionlity.
 2013-11-13 23:28:01 +05:30
Moinak Ghosh
6aacd903ff Structured handling of file types. 
Handling of already compressed data based on compression algorithm.
Add a few more extension types.
 2013-11-09 16:46:19 +05:30
Moinak Ghosh
cae9de9b2e Leverage file type detection(archiver) to improve compression performance. 
Use detected file/data type(archiver) for Adaptive compression modes.
Update type flags and add more extensions.
 2013-11-08 23:50:28 +05:30
Moinak Ghosh
b7facc929e Add file type detection based on magic values. 
Add more comments.
Add more extensions.
 2013-11-07 23:57:15 +05:30
Moinak Ghosh
991482403b Add extension based file type detection and setting segment data type. 
Use Bob Jenkins Minimal Perfect Hash to check for known extensions.
Use semaphore signaling and direct buffer copy for extraction.
Miscellaneous fixes.
 2013-11-07 21:48:54 +05:30
Moinak Ghosh
7ed532133e Avoid using pipe during archive creation. Use semaphores and direct memory copy. 2013-11-02 23:43:59 +05:30
Moinak Ghosh
8e4b774c8c More changes for archiving. 
Allow multiple filenames on command line when archiving.
Remove unneded small block writes with libarchive.
 2013-10-27 20:36:48 +05:30
Moinak Ghosh
46b11def08 Archiving support using Libarchive: Work in progress changes #3. 
Make log_msg() add newline by default.
 2013-10-24 00:16:04 +05:30
Moinak Ghosh
7f81869874 Archiving support using Libarchive: Work in progress changes. 
Change all perror() calls to use logger.
Make the config script a little verbose.
 2013-10-20 23:54:27 +05:30
Moinak Ghosh
28fd9848f9 Ability to specify output compressed pathname. 
Fix log level handling.
Trim commented code.
 2013-10-10 21:19:44 +05:30
Moinak Ghosh
8c1f4ebe61 Add a simple log facility. 
Refactor all printfs to use log facility.
 2013-10-02 20:45:33 +05:30
Moinak Ghosh
fa78621cbf Cleanup pointer casting in code to use macros. 2013-09-22 20:11:15 +05:30
Moinak Ghosh
38c0869f5c Update Changelog and tweak free memory detection for 2.4 release. 
Add identifiers to error messages for clarity.
Fix init of dedupe block size.
Tweak free memory detection to include swap and shared memory consideration.
 2013-09-05 21:12:37 +05:30
Moinak Ghosh
d31c6433c2 Update free memory computation to include cached buffers. 
Fix a potential rare corner case.
 2013-08-17 11:31:44 +05:30
Moinak Ghosh
fe18afbcf4 Use wrapper script to set paths when launching pcompress from build directory. 
Use smaller max block size when doing global dedupe.
Fix init of executable name.
 2013-08-07 22:03:52 +05:30
Moinak Ghosh
7743792018 Make default symbol visibility to hidden with explicit public visibility specified. 
Add missing static scope to a few more places.
 2013-06-03 20:51:00 +05:30
Moinak Ghosh
c859cf35d5 Make Pcompress functionality into a library - initial changes. 2013-06-02 20:54:33 +05:30
Moinak Ghosh
ddaa3b6b6d Drastic simplification of Min-heap code and resultant Delta speedup. 2013-05-25 17:34:38 +05:30
Moinak Ghosh
c6da2325e3 Allow SKEIN to be used as a Global Dedupe chunk lookup hash. 2013-05-04 15:59:29 +05:30