greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
pcompress/pcompress.h

206 lines
7.5 KiB
C
Raw Normal View History

Initial Commit.
2012-05-28 14:49:29 +00:00
/*
* This file is a part of Pcompress, a chunked parallel multi-
* algorithm lossless compression and decompression program.
*
* Copyright (C) 2012 Moinak Ghosh. All rights reserved.
* Use is subject to license terms.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
Update License info to LGPLv3.
2012-07-07 16:48:29 +00:00
* version 3 of the License, or (at your option) any later version.
Initial Commit.
2012-05-28 14:49:29 +00:00
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* moinakg@belenix.org, http://moinakg.wordpress.com/
*
* This program includes partly-modified public domain source
* code from the LZMA SDK: http://www.7-zip.org/sdk.html
*/
#ifndef _PCOMPRESS_H
#define _PCOMPRESS_H
#include <sys/types.h>
#include <pthread.h>
#include <semaphore.h>
#ifdef __cplusplus
extern "C" {
#endif
Add xxHash for Rabin block checksums, slightly faster than CRC64. Fix missing initialization of character counts table. Some file reorganization.
2012-09-02 15:10:32 +00:00
#include <rabin_dedup.h>
Implement HMAC Functionality. Refactor crypto utility functions in a separate file. Add HMAC verification for file header. Add a few extra input validation checks. Include space in chunk header for HMAC.
2012-10-17 18:02:35 +00:00
#include <crypto_utils.h>
Implement Deduplication based on Rabin Fingerprinting: work in progress. Fix bug that prevented pipe mode from being used. Allow building without specialized allocator. Use basic optimize flag in debuig build.
2012-06-29 12:53:55 +00:00
Add support for Skein512 and Skein256 checksums Import Skein code from NIST CD submission Make checksum algorithms pluggable Fix handling of huge buffers (>2GB) in LZP Cleanup of some buffer sizing code Speed up CRC64 calculation in dedupe chunking
2012-08-31 17:06:06 +00:00
#define CHUNK_FLAG_SZ 1
Initial Commit.
2012-05-28 14:49:29 +00:00
#define ALGO_SZ 8
#define MIN_CHUNK 2048
Changes for generalized runtime SSE/AVX/XOP detection. Multi instruction set XXhash build with runtime selection. Extend CPUID code to detect more instruction sets. Add options for BLAKE2 hash. Move GCC builtins into utils header. Bump file format version number due to extended digest flags. Add descriptions to digest list.
2013-01-24 18:40:12 +00:00
#define VERSION 6
Implement Deduplication based on Rabin Fingerprinting: work in progress. Fix bug that prevented pipe mode from being used. Allow building without specialized allocator. Use basic optimize flag in debuig build.
2012-06-29 12:53:55 +00:00
#define FLAG_DEDUP 1
Support for chunk-level encryption using AES and Scrypt based PBE. Couple of minor fixes.
2012-10-15 06:40:00 +00:00
#define FLAG_DEDUP_FIXED 2
#define FLAG_SINGLE_CHUNK 4
Bump version, update Changelog and documentation for 1.2 release.
2013-01-03 18:10:21 +00:00
#define UTILITY_VERSION "1.2.0"
Support for chunk-level encryption using AES and Scrypt based PBE. Couple of minor fixes.
2012-10-15 06:40:00 +00:00
#define MASK_CRYPTO_ALG 0x30
Implement HMAC Functionality. Refactor crypto utility functions in a separate file. Add HMAC verification for file header. Add a few extra input validation checks. Include space in chunk header for HMAC.
2012-10-17 18:02:35 +00:00
#define MAX_LEVEL 14
Initial Commit.
2012-05-28 14:49:29 +00:00
#define COMPRESSED 1
#define UNCOMPRESSED 0
#define CHSIZE_MASK 0x80
#define BZIP2_A_NUM 16
#define LZMA_A_NUM 32
Fix handling of compression flags in adaptive mode Fix error handling when chunk size is too small for dedupe Bump version to 0.6
2012-08-10 05:17:11 +00:00
#define CHUNK_FLAG_DEDUP 2
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
#define CHUNK_FLAG_PREPROC 4
Fix handling of compression flags in adaptive mode Fix error handling when chunk size is too small for dedupe Bump version to 0.6
2012-08-10 05:17:11 +00:00
#define COMP_EXTN ".pz"
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
#define PREPROC_TYPE_LZP 1
Implement Adaptive Delta Encoding.
2012-12-04 18:39:47 +00:00
#define PREPROC_TYPE_DELTA2 2
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
#define PREPROC_COMPRESSED 128
Fix handling of compression flags in adaptive mode Fix error handling when chunk size is too small for dedupe Bump version to 0.6
2012-08-10 05:17:11 +00:00
/*
Add support for Skein512 and Skein256 checksums Import Skein code from NIST CD submission Make checksum algorithms pluggable Fix handling of huge buffers (>2GB) in LZP Cleanup of some buffer sizing code Speed up CRC64 calculation in dedupe chunking
2012-08-31 17:06:06 +00:00
* Sizes of chunk header components.
*/
#define COMPRESSED_CHUNKSZ (sizeof (uint64_t))
#define ORIGINAL_CHUNKSZ (sizeof (uint64_t))
#define CHUNK_HDR_SZ (COMPRESSED_CHUNKSZ + cksum_bytes + ORIGINAL_CHUNKSZ + CHUNK_FLAG_SZ)
/*
* lower 3 bits in higher nibble indicate chunk compression algorithm
* in adaptive modes.
Fix handling of compression flags in adaptive mode Fix error handling when chunk size is too small for dedupe Bump version to 0.6
2012-08-10 05:17:11 +00:00
*/
Fix chunk flag setup when compression fails in adaptive mode. Prevent display of non-fatal errors during compression.
2012-11-09 15:37:23 +00:00
#define COMPRESS_NONE 0
Initial Commit.
2012-05-28 14:49:29 +00:00
#define COMPRESS_LZMA 1
#define COMPRESS_BZIP2 2
#define COMPRESS_PPMD 3
Use Libbsc for XML data in adapt2 mode.
2012-11-04 15:43:26 +00:00
#define COMPRESS_BSC 4
Fix handling of compression flags in adaptive mode Fix error handling when chunk size is too small for dedupe Bump version to 0.6
2012-08-10 05:17:11 +00:00
#define CHDR_ALGO_MASK 7
Initial Commit.
2012-05-28 14:49:29 +00:00
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern uint32_t zlib_buf_extra(uint64_t buflen);
extern int lz4_buf_extra(uint64_t buflen);
Get rid of size_t in places where 64-bitness is assumed.
2012-12-09 04:45:06 +00:00
extern int zlib_compress(void *src, uint64_t srclen, void *dst,
uint64_t *destlen, int level, uchar_t chdr, void *data);
extern int lzma_compress(void *src, uint64_t srclen, void *dst,
uint64_t *destlen, int level, uchar_t chdr, void *data);
extern int bzip2_compress(void *src, uint64_t srclen, void *dst,
uint64_t *destlen, int level, uchar_t chdr, void *data);
extern int adapt_compress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int ppmd_compress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int lz_fx_compress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int lz4_compress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int none_compress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int zlib_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int lzma_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int bzip2_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int adapt_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int ppmd_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int lz_fx_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int lz4_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int none_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int adapt_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int adapt2_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int lzma_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int ppmd_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int bzip2_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int zlib_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int lz_fx_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int lz4_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int none_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Add multithreaded LZMA port from p7zip Compute balanced thread count between chunk threads and algo threads Generic way to handle querying algorithm parameters Clean up unnecessary includes
2012-08-18 04:50:52 +00:00
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern void lzma_props(algo_props_t *data, int level, uint64_t chunksize);
extern void lzma_mt_props(algo_props_t *data, int level, uint64_t chunksize);
extern void lz4_props(algo_props_t *data, int level, uint64_t chunksize);
extern void zlib_props(algo_props_t *data, int level, uint64_t chunksize);
extern void ppmd_props(algo_props_t *data, int level, uint64_t chunksize);
extern void lz_fx_props(algo_props_t *data, int level, uint64_t chunksize);
extern void bzip2_props(algo_props_t *data, int level, uint64_t chunksize);
extern void adapt_props(algo_props_t *data, int level, uint64_t chunksize);
extern void none_props(algo_props_t *data, int level, uint64_t chunksize);
Add basic compression statistics and new option to display them.
2012-05-31 16:06:33 +00:00
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
extern int zlib_deinit(void **data);
Initial Commit.
2012-05-28 14:49:29 +00:00
extern int adapt_deinit(void **data);
extern int lzma_deinit(void **data);
extern int ppmd_deinit(void **data);
Make LZFX Hash size dynamic. Use smaller min rabin block when using fast compression algos. Add missing check for algo init function return value.
2012-07-23 16:13:12 +00:00
extern int lz_fx_deinit(void **data);
Add support for LZ4 compression including multi-pass LZ4. Add missing Read_Adjusted() declaration, was causing a crash with 2GB chunks. Fix minor cut-paste issues in comments.
2012-07-25 15:37:36 +00:00
extern int lz4_deinit(void **data);
Fix handling of incompressible chunks. Fix handling of various dedup failures. Add NULL compression option for dedup only compression.
2012-08-05 17:05:51 +00:00
extern int none_deinit(void **data);
Initial Commit.
2012-05-28 14:49:29 +00:00
Add basic compression statistics and new option to display them.
2012-05-31 16:06:33 +00:00
extern void adapt_stats(int show);
extern void ppmd_stats(int show);
extern void lzma_stats(int show);
extern void bzip2_stats(int show);
extern void zlib_stats(int show);
Add LZFX Compression support, a very fast lightweight compressor. Avoid a branch in the rabin loop.
2012-07-22 18:45:08 +00:00
extern void lz_fx_stats(int show);
Add support for LZ4 compression including multi-pass LZ4. Add missing Read_Adjusted() declaration, was causing a crash with 2GB chunks. Fix minor cut-paste issues in comments.
2012-07-25 15:37:36 +00:00
extern void lz4_stats(int show);
Fix handling of incompressible chunks. Fix handling of various dedup failures. Add NULL compression option for dedup only compression.
2012-08-05 17:05:51 +00:00
extern void none_stats(int show);
Add basic compression statistics and new option to display them.
2012-05-31 16:06:33 +00:00
Add support for libbsc a high-performance block sorting compressor. Enable external algorithm threading for single chunk compressed files. Update docs.
2012-08-27 16:21:55 +00:00
#ifdef ENABLE_PC_LIBBSC
Get rid of size_t in places where 64-bitness is assumed.
2012-12-09 04:45:06 +00:00
extern int libbsc_compress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
extern int libbsc_decompress(void *src, uint64_t srclen, void *dst,
uint64_t *dstlen, int level, uchar_t chdr, void *data);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern int libbsc_init(void **data, int *level, int nthreads, uint64_t chunksize,
Zlib optimizations. Use raw deflate streams to avoid unnecessary adler32. Change some function signatures to improve algo init function behavior. Fix corner case dedupe bug in error handling flow. Bump archive version signature.
2012-11-22 15:32:50 +00:00
int file_version, compress_op_t op);
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
extern void libbsc_props(algo_props_t *data, int level, uint64_t chunksize);
Add support for libbsc a high-performance block sorting compressor. Enable external algorithm threading for single chunk compressed files. Update docs.
2012-08-27 16:21:55 +00:00
extern int libbsc_deinit(void **data);
extern void libbsc_stats(int show);
#endif
Initial Commit.
2012-05-28 14:49:29 +00:00
/*
* Per-thread data structure for compression and decompression threads.
*/
struct cmp_data {
uchar_t *cmp_seg;
uchar_t *compressed_chunk;
uchar_t *uncompressed_chunk;
Add support for Fixed-Block deduplication. More refactoring of symbol names.
2012-09-16 05:42:58 +00:00
dedupe_context_t *rctx;
Introduce strict compiler flags and fix scores of warnings/issues. Avoid different optimization flags for Dedupe sources. Fix liberal mixing of uint64_t and int64_t (should all be uint64_t). Fix corner case crash when decompressing.
2012-12-27 17:36:48 +00:00
uint64_t rbytes;
uint64_t chunksize;
uint64_t len_cmp, len_cmp_be;
Add support for Skein512 and Skein256 checksums Import Skein code from NIST CD submission Make checksum algorithms pluggable Fix handling of huge buffers (>2GB) in LZP Cleanup of some buffer sizing code Speed up CRC64 calculation in dedupe chunking
2012-08-31 17:06:06 +00:00
uchar_t checksum[CKSUM_MAX_BYTES];
Initial Commit.
2012-05-28 14:49:29 +00:00
int level;
unsigned int id;
compress_func_ptr compress;
compress_func_ptr decompress;
int cancel;
sem_t start_sem;
sem_t cmp_done_sem;
sem_t write_done_sem;
void *data;
pthread_t thr;
Implement HMAC verification for chunk data and header. Do not ask for decryption passwd twice.
2012-10-18 17:25:41 +00:00
mac_ctx_t chunk_hmac;
Implement Adaptive Delta Encoding.
2012-12-04 18:39:47 +00:00
algo_props_t *props;
Initial Commit.
2012-05-28 14:49:29 +00:00
};
#ifdef __cplusplus
}
#endif
#endif
Reference in a new issue Copy permalink