greg/pcompress
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
pcompress/Makefile

137 lines
4.2 KiB
Makefile
Raw Normal View History

Initial Commit.
2012-05-28 14:49:29 +00:00
#
# This file is a part of Pcompress, a chunked parallel multi-
# algorithm lossless compression and decompression program.
#
# Copyright (C) 2012 Moinak Ghosh. All rights reserved.
# Use is subject to license terms.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
Update License info to LGPLv3.
2012-07-07 16:48:29 +00:00
# version 3 of the License, or (at your option) any later version.
Initial Commit.
2012-05-28 14:49:29 +00:00
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# moinakg@belenix.org, http://moinakg.wordpress.com/
#
# This program includes partly-modified public domain source
# code from the LZMA SDK: http://www.7-zip.org/sdk.html
#
PROG= pcompress
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
MAINSRCS = main.c utils.c allocator.c zlib_compress.c bzip2_compress.c \
Add support for LZ4 compression including multi-pass LZ4. Add missing Read_Adjusted() declaration, was causing a crash with 2GB chunks. Fix minor cut-paste issues in comments.
2012-07-25 15:37:36 +00:00
lzma_compress.c ppmd_compress.c adaptive_compress.c lzfx_compress.c \
Fix handling of incompressible chunks. Fix handling of various dedup failures. Add NULL compression option for dedup only compression.
2012-08-05 17:05:51 +00:00
lz4_compress.c none_compress.c
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
MAINHDRS = allocator.h pcompress.h utils.h
Initial Commit.
2012-05-28 14:49:29 +00:00
MAINOBJS = $(MAINSRCS:.c=.o)
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
RABINSRCS = rabin/rabin_polynomial.c
RABINHDRS = rabin/rabin_polynomial.h utils.h
RABINOBJS = $(RABINSRCS:.c=.o)
Implement Delta Encoding based on modified bsdiff. Change to more accurate Sketch value computation approach.
2012-07-19 16:11:07 +00:00
BSDIFFSRCS = bsdiff/bsdiff.c bsdiff/bspatch.c bsdiff/rle_encoder.c
BSDIFFHDRS = bsdiff/bscommon.h utils.h allocator.h
BSDIFFOBJS = $(BSDIFFSRCS:.c=.o)
Add multithreaded LZMA port from p7zip Compute balanced thread count between chunk threads and algo threads Generic way to handle querying algorithm parameters Clean up unnecessary includes
2012-08-18 04:50:52 +00:00
LZMASRCS = lzma/LzmaEnc.c lzma/LzFind.c lzma/LzmaDec.c lzma/Threads.c lzma/LzFindMt.c
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
LZMAHDRS = lzma/CpuArch.h lzma/LzFind.h lzma/LzmaEnc.h lzma/Types.h \
Add multithreaded LZMA port from p7zip Compute balanced thread count between chunk threads and algo threads Generic way to handle querying algorithm parameters Clean up unnecessary includes
2012-08-18 04:50:52 +00:00
lzma/LzHash.h lzma/LzmaDec.h utils.h lzma/LzFindMt.h lzma/Threads.h lzma/windows.h \
lzma/Common/MyWindows.h lzma/Common/MyGuidDef.h lzma/basetyps.h
Initial Commit.
2012-05-28 14:49:29 +00:00
LZMAOBJS = $(LZMASRCS:.c=.o)
Add LZFX Compression support, a very fast lightweight compressor. Avoid a branch in the rabin loop.
2012-07-22 18:45:08 +00:00
LZFXSRCS = lzfx/lzfx.c
LZFXHDRS = lzfx/lzfx.h
LZFXOBJS = $(LZFXSRCS:.c=.o)
Add support for LZ4 compression including multi-pass LZ4. Add missing Read_Adjusted() declaration, was causing a crash with 2GB chunks. Fix minor cut-paste issues in comments.
2012-07-25 15:37:36 +00:00
LZ4SRCS = lz4/lz4.c lz4/lz4hc.c
LZ4HDRS = lz4/lz4.h lz4/lz4hc.h
LZ4OBJS = $(LZ4SRCS:.c=.o)
Initial Commit.
2012-05-28 14:49:29 +00:00
PPMDSRCS = lzma/Ppmd8.c lzma/Ppmd8Enc.c lzma/Ppmd8Dec.c
PPMDHDRS = lzma/Ppmd.h lzma/Ppmd8.h
PPMDOBJS = $(PPMDSRCS:.c=.o)
CRCSRCS = lzma/crc64_fast.c lzma/crc64_table.c
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
CRCHDRS = lzma/crc64_table_le.h lzma/crc64_table_be.h lzma/crc_macros.h
Initial Commit.
2012-05-28 14:49:29 +00:00
CRCOBJS = $(CRCSRCS:.c=.o)
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
LZPSRCS = lzp/lzp.c
LZPHDRS = lzp/lzp.h
LZPOBJS = $(LZPSRCS:.c=.o)
BAKFILES = *~ lzma/*~ lzfx/*~ lz4/*~ rabin/*~ bsdiff/*~ lzp/*~
Initial Commit.
2012-05-28 14:49:29 +00:00
RM = rm -f
Add multithreaded LZMA port from p7zip Compute balanced thread count between chunk threads and algo threads Generic way to handle querying algorithm parameters Clean up unnecessary includes
2012-08-18 04:50:52 +00:00
CPPFLAGS = -I. -I./lzma -I./lzfx -I./lz4 -I./rabin -I./bsdiff -DNODEFAULT_PROPS \
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
-DFILE_OFFSET_BITS=64 -D_REENTRANT -D__USE_SSE_INTRIN__ -D_LZMA_PROB32 \
-I./lzp
Initial Commit.
2012-05-28 14:49:29 +00:00
VEC_FLAGS = -ftree-vectorize
LOOP_OPTFLAGS = $(VEC_FLAGS) -floop-interchange -floop-block
LDLIBS = -ldl -lbz2 $(ZLIB_DIR) -lz -lm
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
OBJS = $(MAINOBJS) $(LZMAOBJS) $(PPMDOBJS) $(LZFXOBJS) $(LZ4OBJS) $(CRCOBJS) \
$(RABINOBJS) $(BSDIFFOBJS) $(LZPOBJS)
Initial Commit.
2012-05-28 14:49:29 +00:00
New capability in allocator to add slab caches with user-specified size. Use specific slab caches in main program and Lzma to slightly reduce memory footprint. Fix missing hashtable counter update in allocator. Slight cleanup of repeated computation with macro in Lzma.
2012-05-31 07:36:40 +00:00
ifdef DEBUG
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
LINK = g++ -m64 -pthread -msse3
Compute and compare Mean sketch cksum to improve similarity comparison Fix optflags settings in Makefile Small optimization in zero RLE encoder to avoid scanning during lookahead Some minor fixes
2012-08-09 18:27:24 +00:00
COMPILE = gcc -m64 -g -msse3 -c
COMPILE_cpp = g++ -m64 -g -msse3 -c
Implement Delta Encoding based on modified bsdiff. Change to more accurate Sketch value computation approach.
2012-07-19 16:11:07 +00:00
VEC_FLAGS =
Compute and compare Mean sketch cksum to improve similarity comparison Fix optflags settings in Makefile Small optimization in zero RLE encoder to avoid scanning during lookahead Some minor fixes
2012-08-09 18:27:24 +00:00
LOOP_OPTFLAGS =
Reduce memory consumption and improve performance in dedupe Re-introduce crc64 for dedup blocks to avoid wasted memcpy-s Restructe block array to be an array of pointers allocated on demand Fix a corner case issue when splitting chunks at a dedup boundary
2012-08-17 05:33:02 +00:00
GEN_OPT = -O -fno-omit-frame-pointer
RABIN_OPT = -O -fno-omit-frame-pointer
New capability in allocator to add slab caches with user-specified size. Use specific slab caches in main program and Lzma to slightly reduce memory footprint. Fix missing hashtable counter update in allocator. Slight cleanup of repeated computation with macro in Lzma.
2012-05-31 07:36:40 +00:00
else
Compute and compare Mean sketch cksum to improve similarity comparison Fix optflags settings in Makefile Small optimization in zero RLE encoder to avoid scanning during lookahead Some minor fixes
2012-08-09 18:27:24 +00:00
GEN_OPT = -O3
RABIN_OPT = -O2
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
LINK = g++ -m64 -pthread -msse3
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
COMPILE = gcc -m64 -msse3 -c
COMPILE_cpp = g++ -m64 -msse3 -c
New capability in allocator to add slab caches with user-specified size. Use specific slab caches in main program and Lzma to slightly reduce memory footprint. Fix missing hashtable counter update in allocator. Slight cleanup of repeated computation with macro in Lzma.
2012-05-31 07:36:40 +00:00
CPPFLAGS += -DNDEBUG
Improve Rabin computations using an irreducible polynomial Slight improvement to similarity computation A simple mechanism to include DEBUG mode stats Include stdint for common int types
2012-08-15 14:43:40 +00:00
endif
Implement Parallel deduplication support. Restructure compression functions to take chunk flag as argument. Add missing error flag printing in LZMA. Only create enough threads as needed by chunk size and file size. Minor cleanups and variable name changes.
2012-07-01 16:14:02 +00:00
ifdef DEBUG_NO_SLAB
CPPFLAGS += -DDEBUG_NO_SLAB
endif
Improve Rabin computations using an irreducible polynomial Slight improvement to similarity computation A simple mechanism to include DEBUG mode stats Include stdint for common int types
2012-08-15 14:43:40 +00:00
ifdef DEBUG_STATS
CPPFLAGS += -DDEBUG_STATS
New capability in allocator to add slab caches with user-specified size. Use specific slab caches in main program and Lzma to slightly reduce memory footprint. Fix missing hashtable counter update in allocator. Slight cleanup of repeated computation with macro in Lzma.
2012-05-31 07:36:40 +00:00
endif
Initial Commit.
2012-05-28 14:49:29 +00:00
all: $(PROG)
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
$(LZMAOBJS): $(LZMASRCS) $(LZMAHDRS)
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
$(COMPILE) $(GEN_OPT) $(CPPFLAGS) $(@:.o=.c) -o $@
Initial Commit.
2012-05-28 14:49:29 +00:00
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
$(CRCOBJS): $(CRCSRCS) $(CRCHDRS)
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
$(COMPILE) $(GEN_OPT) $(VEC_FLAGS) $(CPPFLAGS) $(@:.o=.c) -o $@
Initial Commit.
2012-05-28 14:49:29 +00:00
$(PPMDOBJS): $(PPMDSRCS) $(PPMDHDRS)
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
$(COMPILE) $(GEN_OPT) $(VEC_FLAGS) $(CPPFLAGS) $(@:.o=.c) -o $@
Initial Commit.
2012-05-28 14:49:29 +00:00
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
$(RABINOBJS): $(RABINSRCS) $(RABINHDRS)
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
$(COMPILE) $(RABIN_OPT) $(VEC_FLAGS) $(CPPFLAGS) $(@:.o=.c) -o $@
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
Implement Delta Encoding based on modified bsdiff. Change to more accurate Sketch value computation approach.
2012-07-19 16:11:07 +00:00
$(BSDIFFOBJS): $(BSDIFFSRCS) $(BSDIFFHDRS)
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
$(COMPILE) $(GEN_OPT) $(VEC_FLAGS) $(CPPFLAGS) $(@:.o=.c) -o $@
Implement Delta Encoding based on modified bsdiff. Change to more accurate Sketch value computation approach.
2012-07-19 16:11:07 +00:00
Add LZFX Compression support, a very fast lightweight compressor. Avoid a branch in the rabin loop.
2012-07-22 18:45:08 +00:00
$(LZFXOBJS): $(LZFXSRCS) $(LZFXHDRS)
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
$(COMPILE) $(GEN_OPT) $(VEC_FLAGS) $(CPPFLAGS) $(@:.o=.c) -o $@
Add support for LZ4 compression including multi-pass LZ4. Add missing Read_Adjusted() declaration, was causing a crash with 2GB chunks. Fix minor cut-paste issues in comments.
2012-07-25 15:37:36 +00:00
$(LZ4OBJS): $(LZ4SRCS) $(LZ4HDRS)
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
$(COMPILE) $(GEN_OPT) $(VEC_FLAGS) $(CPPFLAGS) $(@:.o=.c) -o $@
Add LZFX Compression support, a very fast lightweight compressor. Avoid a branch in the rabin loop.
2012-07-22 18:45:08 +00:00
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
$(LZPOBJS): $(LZPSRCS) $(LZPHDRS)
$(COMPILE) $(GEN_OPT) $(VEC_FLAGS) $(CPPFLAGS) $(@:.o=.c) -o $@
Add ability to adjust chunk boundary based on Rabin Fingerprinting to improve compression. Remove unnecessary checks in compression loop.
2012-06-21 14:57:05 +00:00
$(MAINOBJS): $(MAINSRCS) $(MAINHDRS)
Bias fingerprint value with occurrence counts for a better sketch Fix latent bug when calling algo deinit in decompression code Reduce diff threshold for slightly greater delta encoding Limit similar buffer size difference for less wasted diffing Change zlib compression wrapper to use faster deflateReset mechanism Reduce optimization level for Dedupe code, it goes faster
2012-08-08 17:10:58 +00:00
$(COMPILE) $(GEN_OPT) $(LOOP_OPTFLAGS) $(CPPFLAGS) $(@:.o=.c) -o $@
Initial Commit.
2012-05-28 14:49:29 +00:00
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
$(PROG): $(OBJS)
$(LINK) -o $@ $(OBJS) $(LDLIBS)
Initial Commit.
2012-05-28 14:49:29 +00:00
clean:
Add LZP Pre-Compression support ported from libbsc. Add generic pre-processing wrappers for future support of other pre-processors. Clean up computation of Rabin block sizes. Compute Rabin scratch space accurately to avoid RAM wastage.
2012-08-23 17:28:44 +00:00
$(RM) $(PROG) $(OBJS) $(BAKFILES)
Initial Commit.
2012-05-28 14:49:29 +00:00
Reference in a new issue Copy permalink