1 line
No EOL
12 KiB
JSON
1 line
No EOL
12 KiB
JSON
{"google":"","name":"Pcompress","body":"Pcompress\r\n=========\r\n\r\nCopyright (C) 2012 Moinak Ghosh. All rights reserved.\r\nUse is subject to license terms.\r\nmoinakg (_at) gma1l _dot com.\r\nComments, suggestions, code, rants etc are welcome.\r\n\r\nPcompress is a utility to do compression and decompression in parallel by\r\nsplitting input data into chunks. It has a modular structure and includes\r\nsupport for multiple algorithms like LZMA, Bzip2, PPMD, etc, with SKEIN\r\nchecksums for data integrity. It can also do Lempel-Ziv pre-compression\r\n(derived from libbsc) to improve compression ratios across the board. SSE\r\noptimizations for the bundled LZMA are included. It also implements\r\nchunk-level Content-Aware Deduplication and Delta Compression features\r\nbased on a Semi-Rabin Fingerprinting scheme. Delta Compression is done\r\nvia the widely popular bsdiff algorithm. Similarity is detected using a\r\ncustom hashing of maximal features of a block. When doing chunk-level\r\ndedupe it attempts to merge adjacent non-duplicate blocks index entries\r\ninto a single larger entry to reduce metadata. In addition to all these it\r\ncan internally split chunks at rabin boundaries to help dedupe and\r\ncompression.\r\n\r\nIt has low metadata overhead and overlaps I/O and compression to achieve\r\nmaximum parallelism. It also bundles a simple slab allocator to speed\r\nrepeated allocation of similar chunks. It can work in pipe mode, reading\r\nfrom stdin and writing to stdout. It also provides some adaptive compression\r\nmodes in which multiple algorithms are tried per chunk to determine the best\r\none for the given chunk. Finally it supports 14 compression levels to allow\r\nfor ultra compression modes in some algorithms.\r\n\r\nPcompress also supports encryption via AES and uses Scrypt from Tarsnap\r\nfor Password Based Key generation.\r\n\r\nNOTE: This utility is Not an archiver. It compresses only single files or\r\n datastreams. To archive use something else like tar, cpio or pax.\r\n\r\nUsage\r\n=====\r\n\r\n To compress a file:\r\n pcompress -c <algorithm> [-l <compress level>] [-s <chunk size>] <file>\r\n Where <algorithm> can be the folowing:\r\n lzfx - Very fast and small algorithm based on LZF.\r\n lz4 - Ultra fast, high-throughput algorithm reaching RAM B/W at level1.\r\n zlib - The base Zlib format compression (not Gzip).\r\n lzma - The LZMA (Lempel-Ziv Markov) algorithm from 7Zip.\r\n lzmaMt - Multithreaded version of LZMA. This is a faster version but\r\n uses more memory for the dictionary. Thread count is balanced\r\n between chunk processing threads and algorithm threads.\r\n bzip2 - Bzip2 Algorithm from libbzip2.\r\n ppmd - The PPMd algorithm excellent for textual data. PPMd requires\r\n at least 64MB X CPUs more memory than the other modes.\r\n\r\n libbsc - A Block Sorting Compressor using the Burrows Wheeler Transform\r\n like Bzip2 but runs faster and gives better compression than\r\n Bzip2 (See: libbsc.com).\r\n\r\n adapt - Adaptive mode where ppmd or bzip2 will be used per chunk,\r\n depending on heuristics. If at least 50% of the input data is\r\n 7-bit text then PPMd will be used otherwise Bzip2.\r\n adapt2 - Adaptive mode which includes ppmd and lzma. If at least 80% of\r\n the input data is 7-bit text then PPMd will be used otherwise\r\n LZMA. It has significantly more memory usage than adapt.\r\n none - No compression. This is only meaningful with -D and -E so Dedupe\r\n can be done for post-processing with an external utility.\r\n <chunk_size> - This can be in bytes or can use the following suffixes:\r\n g - Gigabyte, m - Megabyte, k - Kilobyte.\r\n Larger chunks produce better compression at the cost of memory.\r\n <compress_level> - Can be a number from 0 meaning minimum and 14 meaning\r\n maximum compression.\r\n\r\nNOTE: The option \"libbsc\" uses Ilya Grebnov's block sorting compression library\r\n from http://libbsc.com/ . It is only available if pcompress in built with\r\n that library. See INSTALL file for details.\r\n \r\n To decompress a file compressed using above command:\r\n pcompress -d <compressed file> <target file>\r\n\r\n To operate as a pipe, read from stdin and write to stdout:\r\n pcompress -p ...\r\n\r\n Attempt Rabin fingerprinting based deduplication on chunks:\r\n pcompress -D ...\r\n pcompress -D -r ... - Do NOT split chunks at a rabin boundary. Default\r\n is to split.\r\n\r\n Perform Delta Encoding in addition to Identical Dedup:\r\n pcompress -E ... - This also implies '-D'. This performs Delta Compression\r\n between 2 blocks if they are 40% to 60% similar. The\r\n similarity %age is selected based on the dedupe block\r\n size to balance performance and effectiveness.\r\n pcompress -EE .. - This causes Delta Compression to happen if 2 blocks are\r\n at least 40% similar regardless of block size. This can\r\n effect greater final compression ratio at the cost of\r\n higher processing overhead.\r\n\r\n Number of threads can optionally be specified: -t <1 - 256 count>\r\n Other flags:\r\n '-L' - Enable LZP pre-compression. This improves compression ratio of all\r\n algorithms with some extra CPU and very low RAM overhead. Using\r\n delta encoding in conjunction with this may not always be beneficial.\r\n '-S' <cksum>\r\n - Specify chunk checksum to use: CRC64, SKEIN256, SKEIN512, SHA256 and\r\n SHA512. Default one is SKEIN256. The implementation actually uses SKEIN\r\n 512-256. This is 25% slower than simple CRC64 but is many times more\r\n robust than CRC64 in detecting data integrity errors. SKEIN is a\r\n finalist in the NIST SHA-3 standard selection process and is one of\r\n the fastest in the group, especially on x86 platforms. BLAKE is faster\r\n than SKEIN on a few platforms.\r\n SKEIN 512-256 is about 60% faster than SHA 512-256 on x64 platforms.\r\n\r\n '-F' - Perform Fixed Block Deduplication. This is faster than fingerprinting\r\n based content-aware deduplication in some cases. However this is mostly\r\n usable for disk dumps especially virtual machine images. This generally\r\n gives lower dedupe ratio than content-aware dedupe (-D) and does not\r\n support delta compression.\r\n '-M' - Display memory allocator statistics\r\n '-C' - Display compression statistics\r\n\r\nNOTE: It is recommended not to use '-L' with libbsc compression since libbsc uses\r\n LZP internally as well.\r\n\r\n Encryption flags:\r\n '-e' Encrypt chunks with AES. The password can be prompted from the user\r\n or read from a file. Whether 128-Bit or 256-Bit keys are used depends\r\n on how the pcompress binary was built. Default build uses 128-Bit keys.\r\n Unique keys are generated every time pcompress is run even when giving\r\n the same password. Of course enough info is stored in the compressed\r\n file so that the key used for the file can be re-created given the\r\n correct password.\r\n\r\n The Scrypt algorithm from Tarsnap is used\r\n (See: http://www.tarsnap.com/scrypt.html) for generating keys from\r\n passwords. The CTR mode AES mechanism from Tarsnap is also utilized.\r\n\r\n '-w <pathname>'\r\n Provide a file which contains the encryption password. This file must\r\n be readable and writable since it is zeroed out after the password is\r\n read.\r\n\r\nNOTE: When using pipe-mode via -p the only way to provide a password is to use '-w'.\r\n\r\nEnvironment Variables\r\n=====================\r\n\r\nSet ALLOCATOR_BYPASS=1 in the environment to avoid using the the built-in\r\nallocator. Due to the the way it rounds up an allocation request to the nearest\r\nslab the built-in allocator can allocate extra unused memory. In addition you\r\nmay want to use a different allocator in your environment.\r\n\r\nExamples\r\n========\r\n\r\nCompress \"file.tar\" using bzip2 level 6, 64MB chunk size and use 4 threads. In\r\naddition perform identity deduplication and delta compression prior to compression.\r\n\r\n pcompress -D -E -c bzip2 -l6 -s64m -t4 file.tar\r\n\r\nCompress \"file.tar\" using extreme compression mode of LZMA and a chunk size of\r\nof 1GB. Allow pcompress to detect the number of CPU cores and use as many threads.\r\n\r\n pcompress -c lzma -l14 -s1g file.tar\r\n\r\nCompress \"file.tar\" using lz4 at max compression with LZ-Prediction pre-processing\r\nand encryption enabled. Chunksize is 100M:\r\n\r\n pcompress -c lz4 -l3 -e -L -s100m file.tar\r\n\r\nCompression Algorithms\r\n======================\r\n\r\nLZFX\t- Ultra Fast, average compression. This algorithm is the fastest overall.\r\n\t Levels: 1 - 5\r\nLZ4\t- Very Fast, better compression than LZFX.\r\n\t Levels: 1 - 3\r\nZlib\t- Fast, better compression.\r\n\t Levels: 1 - 9\r\nBzip2\t- Slow, much better compression than Zlib.\r\n\t Levels: 1 - 9\r\n\r\nLZMA\t- Very slow. Extreme compression.\r\n\t Levels: 1 - 14\r\n Till level 9 it is standard LZMA parameters. Levels 10 - 12 use\r\n more memory and higher match iterations so are slower. Levels\r\n 13 and 14 use larger dictionaries upto 256MB and really suck up\r\n RAM. Use these levels only if you have at the minimum 4GB RAM on\r\n your system.\r\n\r\nPPMD\t- Slow. Extreme compression for Text, average compression for binary.\r\n In addition PPMD decompression time is also high for large chunks.\r\n This requires lots of RAM similar to LZMA.\r\n\t Levels: 1 - 14.\r\n\r\nAdapt\t- Very slow synthetic mode. Both Bzip2 and PPMD are tried per chunk and\r\n\t better result selected.\r\n\t Levels: 1 - 14\r\nAdapt2\t- Ultra slow synthetic mode. Both LZMA and PPMD are tried per chunk and\r\n\t better result selected. Can give best compression ratio when splitting\r\n\t file into multiple chunks.\r\n\t Levels: 1 - 14\r\n Since both LZMA and PPMD are used together memory requirements are\r\n quite extensive especially if you are also using extreme levels above\r\n 10. For example with 64MB chunk, Level 14, 2 threads and with or without\r\n dedupe, it uses upto 3.5GB physical RAM and requires 6GB of virtual\r\n memory space.\r\n\r\nIt is possible for a single chunk to span the entire file if enough RAM is\r\navailable. However for adaptive modes to be effective for large files, especially\r\nmulti-file archives splitting into chunks is required so that best compression\r\nalgorithm can be selected for textual and binary portions.\r\n\r\nCaveats\r\n=======\r\nThis utility is not meant for resource constrained environments. Minimum memory\r\nusage (RES/RSS) with barely meaningful settings is around 10MB. This occurs when\r\nusing the minimal LZFX compression algorithm at level 2 with a 1MB chunk size and\r\nrunning 2 threads.\r\nNormally this utility requires lots of RAM depending on compression algorithm,\r\ncompression level, and dedupe being enabled. Larger chunk sizes can give\r\nbetter compression ratio but at the same time use more RAM.\r\n\r\n","tagline":"A Parallel Compression and Deduplication utility","note":"Don't delete this file! It's used internally to help with page regeneration."} |