1 line
No EOL
17 KiB
JSON
1 line
No EOL
17 KiB
JSON
{"name":"Pcompress","tagline":"A Parallel Compression and Deduplication utility","body":"Introduction\r\n============\r\nPcompress is an attempt to revisit **Data Compression** using unique combinations of existing and some new techniques. Both high compression ratio and performance are key goals along with the ability to leverage all the cores on a multi-core CPU. It also aims to bring to the table scalable, high-throughput Global **Deduplication** of archival storage. The deduplication capability is also available for single-file compression modes providing very interesting capabilities. Other projects providing some of these features include [Lrzip](http://ck.kolivas.org/apps/lrzip/), [eXdupe](http://www.exdupe.com/). Full archivers providing some of the similar features include the excellent [FreeArc](http://freearc.org/) and [PeaZIP](http://peazip.sourceforge.net/). Pcompress is not an archiver but provides a unique combination of features to both maximize compression ratio and provide high speed.\r\n\r\nFeatures\r\n========\r\n* **Parallel**: Compress and Decompress in parallel by splitting input data into chunks. With Content-Aware Deduplication chunks are split at a content-defined boundary to improve Dedulication and compression.\r\n* **Scalable**: Chunks are independent and can scale to any number of cores provided enough memory is available.\r\n* **Deduplication**: High-speed Content-aware chunk-level Deduplication based on Rabin fingerprinting. Duplicate comparison uses exact byte-for-byte comparison and techniques to reduce Dedupe index size.\r\n* **Global Deduplication**: Scalable, fast Content-aware Deduplication across the entire dataset. Duplicate comparison uses cryptographic hash based matching to detect duplicate blocks. By default SHA256 is used but this can be changed via an environment variable. Global Deduplication can reach petascale using only a relatively tiny in-memory index.\r\n* **Delta Compression**: Deduplication also provides Delta Compression of closely matching chunks using [Bsdiff](http://www.daemonology.net/bsdiff/). [Minhashing](http://en.wikipedia.org/wiki/MinHash) is used to detect similar chunks.\r\n* **Fixed Block option**: Fixed block Deduplication is also supported and works extremely fast.\r\n* **Metadata Compression**: The Dedupe Index is transformed and compressed.\r\n* **Multiple Algorithms**: Support for multiple compression algorithms like LZMA, LZMA-Multithreaded, Bzip2, PPMD, LZ4 etc. Adaptive modes allow selecting an algorithm per chunk based on heuristics.\r\n* **Strong Data Integrity**: Strong Data Integrity verification with option of using BLAKE2, SHA2 or KECCAK. Headers are also checksummed using CRC32.\r\n* **Filters**: Pre-compression filters: LZP, Delta2. These improve compression ratio across the board at a little extra computational cost.\r\n* **LZP**: LZP (Lempel-Ziv Prediction) searches for repeating patterns of bytes.\r\n* **Delta2**: Delta2 Encoding probes for embedded tables of numeric data and Run Length encodes arithmetic sequences at high throughput.\r\n* **Matrix Transform**: A form of [Matrix transpose](http://moinakg.wordpress.com/2012/12/13/linear-algebra-meets-data-compression/) is used to better compress the Dedupe Index.\r\n* **Encryption**: Support for AES Encryption using Key generation based on the strong [Scrypt](http://en.wikipedia.org/wiki/Scrypt) algorithm. AES is used in CTR mode.\r\n* **Message Authentication**: Encryption mode uses HMAC, Skein MAC or Keccak MAC for Data Integrity and Authentication. The MAC approach from iSCSI is followed for improved security ([http://tonyarcieri.com/all-the-crypto-code-youve-ever-written-is-probably-broken](http://tonyarcieri.com/all-the-crypto-code-youve-ever-written-is-probably-broken).\r\n* **Metadata**: Low metadata overhead.\r\n* **Overlapped processing**: Overlapped computation and I/O to maximize throughput.\r\n* **Streamable**: Ability to work in streaming pipe mode reading from stdin and writing to stdout.\r\n* **Custom Allocator**: Uses an internal mempool allocator to speed up repeated allocation of similarly sized chunks. Option to disable this at runtime is provided.\r\n* **Solid Mode**: Given enough available memory an entire file can be compressed inside a single chunk. This however is mostly a single-threaded operation.\r\n* **Padding**: A compressed archive or file can be zero-padded to round off to a multiple of a block size for certain storage media like Tapes.\r\n\r\nOther open-source deduplication software like [OpenDedup](http://opendedup.org/) and [LessFS](http://www.lessfs.com/wordpress/) use fixed block dedupe only. Some software like [BackupPC](http://backuppc.sourceforge.net/) does file-level dedupe only (single-instance storage). Of course OpenDedup and LessFS are Fuse based filesystems doing inline dedupe of primary storage while Pcompress is only meant for archival storage as of today.\r\n\r\nNOTE: This utility is Not an archiver. It compresses only single files or datastreams. To archive use something else like tar, cpio or pax.\r\n\r\nNEWS\r\n====\r\nBlog: [https://moinakg.wordpress.com/tag/pcompress/](https://moinakg.wordpress.com/tag/pcompress/).\r\n\r\nReleases: [http://freecode.com/projects/pcompress](http://freecode.com/projects/pcompress)\r\n\r\nCompression Benchmarks\r\n======================\r\n[Benchmarks Part #1](https://moinakg.wordpress.com/2012/11/01/compression-benchmarks/)\r\n\r\n[Benchmarks Part #2](https://moinakg.wordpress.com/2012/11/03/compression-benchmarks-2/)\r\n\r\nDeduplication Chunking Analysis\r\n===============================\r\n[Content Defined Chunking #1](https://moinakg.wordpress.com/2012/11/11/inside-content-defined-chunking-in-pcompress/)\r\n\r\n[Content Defined Chunking #2](https://moinakg.wordpress.com/2012/11/15/inside-content-defined-chunking-in-pcompress-part-2/)\r\n\r\nRelease Downloads\r\n=================\r\n[http://code.google.com/p/pcompress/downloads/list](http://code.google.com/p/pcompress/downloads/list)\r\n\r\nUsage\r\n=====\r\n\r\n To compress a file:\r\n pcompress -c <algorithm> [-l <compress level>] [-s <chunk size>] <file>\r\n Where <algorithm> can be the folowing:\r\n lzfx - Very fast and small algorithm based on LZF.\r\n lz4 - Ultra fast, high-throughput algorithm reaching RAM B/W at level1.\r\n zlib - The base Zlib format compression (not Gzip).\r\n lzma - The LZMA (Lempel-Ziv Markov) algorithm from 7Zip.\r\n lzmaMt - Multithreaded version of LZMA. This is a faster version but\r\n uses more memory for the dictionary. Thread count is balanced\r\n between chunk processing threads and algorithm threads.\r\n bzip2 - Bzip2 Algorithm from libbzip2.\r\n ppmd - The PPMd algorithm excellent for textual data. PPMd requires\r\n at least 64MB X CPUs more memory than the other modes.\r\n\r\n libbsc - A Block Sorting Compressor using the Burrows Wheeler Transform\r\n like Bzip2 but runs faster and gives better compression than\r\n Bzip2 (See: libbsc.com).\r\n\r\n adapt - Adaptive mode where ppmd or bzip2 will be used per chunk,\r\n depending on heuristics. If at least 50% of the input data is\r\n 7-bit text then PPMd will be used otherwise Bzip2.\r\n adapt2 - Adaptive mode which includes ppmd and lzma. If at least 80% of\r\n the input data is 7-bit text then PPMd will be used otherwise\r\n LZMA. It has significantly more memory usage than adapt.\r\n none - No compression. This is only meaningful with -D and -E so Dedupe\r\n can be done for post-processing with an external utility.\r\n <chunk_size> - This can be in bytes or can use the following suffixes:\r\n g - Gigabyte, m - Megabyte, k - Kilobyte.\r\n Larger chunks produce better compression at the cost of memory.\r\n <compress_level> - Can be a number from 0 meaning minimum and 14 meaning\r\n maximum compression.\r\n\r\nNOTE: The option \"libbsc\" uses Ilya Grebnov's block sorting compression library\r\n from http://libbsc.com/ . It is only available if pcompress in built with\r\n that library. See INSTALL file for details.\r\n \r\n To decompress a file compressed using above command:\r\n pcompress -d <compressed file> <target file>\r\n\r\n To operate as a pipe, read from stdin and write to stdout:\r\n pcompress -p ...\r\n\r\n Attempt Rabin fingerprinting based deduplication on chunks:\r\n pcompress -D ...\r\n pcompress -D -r ... - Do NOT split chunks at a rabin boundary. Default\r\n is to split.\r\n\r\n Perform Delta Encoding in addition to Identical Dedup:\r\n pcompress -E ... - This also implies '-D'. This performs Delta Compression\r\n between 2 blocks if they are 40% to 60% similar. The\r\n similarity %age is selected based on the dedupe block\r\n size to balance performance and effectiveness.\r\n pcompress -EE .. - This causes Delta Compression to happen if 2 blocks are\r\n at least 40% similar regardless of block size. This can\r\n effect greater final compression ratio at the cost of\r\n higher processing overhead.\r\n\r\n Number of threads can optionally be specified: -t <1 - 256 count>\r\n Other flags:\r\n '-L' - Enable LZP pre-compression. This improves compression ratio of all\r\n algorithms with some extra CPU and very low RAM overhead. Using\r\n delta encoding in conjunction with this may not always be beneficial.\r\n\r\n '-P' - Enable Adaptive Delta Encoding. It can improve compresion ratio further\r\n for data containing tables of numerical values especially if those are\r\n in an arithmetic series. In this implementation basic Delta Encoding is\r\n combined with Run-Length encoding and Matrix transpose\r\n NOTE - Both -L and -P can be used together to give maximum benefit on most\r\n datasets.\r\n\r\n '-S' <cksum>\r\n - Specify chunk checksum to use: CRC64, BLAKE2-256, BLAKE2-512, SHA256 and\r\n SHA512. Default one is BLAKE2-256. The implementation actually uses SKEIN\r\n 512-256. This is 25% slower than simple CRC64 but is many times more\r\n robust than CRC64 in detecting data integrity errors. BLAKE is a\r\n finalist in the NIST SHA-3 standard selection process and is one of\r\n the fastest in the group, especially on x86 platforms. BLAKE2 is faster\r\n than BLAKE and even faster than MD5.\r\n BLAKE2 512-256 is about 60% faster than SHA 512-256 on x64 platforms.\r\n\r\n '-F' - Perform Fixed Block Deduplication. This is faster than fingerprinting\r\n based content-aware deduplication in some cases. However this is mostly\r\n usable for disk dumps especially virtual machine images. This generally\r\n gives lower dedupe ratio than content-aware dedupe (-D) and does not\r\n support delta compression.\r\n '-M' - Display memory allocator statistics\r\n '-C' - Display compression statistics\r\n\r\nNOTE: It is recommended not to use '-L' with libbsc compression since libbsc uses\r\n LZP internally as well.\r\n\r\n Encryption flags:\r\n '-e <ALGO>'\r\n Encrypt chunks using the given encryption algorithm. The algo parameter\r\n can be one of AES or SALSA20. Both are used in CTR stream encryption\r\n mode.\r\n The password can be prompted from the user or read from a file. Unique\r\n keys are generated every time pcompress is run even when giving the same\r\n password. Of course enough info is stored in the compresse file so that\r\n the key used for the file can be re-created given the correct password.\r\n\r\n Default key length if 256 bits but can be reduced to 128 bits using the\r\n '-k' option.\r\n\r\n The Scrypt algorithm from Tarsnap is used\r\n (See: http://www.tarsnap.com/scrypt.html) for generating keys from\r\n passwords. The CTR mode AES mechanism from Tarsnap is also utilized.\r\n\r\n '-w <pathname>'\r\n Provide a file which contains the encryption password. This file must\r\n be readable and writable since it is zeroed out after the password is\r\n read.\r\n\r\n '-k <key length>'\r\n Specify the key length. Can be 16 for 128 bit keys or 32 for 256 bit\r\n keys. Default value is 32 for 256 bit keys.\r\n\r\nNOTE: When using pipe-mode via -p the only way to provide a password is to use '-w'.\r\n\r\nEnvironment Variables\r\n=====================\r\n\r\nSet ALLOCATOR_BYPASS=1 in the environment to avoid using the the built-in allocator. Due to the the way it rounds up an allocation request to the nearest slab the built-in allocator can allocate extra unused memory. In addition you may want to use a different allocator in your environment.\r\n\r\nExamples\r\n========\r\n\r\nCompress \"file.tar\" using bzip2 level 6, 64MB chunk size and use 4 threads. In addition perform identity deduplication and delta compression prior to compression.\r\n\r\n pcompress -D -E -c bzip2 -l6 -s64m -t4 file.tar\r\n\r\nCompress \"file.tar\" using extreme compression mode of LZMA and a chunk size of of 1GB. Allow pcompress to detect the number of CPU cores and use as many threads.\r\n\r\n pcompress -c lzma -l14 -s1g file.tar\r\n\r\nCompress \"file.tar\" using lz4 at max compression with LZ-Prediction pre-processing and encryption enabled. Chunksize is 100M:\r\n\r\n pcompress -c lz4 -l3 -e -L -s100m file.tar\r\n\r\nCompression Algorithms\r\n======================\r\n\r\nLZFX\t- Ultra Fast, average compression. This algorithm is the fastest overall.\r\n\t Levels: 1 - 5\r\nLZ4\t- Very Fast, better compression than LZFX.\r\n\t Levels: 1 - 3\r\nZlib\t- Fast, better compression.\r\n\t Levels: 1 - 9\r\nBzip2\t- Slow, much better compression than Zlib.\r\n\t Levels: 1 - 9\r\nLibbsc - A new Block-Sorting compressor similar conceptually to Bzip2 but gives\r\n much better compression.\r\n Levels: 1 - 9\r\n\r\nLZMA\t- Very slow. Extreme compression.\r\n\t Levels: 1 - 14\r\n Till level 9 it is standard LZMA parameters. Levels 10 - 12 use\r\n more memory and higher match iterations so are slower. Levels\r\n 13 and 14 use larger dictionaries upto 256MB and really suck up\r\n RAM. Use these levels only if you have at the minimum 4GB RAM on\r\n your system.\r\n\r\nLzmaMt - Extreme compression, faster than plain LZMA as it is multithreaded.\r\n Compression ratio is only slightly less than plain LZMA.\r\n\r\nPPMD\t- Slow. Extreme compression for Text, average compression for binary.\r\n In addition PPMD decompression time is also high for large chunks.\r\n This requires lots of RAM similar to LZMA.\r\n\t Levels: 1 - 14.\r\n\r\nAdapt\t- Very slow synthetic mode. Both Bzip2 and PPMD are tried per chunk and\r\n\t better result selected.\r\n\t Levels: 1 - 14\r\nAdapt2\t- Ultra slow synthetic mode. Both LZMA and PPMD are tried per chunk and\r\n\t better result selected. Can give best compression ratio when splitting\r\n\t file into multiple chunks.\r\n\t Levels: 1 - 14\r\n Since both LZMA and PPMD are used together memory requirements are\r\n quite extensive especially if you are also using extreme levels above\r\n 10. For example with 64MB chunk, Level 14, 2 threads and with or without\r\n dedupe, it uses upto 3.5GB physical RAM and requires 6GB of virtual\r\n memory space.\r\n\r\nIt is possible for a single chunk to span the entire file if enough RAM is available. However for adaptive modes to be effective for large files, especially multi-file archives, splitting into chunks is required so that best compression algorithm can be selected for textual and binary portions.\r\n\r\nMemory Usage\r\n============\r\nAs can be seen from above memory usage can vary greatly based on compression/\r\npre-processing algorithms and chunk size. A variety of configurations are possible\r\ndepending on resource availability in the system.\r\n\r\nThe minimum possible meaningful settings while still giving about 50% compression\r\nratio and very high speed is with the LZFX algorithm with 1MB chunk size and 2\r\nthreads:\r\n\r\n pcompress -c lzfx -l2 -s1m -t2 <file>\r\n\r\nThis uses about 6MB of physical RAM (RSS). Earlier versions of the utility before\r\nthe 0.9 release comsumed much more memory. This was improved in the later versions.\r\nWhen using Linux the virtual memory consumption may appear to be very high but it\r\nis just address space usage rather than actual RAM and should be ignored. It is only\r\nthe RSS that matters. This is a result of the memory arena mechanism in Glibc that\r\nimproves malloc() performance for multi-threaded applications.\r\n","google":"UA-36422648-1","note":"Don't delete this file! It's used internally to help with page regeneration."} |