mirror of https://github.com/vasi/pixz
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
154dc842a1
Support streaming input, still decompressing in parallel. Extract mode is not supported while streaming. Heuristics are used to skip the file index when in streaming mode. This also ensures pixz properly reads files with bizarre block sizes, or with multiple streams. |
12 years ago | |
---|---|---|
.gitignore | 14 years ago | |
LICENSE | 13 years ago | |
Makefile | 12 years ago | |
README | 12 years ago | |
TODO | 12 years ago | |
common.c | 12 years ago | |
cpu.c | 15 years ago | |
endian.c | 12 years ago | |
list.c | 12 years ago | |
pixz.c | 12 years ago | |
pixz.h | 12 years ago | |
read.c | 12 years ago | |
test.sh | 12 years ago | |
write.c | 12 years ago |
README
Pixz (pronounced 'pixie') is a parallel, indexing version of XZ: https://github.com/vasi/pixz The existing XZ Utils ( http://tukaani.org/xz/ ) provide great compression in the .xz file format, but they have two significant problems: * They are single-threaded, while most users nowadays have multi-core computers. * The .xz files they produce are just one big block of compressed data, rather than a collection of smaller blocks. This makes random access to the original data impossible. With pixz, both these problems are solved. The most useful commands: $ pixz foo.tar foo.tpxz # Compress and index a tarball, multi-core $ pixz -l foo.tpxz # Very quickly list the contents of the compressed tarball $ pixz -d foo.tpxz foo.tar # Decompress it, multi-core $ pixz -x dir/file < foo.tpxz | tar x # Very quickly extract a file, multi-core. # Also verifies that contents match index. $ tar -Ipixz -cf foo.tpxz foo # Create a tarball using pixz for multi-core compression $ pixz bar bar.xz # Compress a non-tarball, multi-core $ pixz -d bar.xz bar # Decompress it, multi-core Specifying input and output: $ pixz < foo.tar > foo.tpxz # Same as 'pixz foo.tar foo.tpxz' $ pixz -i foo.tar -o foo.tpxz # Ditto. These both work for -x, -d and -l too, eg: $ pixz -x -i foo.tpxz -o foo.tar file1 file2 ... # Extract the files from foo.tpxz into foo.tar $ pixz foo.tar # Compress it to foo.tpxz, removing the original $ pixz -d foo.tpxz # Extract it to foo.tar, removing the original Other flags: $ pixz -1 foo.tar # Faster, worse compression $ pixz -9 foo.tar # Better, slower compression $ pixz -p 2 foo.tar # Cap the number of threads at 2 $ pixz -t foo.tar # Compress but don't treat it as a tarball (don't index it) $ pixz -d -t foo.tpxz # Decompress foo, don't check that contents match index $ pixz -l -t foo.tpxz # List the xz blocks instead of files WARNING: Running pixz without the -t flag will cause it to treat the input as a tarball, as long as it looks vaguely tarball-like. This means if the file starts with at least 1024 zero bytes, pixz will assume it's empty, and truncate the output! If your input files aren't tarballs, run with -t or face possible data-loss. Compare to: plzip * About equally complex, efficient * lzip format seems less-used * Version 1 is theoretically indexable...I think ChopZip * Python, much simpler * More flexible, supports arbitrary compression programs * Uses streams instead of blocks, not indexable * Splits input and then combines output, much higher disk usage pxz * Simpler code * Uses OpenMP instead of pthreads * Uses streams instead of blocks, not indexable * Uses temp files and doesn't combine them until the whole file is compressed, high disk/memory usage Comparable tools for other compression algorithms: pbzip2 * Not indexable * Appears slow * bzip2 algorithm is non-ideal pigz * Not indexable dictzip * Not parallel Requirements: * libarchive 2.8 or later * liblzma 4.999.9-beta-212 or later (from the xz distribution)