Splits stdin onto chunk files and writes resulting TAR archive to stdout.
It can help to handle data stream of unknown size without hitting a disk. As long as creating TAR archive as a stream needs to know file size in advance, that is why the idea of the chunker is to split input stream onto chunks with known size.
Initial motivation was to use it for backup streaming to tarsnap.com.
-
pkg install tar-stream-chunker
-
or
cd /usr/ports/archivers/tar-stream-chunker && make install clean
$ tar_stream_chunker { --file-name | -f } <file-name> { --chunk-size | -s } <bytes>
And redirect its stdout to your target and that’s it.
PostgreSQL dump what is streamed to tarsnap:
$ pg_dump | tar_stream_chunker -f dump.sql -s 500000000 | tarsnap -cvf dump.daily.$(date +%Y%m%d.%H%M%S) @-
Let’s see what’s inside a resulting TAR:
$ cat 3MB-file | tar_stream_chunker --file-name file --chunk-size 1000000 > result.tar
$ tar -tf result.tar
file.00001
file.00002
file.00003
And original input can be re-assembled as follows:
$ tar -xf result.tar; cat file.* > original-3MB-file
$ tar -xOf result.tar > original-3MB-file