Skip to content

Appending compressed logs

 1 min An old mirror script appended complete gzip members, letting each run add compressed output without reopening the previous stream.
Published

Oct 31, 2011

My Arch mirror script sent its timestamped log output through gzip with append redirection. A later run started another compressor and appended another compressed stream to the same monthly file.

That works because a gzip file may contain consecutive members. Each has its own framing; a decompressor can read them in order as one continuous output. RFC 1952 defines that file structure.

A small example, using a disposable path:

demo_log=$(mktemp)
printf 'first run\n' | gzip >> "$demo_log"
printf 'second run\n' | gzip >> "$demo_log"
gzip -dc "$demo_log"

The output is both lines. The second command does not resume the first compressor’s internal state; it creates another complete member.

The mirror’s logger added process identity and a timestamp before compression. Its stdout and stderr plumbing also sent readable copies toward the terminal, while the log destination was selected through the corresponding output-stream variable.

The format permits concatenation, not arbitrary concurrent writes. Separate writers interleaving their compressed bytes would not produce clean consecutive members. An interrupted compressor can also leave an incomplete final member. Those are writer-lifecycle problems that append mode does not solve.

For completed sequential runs, the result is simple: keep the old compressed log untouched and append another self-contained piece. No decompress-and-recompress pass is required just to add the next run.