Losslessness

Exceptions to Losslessness

BGZF

Genozip compression is lossless relative to the underlying data, which means that the data reconstructed during decompression is exactly identical to the original data compressed. This means that the MD5 values generated by:

FASTQ: zcat file.fastq.gz | md5sum

BAM: zcat file.bam | md5sum

are the same for the original file and the file reconstructed with genounzip.

Verification of Losslessness

When uncompressing with genounzip (and genocat with some exceptions), genounzip verifies that the reconstructed data is exactly identical to the source data, using an MD5 or Adler32 digest. More details: Verifying file integrity.

Exceptions to Losslessness

1. Compressing already-compressed files (gzip): Many tools that generate genomic files compress them into the gzip format (or a variant of it, BGZF). These typically have a .gz extension. BAM and BCF files, despite not having a .gz extension, are also gzip files.

When genozip compresses a gzip-compressed file, it first decompresses it to recover the original underlying data, and then compresses the data with Genozip. Likewise, when genounzip uncompresses a .genozip file, it recompresses the data back to gzip (or more precisely, BGZF). Since there are many gzip compression libraries, each with dozens of parameter combinations, it is possible that genounzip's gzip compression will achieve a slightly different compression level than the gzip compression of the original file, resulting in the final gzip-compressed file differing from the original gzip file. However, the underlying data is identical: "zcat myfile.gz | md5sum" should yield the same value for the original file and the reconstructed one. It is also the same value as reported by genozip when using the --md5 option.

The --bgzf=exact option instructs genounzip to attempt to identify the gzip library used for compressing the original file. If the library is identified, the gzip compression will be reproduced precisely. Using this option may cause slower decompression as the gzip library used might be slower than the one used by genounzip by default.

It is possible to know whether Genozip is able identify the gzip library used for compressing a file, using genozip --show-gz, for example:

> genozip --show-gz my-file.bam
my-file.bam: Identified as generated with libdeflate_1.7 level 6

2. Compressing already-compressed files (.bz2 .xz .zip): genozip is capable of compressing files that are already compressed with these methods. It does so by first uncompressing the file, and then recompressing with Genozip. In these cases, genounzip recompresses to .gz, not to .bz2 .xz or .zip.

To instruct genounzip to refrain from recompressing, use --bgzf=0.

3. CRAM and BCF: Genozip does not compress CRAM and BCF files natively - it first converts their data to BAM or VCF format respectively, and when uncompressing, it outputs the data as a BAM or VCF file. Other than this file format change, the compression is lossless.

4. There are some cases in which you may request genozip to change the source data before compressing it. In these cases, the digest is not calculated. These cases are:

- Using --optimize or any of the --optimize-* options

- Using --match-chrom-to-reference

- VCF: Using --GL-to-PL or --GP-to-PP

- VCF: Replacing the ID field with a sequential line number with --add-line-numbers

Questions? support@genozip.com

verify