Back to homepage

@kokoalberti kokoalberti.com/articles/

Guide to GeoTIFF compression and optimization with GDAL

Published 2018-08-16 #geo #geofolio #gdal #geotiff #compression

With the fast ZSTD compression (GDAL 2.3 and above) and Limited Error Raster Compression (GDAL 2.4) becoming available in our favourite geospatial toolkit, I thought it would be interesting to run some benchmarks and write a guide on compressing and optimizing GeoTIFF files using the latest versions of GDAL.

Especially if you're working with GDAL's virtual file systems and cloud-optimized GeoTIFFs, deciding on the right compression algorithm and creation options can make a significant difference to indicators such as file size, processing time, and the amount of time and bandwidth consumed when accessing geospatial data over a network.

What's the plan

We're going to run a benchmark to test compression ratio and read/write speeds for various data types and algorithms with their respective configuration options.

Test files

Three test files with commonly used data types have been created for this test: byte.tif, int16.tif, and float32.tif. Each file has been cropped to be around 50Mb each in its uncompressed condition. See the notes and comments section for a download link and more information. Just to give an impression, this is what the Byte, Int16, and float32 images look like zoomed out:

Three sample images

It is important to mention here that these are just test files, and that your results will vary depending on the type of data you're working with. So interpret the results not as the absolute truth, but rather as an indication and a guideline of what the tradeoffs between the different algorithms and configuration options are. Experiment with your own data to find out what works best for you.

Algorithms and creation options

We are going to test the following compression algorithms:

The Deflate, LZW and ZSTD algorithms support the use of predictors, which is a method of storing only the difference from the previous value instead of the actual value. There are three predictor settings we can test:

Predictors work especially well when there is some spatial correlation in the data, and pixels have values which are similar to their neighbours. As the name suggests, the floating point predictor can only be used on floating point data.

The ZSTD and Deflate algorithms support custom compression levels throught the ZLEVEL and ZSTD_LEVEL creation options. Compression levels can be lowered to speed up the algorithm at the expense of the compression ratio, or raised to improve the compression ratio at the expense of speed. We'll test the ZSTD and Deflate algorithms with a low, the default, and a high compression level.

All the different tests and corresponding creation options that are used in the benchmark are defined in the config.ini file.

Benchmark setup

The benchmark is run inside a Docker container on a temporary AWS i3.large EC2 instance, just to ensure a clean machine with a local SSD volume and nothing else running in the background. GDAL is built from source in the Dockerfile, and version 2.4.0dev is used because it is the first version to fully support LERC and ZSTD compression with the predictor option.

All test commands are run in turn by the main gtiff_benchmark.py script, which calls a different gdal_translate command using perf stat for each combination of tests, something like:

perf stat -x ^ -r 5 gdal_translate \
    -q int16.tif output.tif \
    -co COMPRESS=DEFLATE \
    -co PREDICTOR=2

The gdal_translate command is repeated by perf stat with the relevant creation options 5 times. The final results (time, compression ratio, etc) are logged to a CSV file by the main script.

Benchmark results

Write speed

The write tests measure the time required to write each of the 50Mb files with the corresponding compression settings. The speed in the table below is relative to original file size, so it is an indication of how many Mb of raw data can be compressed in one second. For example, if the 50Mb file was written in 0.5 seconds, a speed of 100Mb/s is reported below. The results are as follows:

Results table of write speed

A few observations worth noting:

Compression ratio

Here is the full results table for the compression ratio tests:

Results table of the compression ratio

This also yielded some interesting observations:

Read speed

Here is the full results table for the read speed tests:

Results table for the read speed

Further considerations

This section dives in to some additional issues and GDAL settings that might have an impact on performance. Refer to the GeoTIFF File Format and config options documentation for more information on how they work and how to apply them using the command line tools.

Tiling

When the creation option TILED=YES is enabled, data is stored and compressed in blocks (tiled) rather than line by line (stripped). The size of the tiles can be configured using the BLOCKXSIZE=n and BLOCKYSIZE=n creation options. Tiling is not much use when you're always reading the entire file or only a single pixel at a time, as respectively all or only one tile or strip will have to be decompressed.

For accessing subsets though, tiling can make a difference. For example, the grids below show an image with 100 equally sized tiles (left) and the same number of strips (right). To read data from the red subset, the yellow area will have to be decompressed. In the tiled image we will have to decompress only 9 tiles, whereas in the stripped image on the right we'll have to decompress around twice as many strips.

Example to explain tiling vs. striped

Creating a tiled image may take a little bit longer than one without tiles, so it's best to experiment and check for yourself whether tiling works for you.

Limited Error Raster Compression (LERC)

LERC is available from GDAL 2.4 onwards, and is designed to obtain a better compression ratio at the expense of the precision of the data. The key feature is that LERC lets the user, using the MAX_Z_ERROR creation option, define exactly how lossy the algorithm is allowed to be. LERC can be used standalone as COMPRESS=LERC or in combination with ZSTD or Deflate as COMPRESS=LERC_ZSTD and COMPRESS=LERC_DEFLATE.

As a quick experiment I ran the benchmark again on our floating point dataset with COMPRESS=LERC_DEFLATE, gradually increasing the MAX_Z_ERROR to see the effect on the compression ratio. Remember that in our original results, the best possible compression ratio achieved with Deflate and a floating point predictor was 1.64. The results are interesting to say the least:

Results table for LERC compression ratio

If we reduce the precision to within even 1mm of elevation using MAX_Z_ERROR=0.001, the compression ratio already jumps to 2.07, that's a space saving of more than 50%. The ratio increases rapidly as the precision is reduced, and at a precision of 1 meter with MAX_Z_ERROR=1.0, the space saving is above 80% with a compression ratio of 5.3. So, if you're using floating point data, take a minute to think about what precision is required for your particular application. If it's anything above zero, give LERC some thought.

Overviews

Overviews are duplicate versions of your original data, but resampled to a lower resolution. This is useful when you're rendering or reading zoomed out versions of your data, and there is too much data to be able to calculate these overviews on the fly.

The easiest way to add overviews to a GeoTIFF is to use the gdaladdo utility. Because the overview are in fact datasets as well, they can also be compressed with various algorithms and predictors, much in the same way as your original dataset. In addition to this, you need to decide what resampling algorithm to use in order to convert many high resolution values to a single low resolution value for use in the overview. Here too there are different methods: fast and imprecise such as nearest neighbour, and more complicated ones such as average or cubicspline. Check the gdaladdo documentation and experiment.

NUM_THREADS

You can set NUM_THREADS=ALL_CPUS creation option to enable multi-threaded compression for slow algorithms like Deflate. I gained about a 30% performance improvement when compressing the int16.tif file with COMPRESS=DEFLATE and NUM_THREADS=ALL_CPUS. The NUM_THREADS option only works on compression, so there is no effect on read times.

GDAL_DISABLE_READDIR_ON_OPEN

When GDAL opens a file from a local directory or network, it will automatically try and establish a list of other files present in that location. This can result in a delay when accessing files over a network, or in a local directory with more than a few hundred other files. The behaviour can be prevented by setting the configuration option GDAL_DISABLE_READDIR_ON_OPEN to TRUE.

GDAL_CACHEMAX

The GDAL_CACHEMAX configuration option can be used to set the raster block size cache. In some quick tests I did not see any performance improvements in read or write speeds. Perhaps this was because the default values are already large enough in later versions of GDAL, or that I was not working with extremely large files where caching would be required. Others reported much faster read times on large files, so it may be worth experimenting with. You can set it to 1000MB using --config GDAL_CACHEMAX 1000 in your GDAL command.

Compressing Imagery

While compressing imagery is outside of the scope of this article, I wanted to mention it nevertheless to be complete. When the data you're trying to compress consists of imagery (aerial photographs, true-color satellite images, or colored maps) you can use lossy algorithms such as JPEG to obtain a much improved compression ratio. A lossy algorithm such as JPEG is not suitable though for compressing data where the precision is important, which is why I skipped it here in this benchmark. If you're compressing imagery, have a look at the article by Paul Ramsey for some results and best practices on compressing GeoTIFFs with JPEG compression.

Conclusions

I hope this article has given a thorough overview of how the different compression algorithms in the GeoTIFF drivers perform, and what configuration options can be used to squeeze some extra performance from your GeoTIFF files.

Most importantly, keep experimenting and use the techniques in this article to find out what works best for your data and access methods. Be aware of the type of data you have, and use the appropriate predictors and z-levels for a quick performance boost.

As for the future, the additions of ZSTD (fast) and LERC (magic) in the latest versions of GDAL are really promising, I can't wait to put them to the test in some real world scenarios.

Notes and comments

Thanks for reading! While there is no comment functionality on this website, I do appreciate any feedback, questions, improvements, and other ideas about this article. Feel free to contact me directly via e-mail at koko@geofolio.org.

The test files are an extract from soilgrids.org (Byte), a DEM extracted from an SRTM tile (Int16), and a hi-res DEM from the USGS National Elevation Dataset (Float32). The test files can be downloaded from here.

There are a few other blog posts on this subject that you can have a look at for additional information:

The code for the benchmarks is available at: https://www.github.com/kokoalberti/geotiff-benchmark