Compressing HDF5 to run in PARAVIEW

Question

As mentioned in the related question, I am also getting huge HDF5 output files, while saving in XDMF. Recently, I ran a unit square mesh from Gmsh and the .h5 output file was 54GB.

I transfer these files from the server to my laptop to see the animation in Paraview using sftp command. It takes considerable time at a transfer rate of 2 MB/s.

In the related question, Chris suggested that h5repack can be used to compress. I also read that the .h5 file can be compressed by 33% at this link:

http://www.speedup.ch/workshops/w37_2008/HDF5-Tutorial-PDF/HDF5-Tools.pdf

It talks about various filters and compression extensions. My question is - What do people here, working on FEniCS, use in command line to compress the HDF5 file? What filters are preferable? What kind of compression works for Paraview?

In that link, they use something like this:

h5repack –f SHUF –f GZIP=1 output.he5  \compressed_output.he5

Where they are using SHUF and GZIP. (Refer the link)

Answer 1

I used the gzip filter, you can choose the compression level 1-9, I guess heavier compression is slower. Maybe you can report back with your experiences...

Comments

Hi Chris,

I used this command:

h5repack -f GZIP=1 displacement.h5 compressed.h5

to compress the file 'displacement.h5', which is a 13GB file. It took a couple of minutes. Changing the compression level from 1 to 9, I only observed a reduction of 200 MB, i.e., from 6.4 GB to 6.2 GB. Compression at level '9' took nearly twice the time it took for compression at level '1'.