HDF5 facet function data not able to be read when reading mesh on mpi_com_self()

Question

I have a mesh and facetfunction written to stored in an HDF5 file.

When I read both in using mpi_comm_world(), I have no issue, but when I read in the facet function I get errors.

Unable to read meshfunction topology.
*** Reason:  Mesh dimension mismatch.
*** Where:   This error was encountered inside HDF5File.cpp.
*** Process: 21

h5file = dl.HDF5File(mpi_comm_self(), "meshandfacetfunction.h5", "r")
mesh = dl.Mesh()
h5file(mesh, "/mesh", False)
facets = dl.FacetFunction('size_t', mesh)
h5file.read(facets, "/facets")        #issue with reading in facets
h5file.close()

How can I get both the mesh and the facets read in with mpi_comm_self(). I'm running a problem where I want the mesh and facets to be identical on every mpi-process, instead of partitioned across processors.

Answer 1

This is quite strange behaviour, I agree... But read(Mesh) does not communicate between processes within MPI communicator while read(MeshFunction) does (via all_to_all).

Try reading mesh on mpi_comm_self() and then instantiate new HDF5 file on mpi_comm_world() and read FacetFunction within world communicator, i.e.

h5file_mesh = dl.HDF5File(dl.mpi_comm_self(), "meshandfacetfunction.h5", "r")
mesh = dl.Mesh()
h5file_mesh.read(mesh, "/mesh", False)

h5file_ffunct = dl.HDF5File(dl.mpi_comm_world(), "meshandfacetfunction.h5", "r")
facets = dl.FacetFunction('size_t', mesh)
h5file_ffunct.read(facets, "/facets")        #issue with reading in facets

h5file_mesh.close()
h5file_ffunct.close()

Does it work?