list_timings in parallel (supercomputer)

Question

I'm working on my master thesis and want to make a scaling and profiling of FEniCS. Therefor I will use the supercomputer at Juelich research centre (JUQUEEN).

At the moment I implement some test cases on my local workstation and look for possibilities to collect time and memory information of my fenics applications.
In this context I found the function list_timings() which gives me a nice summary. But if I'm running in parallel each process prints out its own table so that the output becomes useless.
Is there a way that just the master process prints out mean values of the timers?
Are there other functions to get more interesting information of my program?

best regards,
Thomas

Comments

There used to be just one table in parallel as well, so I posted a question on the mailing list

Answer 1

Hi,

Firstly, that output isn't useless, just not processed for your purposes. It would probably be fairly simple to parse it and average it across all processes. Plus, you could do statistics on the profiles of each process which itself is an interesting discussion (e.g.: Mean time +- distribution). Then you can discuss scaling, etc in an interesting way (and share it with us!!)

Also, what backend are you using? If PETSc, you can try to get it's profiling output.
e.g.: Add
PETScOptions.set('log_summary','profile.txt')
to the beginning of your file. It will print out some PETSc statistics to profile.txt, which should collect everything from all the MPI processes.

Comments

• It takes a lot of computing time to write on std::out if there are so many mpi tasks.
• And there is no guaranty that you do not have a "mixing" of the output in one line. Then it is not possible to parse it.

It would be very nice if there is a function inside dolfin which does these statistics. In this case you do not have to use special profiling tools (e.g. to improve your applications or to get information)

Yes I'm using PETSc and I'm already using this "petsc profiling".

How can I share it with you?