There are two aspects:
- algorithms for elliptic/parabolic problems are getting unstable with low diffusivities
- flows are getting unstable with low viscosities
First problem can be saved some stabilization, e.g. SUPG/PSPG, interior penalty, ... and/or using finer mesh/timestep.
Second issue is pretty troublesome. It often means that flow is turbulent. It can theoretically be resolved also by sufficiently fine mesh/timestep but in practice this is rarely achievable (direct numerical simulation - DNS). Therefore so-called turbulence models are being used. There are few classes of turbulence models (large eddy simulation - LES, Reynolds averaged Navier-Stokes - RANS, Reynolds stress models) according to how much is modeled and how much is actually calculated which is the compromise between reality and computational effort.
To your question, yes, sufficiently increasing viscosity definitely helps to avoid these sort of problems but, of course, you solve problem with different viscosity. Reading your statement: viscosity is of order 10^-5 which is neglectable, I have to warn you, small viscosity does not mean that viscosity effects are negligible. It is much more complicated and it definitely does not hold that solution of Navier-Stokes problem tends to solution of Euler problem as viscosity $\rightarrow 0$.
This question is very suitable to being asked in scicomp, altough it would need some improvement - you would be probably asked for Grashof and Rayleigh numbers. There is much broader audience which can answer your (not FEniCS specific) question on scicomp.
To help you, I would calculate Grashof and Rayleigh numbers to estimate whether turbulence occurs. If they're sufficiently low I would stick to fine mesh and some stable discretization in time. Use either
- pseudo time marching with backward Euler discretization, xor
- discretization of stationary problem with algebraic under-relaxation (available as
NewtonSolver.parameters["relaxation_parameter"])
as it is very improbable that you succeed finding stationary solution directly with such low viscosity. (It is also possible that there is no stationary solution to such a flow problem, especially if turbulence is the issue.) If this does not help, I would turned to some stabilization - you can start with advection-diffusion demo for (naive) implementation of SUPG in FEniCS.
