Neural networks are attracting significant attention for solving inverse problems in models governed by differential equations, where the goal is to estimate model parameters based on observational data. They can be employed either as surrogate models or as solvers. In the context of Bayesian inverse problems, estimating model paramters and their associated uncertainties from observational data is computationally expensive. Uncertainty Quantification Variational AutoEncoders (UQ-VAE) offers an effective approach for addressing these problems. We proposed a modification to the existing UQ-VAE loss function to achieve a more robust theoretical result. This modification enhances the performances of UQ-VAEs, though it introduces a higher computational cost due to the inclusion of a sample mean in the loss evaluation. To mitigate this cost, we replaced the sample mean with a more efficient term that preserves the same theoretical guarantee. As a result, the UQ-VAEs are not anymore prone to errors stemming from the slow convergence of the sample mean. Moreover, we achieved improved estimates of both the mean and covariance matrix of the posterior distribution of the parameters. We tested the improved UQ-VAEs on two problems: a Laplace equation with a non-constant diffusion coefficient and a 0D cardiocirculatory model where we estimated cardiac elastances, vascular resistances and compliances using synthetic hypertensive observational data.