Autoregressive and Recurrent models are a foundational technology driving much of the current AI advancements, powering applications like large language models, molecular generation, and neural PDE solvers. Despite their remarkable success, these models face a significant challenge: they struggle to assess the confidence in their predictions, especially when tasked with making predictions outside their training data. This limitation raises safety concerns in many applications, where understanding the uncertainty behind a prediction is as important as the prediction itself. As such, ensuring reliable uncertainty quantification is, therefore, essential for enhancing model safety and performance. To address this shortcoming, we present BARNN: a variational Bayesian Autoregressive and Recurrent Neural Network. BARNN aims to provide a principled way to turn any autoregressive or recurrent model into its Bayesian version. BARNN is based on the variational dropout method, allowing it to be applied to large autoregressive or recurrent neural networks as well. We also introduce a temporal version of the “Variational Mixtures of Posteriors” prior (tVAMP prior) to make Bayesian inference efficient and well-calibrated. We demonstrate the effectiveness of BARNN on various AI4Science tasks, including PDE surrogate modeling and molecular generation. Our results suggest that BARNN can play a key role in advancing AI-driven scientific discovery by providing models that are not only accurate but also aware of their limitations.