Deep Learning models are commonly used for a huge variety of applications, notably in safety-critical fields such as autonomous driving and medical image processing, where reliable uncertainty estimations are crucial. In contrast to frequentist approaches to deep learning, Bayesian Neural Networks provide a framework to assess uncertainty in the training data. However, since BNNs incorporate a multiple of the weights of a frequentist Neural Network with comparable architecture, there is a high demand for efficient optimizers. Under certain conditions, second order optimizers offer a faster convergence rate than gradient descent and are able to profit from parallelized computing architectures. In this work, a Quasi-Newton method is used to train several types of Bayesian Neural Networks and its performance is compared to the most commonly used first order optimizers.