In recent years, machine learning has gained significant importance due to the increasing amount of available data. The numerous different model architectures share a common characteristic - they have parameters or design decisions that are fixed before being trained on data. The right choice of the so-called hyperparameters can have a huge impact on the performance which is why they have to be optimized. Different techniques like grid search, random search, and Bayesian optimization have been developed to tackle this problem.

In this thesis, a new approach called adaptive sparse grid search for hyperparameter optimization is introduced. This technique allows to adapt to the hyperparameter space and the model which leads to less training and evaluation runs compared to normal grid search. Additionally, we present an adapted random search approach which is also adaptive to the data by iteratively adding sampled points. Three different refinement strategies for the concrete sampling are introduced and analyzed.
We compare the new approaches to the other three techniques mentioned regarding budget and resulting model performance using different machine learning tasks and datasets. The results show that adaptive sparse grid search for hyperparameter optimization is performing much better than normal grid search in high dimensions. The comparisons show that each algorithm performs well for specific optimization settings. The iterative sparse grid search is promising and still requires further analysis.