Hyperparameter optimization is a common problem in machine learning. Machine learning algorithms, from logistic regression to neural nets, depend on well tuned hyperparameters to reach maximum effectiveness. Different hyperparameter optimization strategies have varied performance and cost (in time, money, and compute cycles.) So how do you choose? This talk will give a brief introduction to hyperparameter tuning and its importance, as well as the basics of how we apply statistical tests to make confident assertions about which hyperparameter optimization strategies can give you better results, faster.