Skip to content

06 Validation Methods and Performance Metrics

Preparation

Chapter 2, pages 67 - 97.

Chapter 3, pages 103 - 119.

Material

Performance Metrics (ipynb)

Validation Methods (ipynb)

Validation Methods and Performance Metrics (pptx)

Session Description

This lecture will cover several machine learning methodologies. In addition to this, we will discuss what is meant by true/false positives/negatives, and explore alternative performance metrics (so far, we have only encountered the “accuracy”-metric)

Key Concepts

  • The train/test-methodelogy
  • The validation set methodology
  • The cross-validation methodology

  • The leave-one-out-methodology

  • Accuracy

  • Confusion matrix
  • Recall
  • Precision
  • F1-score
  • Precision-recall-curve

Learning Objectives

  • Describe the "validation set"-methodology
  • Describe the "cross validation"-methodology
  • Describe the "leave one out"-methodology
  • Apply each of the 3 methodologies above in sklearn
  • Do hyperparameter tuning in sklearn using each of the 3 methodologies above (e.g. using the GridSearchCV-function in sklearn)
  • Explain and calculate (in python) the following performance metrics for supervised classification:
  • Confusion matrix
  • Accuracy
  • Recall ( = "True Positive Rate" (TPR) )
  • Precision ( = "Positive Prediction Rate" (PPR) )
  • F1-score
  • Precision-recall-curve