Model Evaluation

Confusion Matrix

Actual\Predicted	$C$	$\neg C$
$C$	True Positives (TP)	False Negatives (FN)
$\neg C$	False Positives (FP)	True Negatives (TN)

• Accuracy = (TP + TN) / All
• Error Rate = (FP + FN) / All
• Sensitivity = TP / P
• Specificity = TN / N
• Precision = TP / (TP + FP)
• Recall = TP / (TP + FN) = TP / P
• F Measure / F Score = harmonic mean of precision and recall

$$F=\frac{({\beta }^2+1)PR}{{\beta }^{2}P+R}$$

• F1-measure = F score with $\beta =1$

$$F=\frac{2PR}{P+R}$$

ROC Curve

• ROC = Receiver Operating Characteristics
  • See how a classifier performs with different threshold
  • Visualizes tradeoff between precision and recall
• Procedure
  • Rank the test tuples with likelihood to be true in decreasing order
  • Horizontal axis as False Positive Rate, vertical as True
• Interpretation
  • The area under ROC curve measures the accuracy of the model
• Similarly, we have precision-recall curve

MAE and RMSE

• Mean Absolute Error (MAE) = ${\sum }_i|s_i-c_i|/n$
• Root Mean Squared Error (RMSE) = $\sqrt{{\sum }_i(s_i-c_i{)}^2/n}$

Kendall’s Tau

• tau = (# concordant pairs - # discordant pairs) / number of pairs
• Concordant pair means a positive tuple appears before a negative one in terms of prediction score ranking
• Total number of pairs is $\left(\genfrac{}{}{0ex}{}{n}{2}\right)$