Three Types of Risk in Model Evaluation

The workshop was a one-day event held in conjunction with the 2023 ADSA Annual Meeting in Ottawa, Canada. It was open to all ADSA Annual Meeting participants, including graduate students, faculty members, and industry professionals. The candidate led a two-hour session on "Three Types of Risk in Model Evaluation." This session employed multiple simulation examples to elucidate common pitfalls in model evaluation, covering topics such as the rationale for partitioning data into multiple subsets, the model selection process during hyperparameter tuning, and the optimistic bias that can arise when splitting data without considering experimental design. Additionally, the session explored potential solutions to mitigate these risks in model evaluation. Python code was provided during the session to demonstrate key concepts, enabling attendees to replicate the results.