Methods for Statistical Evaluation of AI
PhD School at the Faculty of SCIENCE at University of Copenhagen
The aim of course is to equip the participants with statistical methods, and knowledge of the newest research within statistical methods, for evaluation of machine learning and AI.
The course runs as a summer school, retreat for five full days. As preparation, the students prepare a poster framing their own research in relation to the theme of the summer school. This includes reading selected papers as preparation before the summer school. The posters will be presented by the students at a poster session the first evening of the summer school. The posters are used for group work and further refinement through an assignment, which finalizes the course.
The five course days exists of lectures, including smaller assignments and discussion to activate the students during lectures. The lecturers consist of both external lecturers as well as lecturers from Danish Universities. Group work is also carried out during the five days of summer school. An individual assignment is handed in one week after the summer school.
Formal requirements
The students need qualifications in statistical methods and machine learning or in one of these areas in depth.
Learning outcome
Knowledge:
• Understand a variety of statistical methods, including PAC bounds, measures for evaluation of AI, and fairness metrics, fairness calibration and their short comings.
Skills:
• Choose between a selection of statistical methods for evaluation of ML and AI
• Assess fairness of ML models
• Calibrate for better fairness, but also understand when calibration can skew fairness even further
• Apply different sampling strategies for building AI systems, depending on the requirements of the application.
Competences:
• Develop the competence to critically assess AI models, identifying strengths, weaknesses, and potential biases in model behaviour.
• Ability to incorporate fairness considerations withing the design and development of ML, ensuring that ethical principles guide the creation and deployment of AI systems.
• Build ability to engage in peer review, provide constructive feedback, and contribute to the collective advancement of knowledge in statistical AI evaluation.
• Promote abilities to identify the gaps and open research challenges in statistical evaluation of AI, including developing methods for necessary and sufficient methods for evaluation of AI models.
Teaching and learning methods
Lectures, group work, and individual assignments.
Lecturers
External guest lecturers:
• Gaël Varoquaux, INRIA, INRIA Saclay-Ile-de-France Research Centre, France :
Course (tentative title): Evaluation of Machine Learning models on typical tabular data.
Short bio: Research direction of the Soda team in INRIA, he is specialized in applications of Machine Learning to health. Prof. Varoquaux is a co-founder of the Scikit-learn library that is a widely used Machine Learning library in Python.
Website: Gaël Varoquaux | Inria
• Benjamin Guedj, Department of Computer Science, University College London, United Kingdom:
Course (tentative title): PAC-Bayes for Interactive Learning.
Short bio: Dr. Benjamin Guedj is specialized in machine learning. He is a Research Fellow at University College London (UCL) in the Department of Computer Science and a research scientist at Inria, France. His research focuses on theoretical machine learning, including statistical learning theory, PAC-Bayes, and generalization for deep learning.
Website: Home - Benjamin Guedj (bguedj.github.io)
• Jose Hernandez-Orallo, Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Spain:
Course (tentative title): Evaluation of cognitive capabilities of AI systems.
Short bio: The main research of Prof. Hernández-Orallo focuses on AI evaluation, benchmarking for measuring AI capabilities, and the safety of AI systems.
Website: Jose Hernandez-Orallo's Home Page (upv.es)
Guest lecturers from Danish institutions:
- Prof. Aasa Feragen, Image Analysis and Computer Graphics, DTU-Compute, Technical University of Denmark.
Course (tentative title): Geometric Statistics in Image Analysis.
- Prof. Christian Igel, Department of Computer Science, University of Copenhagen, Denmark.
Course (tentative title): PAC-Bayesian Analysis of Ensemble Machin Learning Models.
- Prof. Line Clemmensen, Dept. of Mathematical Sciences, University of Copenhagen and DTU-Compute, Technical University of Denmark.
Course (tentative title): Data representativity for Machine Learning and AI systems.
- Prof. Murat Külahci, Statistics and Data Analysis, DTU-Compute, Technical University of Denmark.
Course (tentative title): Active learning in time-series and sequential data.
- Assoc. Prof. Andres Madragosa. Department of Computer Science at Aalborg University, Technical University of Denmark.
Course (tentative title): Generalization of Deep Neural Networks.
Remarks
The participant fee is 5,600 DKK and covers five days incl overnight stays, food, social event, and all course content.
How to apply:
PhD students enrolled at University of Copenhagen (UCPH)
1) Please provide us with "stedkode" and "alias" when you register by choosing "Apply" at the upper right corner of the course description, so that UCPH Accounting Department can make an internal transfer regarding payment of the Participation fee.
If you have been accepted for the course, we will verify your enrollment via mail.
PhD students enrolled at other universities than UCPH
1) Choose “Apply” in the upper right corner of the course description.
If you have been accepted for the course, we will send you a mail with a link to payment of the participant fee.
NB: Payment via creditcard or Mobilepay only. After you have made your payment of the participation fee, we will verify your enrollment via mail.