Statistical Machine Learning

Moodle: Most communication for this class will take place on Moodle. Please sign up here .
Lecture: Wednesdays, 13:30 - 16:00 o'clock, ZOOM for now, S103/226 when again possible
Exercises: Wednesdays, 16:15 - 17:00 o'clock, ZOOM for now, S103/226 when again possible
Language: English

Lectures and exercises take place via Zoom. Please visit the Moodle page to find the link.

Motivation Machine learning is the science of getting computers to act without being explicitly programmed. In the past decade, machine learning has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome. Machine learning is so pervasive today that you probably use it dozens of times a day without knowing it. Many researchers also think it is the best way to make progress towards human-level AI. In this class, you will learn about statistical machine learning techniques, and gain some practice implementing them and getting them to work for yourself.


This course consists of a mix of online lectures and flipped classroom sessions. That is, aside from our online lectures, you may also want to watch the aditional videos provided, even before the lecture. Please prepare a list of 5 questions as extra homework for those videos, since we (all of you) may also try to answer your questions in class. We will also present our own work. We may even ask you to read papers, let's see. Aside from the “5-questions” homework, there are also four written assignments involving both theory and programming. Handing in those assignments regularly will earn you a bonus for the final exam.

This is the syllabus for the Summer 2020 iteration of the course.

Lecture # Date Description Our Slides Videos
1 Introduction April 22, 2020 Course Introduction, Basic ML Intoduction .pdf Intro, T. Mitchell, CMU
Intro, A. Ihler, UCI
2 Linear Algebra April 29, 2020 Vector, matrix, (Pseudo) Inverse, Eigenvectors, ... .pdf Matrix Algebra for Engineers, J. Chasnov, Udacity/HKUST
Linear Algera Basics, C. Pryby, Udacity
Eigenvalues and Eigenvectors, MIT 18.06C Linear Algebra
3 Statistics May 6, 2020 Random Variables, Distributions, Moments, ... .pdf Probabilitiy distributions and their Properties, B. Carlson
Random Var. & prob. dist., M. Rooduijn, E. van Loon, UoA
Random Variables, Expection, Covariance, ..., IIT, India
4 Optimization May 13, 2020 Convex, (Un)constraint Optimization, Lagragian Multiplier, Duality, Gradient descent, Newton, CG, ... .pdf Convex Probelems, T. Balch, A. Chakraborty, Georgia Tech
Convexity, S. Dasgupta, UCSD
Convexity and Optimization, A. Smola, CMU
Duality, S. Dasgupta, UCSD
Lagrangian Multipliers, G.J. Gordon, CMU
5 Bayesian Decision Theory May 20, 2020 Class Conditional Probs, Class Prios, Bayesian Probs, Decision Boundary, Risk Minimization , ... .pdf Empirical and Expected Risk, Smola, UC Berkeley
Risk and Loss Functions, Dini, Pravahan, AT&T
Risk Minimization, Andy Park, Purdue
6 Probabilisty Density Estimation May 27, 2020 Maximum Likelihood Estimation Bayesian Estimation, Nonparametric Density, Curse of Dimensionality, Expectation Maximization, ... .pdf Estimating Probabilities from Data, Weinberger, Cornell
MLE Gaussian, Littman, Isbell, Kolhe, GeorgiaTech
Curse of Dimesionality, Littman, Isbell, Kolhe, GeorgiaTech
Mixture of Gaussians, Fox, U. Washington
Expectation Maximization, Ng, Stanford
kNN and Parzen Window, Lavrenko, U. Edingburgh
Hyper Cube, Dini, Pravahan, AT&T
7a Expectation Maximization June 10, 2020 Extra Slides on Expectation Maximization (EM), ... .pdf k-Means Clustering, Ihler, UCI
Gaussian Mixtures, EM, Ihler, UCI
Gaussian Mixtures, EM, Lavrenko, U. Edinburgh
7b Clustering and Evaluation June 10, 2020 K-Means, Agglomerative Clustering, Mean Shift, Bias and Variance, ... .pdf Mean Shift, Bobick, Essa, Chakraborty, GeorgiaTech
Overfitting, Ihler, UCI
Bias and Variance , Ihler, UCI
Bias and Variance, Ng,
Cross-Validation, Thrun, Udacity
8 Regression June 17, 2020 Linear Regression, ML Linear Regression, Bayesian Linear Regression ... .pdf Lineare Regression, Ihler, UCI
ML and Linear Regression, de Freitas, UBC
Regularization and Regression, de Freitas, UBC
Bayesian Regression, Lawrence, Sheffield U.
9 Classification June 24, 2020 Discriminant, Fisher's Linear Discriminant, Perceptron, Logistic Regression ... .pdf Lineare Classifiers: Basics, Ihler, UCI
Lineare Classifiers: Learning Parameters, Ihler, UCI
Intro to Linear Classificaiton, Jensen, TUM
Linear Discriminant Models, Jensen, TUM
Linear Discriminant Analyis, Starmer, StatQuest
Linear Discriminant Analysis, Lavrenko, U. Edinburgh
10 Dim. Reduction & ERM July 1, 2020 Projection, PCA, Empirical Risk Minimization (ERM), CV Dimension, ... .pdf PCA and SVD, Ihler, UCI
VC Dimension, Ihler, UCI
Generalization Bound, VC Dimension, Huang, Virginia Tech
Approx./Estimation & ERM, Avati, Stanford
11 SVM July 8, 2020 Hyperplane, Max Margin, Support Vector Machines, Kernels, Slack, ... .pdf Support Vector Classification, Smola, CMU
SVM - Linear SVM/primal form, Ihler, UCI
SVM - Lagrangian and Dual, Ihler, UCI
SVM - Kernels, Ihler, UCI
12 Neural Networks July 8, 2020 Activation Functions, Neural Networks (NN), Stochastic Gradient, Backpropagation, Convolutional NN, Drop Out, Adam ... .pdf Neural Networks, Ihler, UCI
Backpropagation, Ihler, UCI
Backpropagation, Lavrenko, U. Edinburgh
Convolutional NNs, Soleimany, MIT
Backpropagation, Yeung, Stanford
Convolutional NNs, Yeung, Stanford
13 Gaussian Processes July 15, 2020 Kernels Revisited, Dual of RBF Networks, Gaussian Processes, Bayesian learning ... .pdf Gaussian Process Regression, Smola, CMU
Introduction to Gaussian Processes, Lawrence, U. Sheffield
Gaussian Processes for Machine Learning, Cunningham, Cambridge
Gaussian Prcoesses, Poupart, U. Waterloo
Introduction to Gaussian Processes, de Freitas, UBC

Assignments and Exercises

Aside from the “5-questions” homework, there are also four written assignments involving both theory and programming. Handing in those assignments regularly will earn you a bonus for the final exam. Pleses visit Moodle for further information, and for submitting them.

Further Material and Links