Descriptif

Objectives

The objective of this course is to introduce the theory of linear and non-linear regression. Regression is a simple yet versatile model used in many problems both in applications and in fundamental research. It is therefore absolutely essential for a data scientist to understand the theory of regression analysis. As such, this course will be theory oriented. Practical aspects will also be treated in tutorials. 

The first part of the course will be devoted to the theory of linear models. The theoretical analysis of the statistical properties of this model relies on an elegant combination of basic Linear Algebra and Probability theory. The main goal of this part will be to develop an in-depth understanding of statistical inference in the linear model relying only on the geometric structure of the model.

In the second part of the course, we will explore important topics about regression: Model selection, penalized empirical risk regularization, high-dimensional statistics and non-parametric statistics. Time permitting, we will also investigate generalized linear models such as logistic regression.

Syllabus

Part I: Linear Models

1. Gaussian Vectors
• Random vectors
• Gaussian Vectors
• Cochran Theorem
2. Gauss-Markov Estimation
• Linear functionals
• Gauss-Markov optimality 
• Estimation of mean and variance
3. Normal Estimation
• MLE
• Optimality of MLE
• James-Stein Phenomenon
4. Statistical Testing and Confidence Intervals
• LRT and Fisher Test
• CI for linear functionals of mean
5. Anova/Ancova

Part II: Advanced Topics

1. Model Selection
• Penalized Empirical Risk minimization : AIC, BIC
• Case of orthogonal design
• Practical model selection
2. High-dimensional Statistics
• Sparsity oracle inequalities
• LASSO Estimator
• Variable Selection
3. Nonparametric regression
• Kernel Estimator
• Pointwise error of estimation
• Minimax rate of estimation
• Tuning the bandwidth parameter
4. Logistic Regression

• Course evaluation : Final Exam
• Course Language: Either in English or in French with lecture material  in English