Back to Teaching
ISE 633 · 4 Units

Large-Scale Optimization for Machine Learning

Theory and tools for large-scale optimization arising in modern data science and machine learning applications.

Course Description

The course covers the theory and tools for large-scale optimization that arise in modern data science and machine learning applications. Topics include stochastic optimization, accelerated methods, parallelization, nonsmooth optimization, online optimization, variance reduction, differential privacy in optimization, and min-max games.

Topics Covered

  • Background & Preliminaries
    • Optimization overview and examples in machine learning
    • Linear algebra and probability basics
  • Unconstrained Optimization
    • Optimality conditions; convex vs. non-convex problems
    • First-order methods and convergence analysis
    • Momentum and accelerated methods
  • Constrained Optimization
    • Fairness and safety in ML
    • KKT conditions and Lagrange multipliers
    • Projection-based algorithms
  • Nonsmooth Optimization
    • Lasso, nuclear norm regularizers, nonsmooth activations
    • Block coordinate descent, ADMM, proximal gradient
  • Empirical Risk Minimization & Stochastic Optimization
    • SGD, stochastic variance reduced gradient descent
    • Parallel optimization: synchronous vs. asynchronous
  • Online Convex Optimization
    • Regret analysis; follow-the-leader and variants
  • Learning via Min-Max Optimization
    • GANs, adversarial learning, fair learning
    • Gradient descent-ascent and proximal-type methods
  • Second-Order Methods
    • Hessian-vector products, conjugate gradient methods
  • Differential Privacy & Optimization
    • Privacy definitions; lower and upper bounds on differentially private convex optimization

Textbooks

There is no required textbook. The following are recommended references:

  • S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. (Free online)
  • S. Shalev-Shwartz and S. Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, 2014. (Free online)
  • A. Shapiro, D. Darinka, and A. Ruszczynski, Lectures on Stochastic Programming, SIAM, 2009. (Free online)
  • C. Dwork and A. Roth, The Algorithmic Foundations of Differential Privacy, 2014. (Free online)

Assignments & Policies

  • 5–6 homework assignments submitted via Blackboard (PDF only); two lowest scores dropped.
  • Each student must scribe lecture notes for one lecture (deadline: one week after the lecture).
  • Late submissions are not accepted under any circumstances.
  • Discussion with peers is encouraged; each student must submit their own work.
  • Questions should be posted to the course Slack channel; grade-related queries should be emailed to the TA.