Course completion
  • An optimization problem consists in finding parameters which minimize or maximize an objective function.
  • In the general case, a function can have several local optima or plateaus.
  • High dimensional spaces behave in non intuitive ways which can dramatically affect optimization performance.
  • Any local optimum of a convex function is in fact a global optimum.
  • Local optima can sometimes be found analytically using properties of the gradient and the Hessian.
  • Gradient descent is an iterative optimization method which follows the direction of steepest descent at each step and can benefit from high dimensional spaces.
  • In black-box optimization the landscape of the objective function cannot be studied analytically and can only be discovered through evaluation.
  • Evolutionary algorithms are optimization methods particularly adapted to the black box scenario, which involve moving a population of candidate solutions towards better fitness.
  • Evolutionary algorithms allow the practitioner to choose a useful implicit metric with the mutation and cross-over operators.
  • EDAs are a mathematically principled form of evolutionary algorithm which move towards better solutions in parameter space by updating a proposal distribution.

Next: From optimization to machine learning