- An optimization problem consists in finding parameters which minimize or maximize an objective function.
- In the general case, a function can have several local optima or plateaus.
- High dimensional spaces behave in non intuitive ways which can dramatically affect optimization performance.
- Any local optimum of a convex function is in fact a global optimum.
- Local optima can sometimes be found analytically using properties of the gradient and the Hessian.
- Gradient descent is an iterative optimization method which follows the direction of steepest descent at each step and can benefit from high dimensional spaces.
- In black-box optimization the landscape of the objective function cannot be studied analytically and can only be discovered through evaluation.
- Evolutionary algorithms are optimization methods particularly adapted to the black box scenario, which involve moving a population of candidate solutions towards better fitness.
- Evolutionary algorithms allow the practitioner to choose a useful implicit metric with the mutation and cross-over operators.
- EDAs are a mathematically principled form of evolutionary algorithm which move towards better solutions in parameter space by updating a proposal distribution.