Question

What is Regularization and what kind of problems does regularization solve?

Answer 1

It is the process of adding a tuning parameter to a model so as to induce smoothness in order to prevent overfitting or to solve an ill-posed problem.

It works by adding a constant to multiple to an existing weight vector, with the constant being often either the L1 (Lasso) or L2 (ridge), or any norm. This results in the model predictions minimizing (shrinks) the mean of the loss function calculated on the regularized training set.