Cost Function and Performance Metrics in Deep Learning

Beginners can confuse cost functions with Performance metrics.

Some techniques are used When creating a deep learning model to help design the optimum neural networks architecture such as cost functions and performance/evaluation metrics. Many mistake those terms and use them interchangeably!

In this article, we will explain the difference between cost functions and Performance metrics, their common types, and when we use each one.

Cost functions in deep learning:

The cost function is a mechanism that calculates the error between the predicted value by the model and the actual value.

In deep learning, the cost function is the sum of errors (the loss function) in each iteration. We calculate the loss function of each example and sum the total error.

To give more context to cost functions, we need to talk about optimizers.

An optimizer function is used to find the minimum cost function value of the model to specify the optimum weight and bias of the neural network architecture.

the goal of cost functions

The cost function measures how good the neural network model predictions are while training (the learning process) and help us reach the optimal model with the optimal parameters.

Common cost functions:

Choosing a cost function for your deep learning model is related strongly to the type of activation function you used. Those two elements are connected.

Here are some of the most-used cost functions in each problem type :

Regression loss functions :

Mean Squared Error

Classification loss functions:

Cross-Entropy (Logarithmic loss)

Style Transfer cost function:

Other cost functions could be used in different neural network problems, such as style transfer where the cost function is calculated as the following:

While cost functions are used in the training process, the performance metrics are used after training to evaluate the model.

Performance metrics in deep learning

Performance metrics, Also called Evaluation metrics, are metrics that measure the quality of the machine learning and deep learning model.

The goal of the performance metrics:

They measure how well the model performs on unseen data to judge its overall performance before using it in production.

Choosing the metric is related to the type of problem discussed rather than the type of model (neural networks, SVM, linear regression…).

Common performance metrics:

common evaluation metrics in each problem type are:

Performance metrics for Regression:

Mean Squared Error

RMSE (Root Mean Square Error)

MAE

Adjusted R²

Performance metrics for classification:

Confusion Matrix and Classification Report( precision, recall, F1-score)

Log Loss

Area Under ROC Curve.

Summary:

In nutshell, cost functions are used on the training phase and performance metrics are used after training the model. Hence, we can find the same function used for calculating cost and also as performance metrics such as MSE.