HomeArtificial IntelligenceLoss Features in TensorFlow

Loss Features in TensorFlow

Final Up to date on July 15, 2022

Loss metric is essential for neural networks. As all machine studying mannequin is a optimization drawback or one other, the loss is the target operate to reduce. In neural networks, the optimization is completed with gradient descent and backpropagation. However what are loss features and the way are they affecting our neural networks?

On this submit, we’ll cowl what loss features are and go into some generally used loss features and how one can apply them to your neural networks.

After studying this text, you’ll be taught:

  • What are loss features and the way they’re totally different from metrics
  • Widespread loss features for regression and classification issues
  • How you can use loss features in your TensorFlow mannequin

Let’s get began!

Loss features in TensorFlow.
Picture by Ian Taylor. Some rights reserved.


This text is break up into 5 part; they’re:

  • What are loss features?
  • Imply absolute error
  • Imply squared error
  • Categorical cross-entropy
  • Loss features in observe

What are loss features?

In neural networks, loss features assist optimize the efficiency of the mannequin. They’re normally used to measure some penalty that the mannequin is incurring on its predictions, such because the deviation of the prediction away from the bottom fact label. Loss features are normally differentiable throughout their area (however it’s allowed that the gradient is undefined just for very particular factors, reminiscent of x = 0, which is mainly ignored in observe). Within the coaching loop, we differentiate them with respect to parameters and used these gradients for our backpropagation and gradient descent steps to optimize our mannequin on the coaching set.

Loss features are additionally barely totally different from metrics. Whereas loss features can inform us the efficiency of our mannequin, they won’t be of direct curiosity or simply explainable by people. That is the place metrics are available. Metrics reminiscent of accuracy are way more helpful for people to grasp the efficiency of a neural community regardless that they won’t be good selections for loss features since they won’t be differentiable.

Within the following, let’s discover some frequent loss features, particularly, the imply absolute error, imply squared error, and categorical cross entropy.

Imply Absolute Error

The imply absolute error (MAE) measures absolutely the distinction between predicted values and the bottom fact labels and takes the imply of the distinction throughout all coaching examples. Mathematically, it is the same as $frac{1}{m}sum_{i=1}^mlverthat{y}_i–y_irvert$ the place $m$ is the variety of coaching examples, $y_i$ and $hat{y}_i$ are the bottom fact and predicted values respectively, and we’re averaging over all coaching examples. The MAE is rarely destructive, however it will be zero provided that the prediction matched the bottom fact completely. It’s an intuitive loss operate and may additionally be used as one in all our metrics, particularly for regression issues since we’d need to decrease the error in our predictions.

Let’s have a look at what the imply absolute error loss operate seems to be like graphically:

Imply absolute error loss operate, floor fact at x = 0 and x-axis characterize the expected worth

Just like activation features, we’re normally additionally excited about what the gradient of the loss operate seems to be like since we’re utilizing the gradient afterward to do backpropagation to coach our mannequin’s parameters.

Imply absolute error loss operate (blue) and gradient (orange)

We discover that there’s a discontinuity within the gradient operate for the imply absolute loss operate however we are likely to ignore it because it happens solely at x = 0 which in observe hardly ever occurs since it’s the likelihood of a single level in a steady distribution.

Let’s check out the right way to implement this loss operate in TensorFlow utilizing the the Keras losses module:

which supplies us 2.0 because the output as anticipated, since $ frac{1}{2}(lvert 2-1rvert + lvert 3-0rvert) = frac{1}{2}(4) = 4 $. Subsequent, let’s discover one other loss operate for regression fashions with barely totally different properties, the imply squared error.

Imply Squared Error

One other in style loss operate for regression fashions is the imply squared error (MSE), which is the same as $frac{1}{m}sum_{i=1}^m(hat{y}_i–y_i)^2$. It’s much like the imply absolute error because it additionally measures the deviation of the expected worth from the bottom fact worth. Nevertheless, the imply squared error squares this distinction (all the time non-negative since sq. of actual numbers are all the time non-negative), which supplies it barely totally different properties.

One notable one is that the imply squared error favors a lot of small errors over a small variety of giant errors, which results in fashions which have much less outliers or no less than outliers which might be much less extreme than fashions skilled with a imply absolute error. It is because a big error would have a considerably bigger impression on the error, and consequently the gradient of the error, when in comparison with a small error.


Imply squared error loss operate, floor fact at x = 0 and x-axis characterize the expected worth

Then, wanting on the gradient,

Imply squared error loss operate (blue) and gradient (orange)

Discover that bigger errors would result in a bigger magnitude for the gradient and in addition a bigger loss. Therefore, for instance, two coaching examples that deviate from their floor truths by 1 unit would result in a lack of 2, whereas a single coaching instance that deviates from its floor fact by 2 items would result in a lack of 4, therefore having a bigger impression.

Let’s have a look at the right way to implement the imply squared loss in TensorFlow.

which supplies the output 5.0 as anticipated since $frac{1}{2}[(2-1)^2 + (3-0)^2] = frac{1}{2}(10) = 5$. Discover that the second instance with a predicted worth of three and precise worth of 0 contributes 90% of the error beneath the imply squared error vs 75% of the error beneath imply absolute error.

Generally, you might even see individuals use root imply squared error (RMSE) as a metric. That is to take the sq. root of MSE. From the angle of a loss operate, MSE and RMSE are equal.

Each MAE and MSE are measuring values in a steady vary. Therefore they’re for regression issues. For classification issues, we are able to use categorical cross-entropy.

Categorical Cross-entropy

The earlier two loss features are for regression fashions, the place the output could possibly be any actual quantity. Nevertheless, for classification issues, there’s a small, discrete set of numbers that the output might take. Moreover, the quantity that we use to label-encode the courses are arbitrary, and with no semantic that means (e.g. if we used the labels 0 for cat, 1 for canine, and a pair of for horse, it doesn’t characterize {that a} canine is half cat and half horse). Subsequently it mustn’t have an effect on the efficiency of the mannequin.

In a classification drawback, the mannequin’s output is a vector of likelihood for every class. In Keras fashions, normally we anticipate this vector to be “logits”, i.e., actual numbers to be reworked to likelihood utilizing softmax operate, or the output of a softmax activation operate.

The cross-entropy between two likelihood distributions is a measure of the distinction between the 2 likelihood distributions. Exactly, it’s $-sum_i P(X = x_i) log Q(X = x_i)$ for likelihood $P$ and $Q$. In machine studying, we normally have the likelihood $P$ offered by the coaching knowledge and $Q$ predicted by the mannequin, which $P$ is 1 for the proper class and 0 for each different class. The expected likelihood $Q$, nonetheless, is normally valued between 0 and 1. Therefore when used for classification issues in machine studying, this components may be simplified into: $$textual content{categorical cross entropy} = – log p_{gt}$$ the place $p_{gt}$ is the model-predicted likelihood of the bottom fact class for that specific pattern.

Cross-entropy metric have a destructive signal as a result of $log(x)$ tends to destructive infinity as $x$ tends to zero. We wish the next loss when the likelihood approaches 0 and a decrease loss when the likelihood approaches 1. Graphically,

Categorical cross entropy loss operate, the place x is the expected likelihood of the bottom fact class

Discover that the loss is precisely 0 if the likelihood of the bottom fact class is 1 as desired. Additionally, because the likelihood of the bottom fact class tends to 0, the loss tends to constructive infinity as properly, therefore considerably penalizing dangerous predictions. You would possibly acknowledge this loss operate for logistic regression, and they’re comparable besides the logistic regression loss is restricted to the case of binary courses.

Categorical cross entropy loss operate (blue) and gradient (orange)

Now, wanting on the gradient of the cross entropy loss,

Wanting on the gradient, we are able to see that the gradient is mostly destructive which can be anticipated since to lower this loss, we’d need the likelihood on the bottom fact class to be as excessive as potential, and recall that gradient descent goes in the other way of the gradient.

There are two alternative ways to implement categorical cross entropy in TensorFlow. The primary technique takes in one-hot vectors as enter,

This offers the output as 0.2876821 which is the same as $-log(0.75)$ as anticipated. The opposite manner of implementing the explicit cross entropy loss in TensorFlow is utilizing a label-encoded illustration for the category, the place the category is represented by a single non-negative integer indicating the bottom fact class as an alternative.

which likewise offers the output 0.2876821.

Now that we’ve explored loss features for each regression and classification fashions, let’s check out how we use loss features in our machine studying fashions.

Loss Features in Observe

Let’s discover how we are able to use loss features in observe. We’ll discover this by a easy dense mannequin on the MNIST digit classification dataset.

First, we get the obtain the info from Keras datasets module,

Then, we construct our mannequin,

And we have a look at the mannequin structure outputted from the above code,

We will then compile our mannequin, which can be the place we introduce the loss operate. Since this can be a classification drawback, we’ll use the cross entropy loss. Particularly, because the MNIST dataset in Keras datasets is represented as a label as an alternative of an one-hot vector, we’ll use the SparseCategoricalCrossEntropy loss.

And at last, we prepare our mannequin.

And our mannequin efficiently trains with the next output:

And that’s one instance of the right way to use a loss operate in a TensorFlow mannequin.

Additional Studying

Under are the documentation of the loss features from TensorFlow/Keras:


On this submit, you could have seen loss features and the function that they play in a neural community. You will have additionally seen some in style loss features utilized in regression and classification fashions, in addition to the right way to use the cross entropy loss operate in a TensorFlow mannequin.

Particularly, you realized:

  • What are loss features and the way they’re totally different from metrics
  • Widespread loss features for regression and classification issues
  • How you can use loss features in your TensorFlow mannequin

Develop Deep Studying Initiatives with Python!

Deep Learning with Python

 What If You May Develop A Community in Minutes

…with only a few strains of Python

Uncover how in my new E book:

Deep Studying With Python

It covers end-to-end tasks on matters like:

Multilayer PerceptronsConvolutional Nets and Recurrent Neural Nets, and extra…

Lastly Carry Deep Studying To

Your Personal Initiatives

Skip the Teachers. Simply Outcomes.

See What’s Inside



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments