#1 Spot for Defeating Online Exams

what-function-defines-accuracy-in-keras-when-the-loss-is-mean-squared-error-(mse)
[bsa_pro_ad_space id=1]

During the development of machine learning models in Keras, Mean Squared Error (MSE) is frequently employed as the loss function for regression tasks. In contrast to classification, regression concerns continuous values, rendering standard accuracy measurements inapplicable. Since MSE concentrates on reducing errors instead of predicting distinct labels, Keras does not furnish accuracy metrics for regression endeavors. Rather, alternatives such as Mean Absolute Error (MAE), R^2 Score, and Mean Absolute Percentage Error (MAPE) are utilized. Nevertheless, a personalized accuracy function can be formulated, which will aid in assessing predictions within an acceptable error margin. This offers an accuracy-like evaluation for regression models.

This article will delve into Mean Squared Error, the formulation of an accuracy function for regression, and the way Keras defines accuracy when utilizing MSE as the loss function. So let’s begin!

Table of Contents

What is Mean Squared Error?

Mean Squared Error (MSE) is generally applied to regression assignments. It measures the disparity between predicted values and actual values.

The formula for MSE (Mean Squared Error) is provided below:

What is Mean Squared Error Output

Where,

  • y true represents the actual values.
  • y pred indicates the predicted values.
  • n signifies the total number of samples.

A low MSE indicates that the model’s predictions are closer to the true values.

Does Accuracy Function with MSE?

In classification scenarios, accuracy is a clear-cut concept. It measures the number of predictions that correspond with the actual labels. Accuracy can be articulated as:

Accuracy = Correct Predictions / Total Predictions

The notions of “correct” or “incorrect” predictions do not apply in regression tasks, where the aim is to forecast continuous values. Instead of binary correctness, predictions are assessed by their proximity to actual values. Therefore, accuracy is not an appropriate metric for regression tasks.

MSE (Mean Squared Error) quantifies the average squared differences between the predicted and actual values. As a result, it does not concentrate on tallying correct predictions; rather, it is focused on minimizing the overall error. Owing to this distinction, accuracy is not preset as a metric by Keras when MSE is employed as the loss function. Instead, alternative evaluation metrics like Mean Absolute Error (MAE), R^2 Score, and Mean Absolute Percentage Error (MAPE) are utilized to evaluate model performance in regression contexts.

Creating an Accuracy Function for Regression

Keras does not include a built-in metric for accuracy with MSE. Hence, you can establish your personalized accuracy function to assess accuracy while applying Mean Squared Error (MSE) as the loss function. Unlike classification, where accuracy is unambiguous, regression accuracy assesses how closely the predicted values align with the actual values.

1. Employing Mean Absolute Percentage Error (MAPE)

Accuracy in regression can be gauged using Mean Absolute Percentage Error (MAPE). It calculates the average percentage error between actual and predicted values. The formula for MAPE is presented below:

MAPE Image

MAPE articulates the error as a percentage, making it a beneficial measure.metric for assessing the correspondence of the model’s forecasts with the actual values. A lower MAPE indicates enhanced model performance.

2. Creating a Custom Accuracy Function in Keras

Alternatively, one can establish a custom accuracy function that treats a prediction as “correct” even when it lies within a designated tolerance level of the actual value. For instance, you may consider the prediction valid if the anticipated data is within a 10% deviation from the actual value.

This method aids in converting the regression output of the model into a metric resembling accuracy. This facilitates the interpretation of the model’s efficacy in contexts where absolute precision is not feasible. By incorporating these functions in Keras, you can more effectively monitor the advancements of the model for regression applications.

Execution of Custom Accuracy in Keras

We will proceed to train a fundamental Keras Regression Model and subsequently define a custom accuracy function. Below are the outlined steps for its execution.

Step 1: Importing the Required Libraries

Example:

Python

Code Copied!

Clarification:

The preceding section solely imports the requisite libraries such as NumPy, TensorFlow/Keras, and Scikit-Learn needed to build and train the neural network. This snippet will not produce any output since it simply contains the import statements and does not define or execute any functions, models, or data manipulation procedures.

Step 2: Creating a Sample Dataset

In this step, we will construct a straightforward dataset where y = 3x + 5.

Example:

Python

Code Copied!

Clarification:

The code above is designed for generating artificial data. It employs a linear equation with added noise, subsequently dividing the data into training and testing sets following the 80:20 distribution. This code does not produce any output since it solely splits the data without inclusive print or visualization commands to exhibit the generated values.

Step 3: Crafting a Custom Accuracy Function

At this point, we will outline an accuracy function to ascertain the count of predictions that fall within 10% of the actual value.

Illustration:

Python

Code Copied!

Clarification:

In the preceding code, the function custom_accuracy serves to determine the ratio of predictions that fall within 10% of the true values. This is achieved through the assessment of the relative error and the calculation of the mean for the correct predictions. The code does not produce any output as it merely defines a function without executing any data inputs.

Step 4: Developing the Regression Model

At this stage, we will assemble a basic neural network utilizing Mean Squared Error along with our bespoke loss accuracy function.

Illustration:

“““html
class=”main-editor-wrapper”>

Code Duplicated!

Clarification:

This code snippet is utilized to define a neural network featuring two concealed layers. Subsequently, it compiles it using the Adam Optimizer alongside Mean Squared Error (MSE) loss function. It trains the model on the dataset while monitoring a custom accuracy measure. The code does not produce an output as the model's training function ( model.fit ) operates in silence ( verbose = 0 ). Consequently, no logs or progress notifications are displayed.

Step 5: Model Assessment

In this phase, we will assess the model's performance.

Sample:

Python

Code Duplicated!

Output:

Model Evaluation Output

Analysis:

The output above indicates that the trained model has an MSE loss of 4.3449 on the test dataset, accompanied by a custom accuracy metric of 0.6641. This signifies that 66.41% of the predictions were within the established accuracy range.

Alternative Metrics for Assessing Regression Accuracy

For different methods of evaluating accuracy in regression, consider the following alternatives:

1. R^2 Score (Coefficient of Determination)

This metric is utilized to assess how closely the predicted values align with the actual outcomes. A significant discrepancy indicates better performance.

Example:

Python

Code Copied!

Output:

R^2 Score Output

Analysis:

The above illustrates the calculation of the R^2 (coefficient of determination) score. This metric evaluates the model's performance by correlating actual and predicted values, subsequently outputting the results rounded to four decimal places.

2. Mean Absolute Error (MAE)

This metric quantifies the average absolute disparity between the actual and estimated values.

Example:

Python

```

Result:

Mean Absolute Error Output

Clarification:

The code above computes the Mean Absolute Error (MAE) between the actual and predicted values. This quantifies the average prediction deviation of the model. The result is subsequently displayed, rounded to four decimal digits.

Summary

When utilizing Mean Squared Error (MSE) as a loss function in Keras, conventional accuracy metrics are inapplicable. This is due to the nature of regression tasks, which lack a definitive “correct” or “incorrect” prediction framework. Instead, one may create tailored accuracy metrics such as Mean Absolute Percentage Error (MAPE) or an accuracy measure based on tolerances, which will provide meaningful evaluations of the model. Additionally, metrics such as R^2 score and Mean Absolute Error (MAE) contribute valuable insights into the model's predictive effectiveness. By selecting appropriate assessment methods, one can enhance the evaluation and refinement of accuracy for the regression model in Keras.

Common Queries

The article What Function Defines Accuracy in Keras When the Loss is Mean Squared Error (MSE) first appeared in Intellipaat Blog.

Staff SEO Expert