Feature Scaling: Why Feature Scaling Affects Model Training

Feature scaling is often overlooked because it seems like just another data preprocessing step. However, in practice, it often helps models train faster and more stably. Imagine one feature has values ranging from 0 to 1, while another has values ranging from 0 to 10,000. Although both features may be equally important for prediction, it's more difficult for the optimizer to work with such data.

This means it has to take more steps to find a good solution. Additionally, regularization becomes less effective because features with different scales require coefficients of different magnitudes. Let's look at how this looks in a simple example.

Install dependencies:

pip install numpy scikit-learn

Import libraries:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

Let's create a small synthetic dataset. It will have two features: the first has a normal scale, and the second is about a thousand times larger.

Importantly, both features actually influence the target variable. That is, the only difference between them is the scale.

np.random.seed(42)
x_small = np.random.normal(0, 1, 300)
x_large = np.random.normal(0, 1000, 300)

X = np.vstack([x_small, x_large]).T

y = (x_small + 0.001 * x_large > 0).astype(int)

Now, let's split the data into training and testing sets. We won't scale anything yet—first, let's see how the model behaves on the original data.

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.3,
    random_state=42,
    stratify=y
)

Let's train a logistic regression model without scaling.

In addition to the model's quality, let's also look at the number of iterations (n_iter_). This metric shows how much work the optimizer had to do to find the coefficients.

model = LogisticRegression()
model.fit(X_train, y_train)

pred = model.predict_proba(X_test)[:, 1]

print("ROC-AUC:", roc_auc_score(y_test, pred))
print("Iterations:", model.n_iter_)

Now, let's scale the features to the same scale using StandardScaler.

It calculates the mean and standard deviation only for the training set and then uses the same values for the test set. This is important because the model should not "peek" at the test data during training.

After this transformation, both features are approximately on the same scale, and it becomes easier for the optimizer to work with them.

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Now, let's retrain the model.

We're using the same model, the same data, and the same parameters. The only difference is that the features are now scaled.

model = LogisticRegression()
model.fit(X_train_scaled, y_train)

pred = model.predict_proba(X_test_scaled)[:, 1]

print("ROC-AUC (scaled):", roc_auc_score(y_test, pred))
print("Iterations (scaled):", model.n_iter_)

Most often, the ROC-AUC doesn't change much. However, the number of iterations becomes smaller. This means that the optimizer found a solution faster, and the training was more stable.

🔥 Feature scaling is a simple data preprocessing step that, in many cases, allows the model to train faster and more stably. For logistic regression, SVMs, neural networks, and other algorithms that use numerical optimization, it's best not to skip it.

✨ #DataScience #MachineLearning #Python #Coding #Tech #AI

✨ Join Best TG Channels https://t.iss.one/addlist/0f6vfFbEMdAwODBk

⭐️ Join Our WhatsApp Channel https://whatsapp.com/channel/0029VaC7Weq29753hpcggW2A