Linear & Logistic Regression

Introduction

At the heart of predictive modeling lie two of the most fundamental and widely used statistical algorithms: Linear Regression and Logistic Regression. Despite the word "regression" in both names, they serve two distinct purposes in supervised machine learning. Linear regression is used to predict continuous numbers (like predicting house prices based on square footage), while logistic regression is used for classification (like predicting whether an email is spam or not spam).

Why do these classic algorithms matter in the age of massive Deep Learning models? Because they are the foundational building blocks. A single neuron in a neural network is essentially performing a logistic regression. Furthermore, in the real world, these models are preferred when interpretability is crucial. A bank deciding whether to approve a loan needs to explain why it made its decision—a task much easier with logistic regression than with a billion-parameter neural network.

Core Concepts Explained Intuitively

Linear Regression: Finding the Best Fit

Imagine scattering points on a graph where the X-axis is the size of a house and the Y-axis is its price. Linear regression aims to draw a straight line through these points that best captures the trend.

Mathematically, the line is defined as: $\hat{y} = w \cdot x + b$ Where:

• $\hat{y}$ is the predicted value.
• $x$ is the input feature.
• $w$ is the weight (the slope of the line).
• $b$ is the bias (where the line intercepts the Y-axis).

The Cost Function (MSE)

How do we know if our line is a "good" fit? We use a Cost Function to measure the error between our line's predictions and the actual data points. For linear regression, we typically use Mean Squared Error (MSE). It measures the average squared distance between the actual values and the predicted values. The goal of the algorithm is to find the parameters ($w$ and $b$) that minimize this cost function.

Gradient Descent: The Engine of Learning

How do we find the parameters that minimize the cost? Enter Gradient Descent. Imagine you are blindfolded on a hilly terrain and want to reach the lowest point (the minimum cost). You feel the slope of the ground under your feet (the gradient or derivative of the cost function). You take a step in the direction of the steepest downward slope. You repeat this until you reach a valley where the ground is flat (gradient is zero).

The size of the step you take is governed by the Learning Rate ($\alpha$). If the step is too small, it takes forever to reach the bottom. If the step is too large, you might overshoot the valley completely.

Logistic Regression: Predicting Probabilities

What if we want to classify whether an image is a dog (1) or a cat (0)? A straight linear regression line might output values like -2.5 or 14.3, which don't make sense as probabilities.

Logistic Regression solves this by wrapping the linear equation inside a Sigmoid Function. $\sigma(z) = \frac{1}{1 + e^{-z}}$ Where $z = w \cdot x + b$.

The sigmoid function maps any real number to a value strictly between 0 and 1. We can then interpret this output as a probability. For example, if $\sigma(z) = 0.85$, there is an 85% probability the image is a dog. If the probability is > 0.5, we classify it as a dog; otherwise, a cat.

Code Examples

1. Beginner Code Example: Linear Regression with Scikit-Learn

This is how you would implement linear regression in practice using Python's standard machine learning library.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# 1. Generate some synthetic data (House sizes vs Prices)
np.random.seed(42)
X = 2 * np.random.rand(100, 1) # Square footage (in 1000s)
y = 4 + 3 * X + np.random.randn(100, 1) # Price (in $100,000s) + noise

# 2. Initialize and train the model
model = LinearRegression()
model.fit(X, y)

# 3. Make predictions
X_new = np.array([[0], [2]])
y_predict = model.predict(X_new)

# 4. Visualize the best fit line
plt.plot(X_new, y_predict, "r-", linewidth=2, label="Predictions")
plt.plot(X, y, "b.")
plt.xlabel("Square Footage")
plt.ylabel("Price")
plt.legend()
plt.show()

print(f"Learned Weight (Slope): {model.coef_[0][0]:.2f}")
print(f"Learned Bias (Intercept): {model.intercept_[0]:.2f}")

2. Intermediate Code Example: Logistic Regression for Classification

Let's predict whether a student passes an exam based on hours studied.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix

# Features: [Hours Studied, Previous Test Score]
X = np.array([[1, 50], [2, 55], [3, 60], [4, 65], [5, 70], 
              [6, 75], [7, 80], [8, 85], [9, 90], [10, 95]])
# Labels: 0 (Fail), 1 (Pass)
y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train the Logistic Regression model
clf = LogisticRegression()
clf.fit(X_train, y_train)

# Predictions and probabilities
preds = clf.predict(X_test)
probs = clf.predict_proba(X_test) # Shows probability of [Class 0, Class 1]

print(f"Accuracy: {accuracy_score(y_test, preds) * 100}%")
print(f"Probabilities for test set:\n{probs}")

3. Advanced Code Example: Implementing Gradient Descent from Scratch

To truly understand linear regression, you must build its learning engine from scratch using raw NumPy math, without scikit-learn.

import numpy as np

# Generate data
X = 2 * np.random.rand(100, 1)
# Add a bias term (column of 1s) to X for easier matrix math
X_b = np.c_[np.ones((100, 1)), X] 
y = 4 + 3 * X + np.random.randn(100, 1)

# Hyperparameters
learning_rate = 0.1
iterations = 1000
m = 100 # number of instances

# Initialize random weights
theta = np.random.randn(2, 1)

# Gradient Descent Loop
for iteration in range(iterations):
    # 1. Calculate predictions: dot product of X_b and theta
    predictions = X_b.dot(theta)
    
    # 2. Calculate errors
    errors = predictions - y
    
    # 3. Calculate gradients (Partial derivatives of MSE)
    # The math reduces beautifully to: (2/m) * X^T * (Errors)
    gradients = 2/m * X_b.T.dot(errors)
    
    # 4. Update weights
    theta = theta - learning_rate * gradients

print("Final Weights found by Gradient Descent:")
print(f"Bias: {theta[0][0]:.3f}, Weight: {theta[1][0]:.3f}")

Common Mistakes

1. Not Scaling Features in Gradient Descent:

   • Mistake: Running gradient descent on features with vastly different scales (e.g., Number of Bedrooms [1-5] vs House Price [$100,000 -$1,000,000]).
   • Fix: The cost function contour will be extremely elongated, making gradient descent bounce around slowly. Always standardize/normalize features (e.g., using StandardScaler) before using gradient descent.

2. Using Accuracy for Imbalanced Logistic Regression:

   • Mistake: Evaluating a fraud detection model (where only 1% of transactions are fraud) using plain accuracy. A model that blindly predicts "No Fraud" every time will be 99% accurate but completely useless.
   • Fix: Use metrics like Precision, Recall, F1-Score, or the ROC-AUC curve for imbalanced classification tasks.

3. Ignoring the Assumptions of Linear Regression:

   • Mistake: Using linear regression when the relationship between $X$ and $y$ is highly non-linear (like an exponential curve).
   • Fix: Always plot your data. If it's non-linear, you may need polynomial regression, log-transformations, or a completely different model (like Decision Trees).

Interview Q&A

Q1: What is the difference between Linear and Logistic Regression? A1: Linear regression is used to predict continuous values (regression) and uses Mean Squared Error as its cost function. Logistic regression is used to predict discrete categories (classification), uses the Sigmoid function to output probabilities, and uses Log-Loss (Cross-Entropy) as its cost function.

Q2: What is the Learning Rate in Gradient Descent? What happens if it's too high or too low? A2: The learning rate is a scalar that determines the step size at each iteration while moving toward a minimum of a loss function. If it is too low, the algorithm will take a very long time to converge. If it is too high, the algorithm may overshoot the minimum and even diverge, causing the loss to increase.

Q3: How do you interpret the coefficients ($w$) in Linear Regression? A3: Assuming no interaction terms, a coefficient $w_i$ represents the expected change in the dependent variable $y$ for a one-unit change in the independent variable $x_i$, holding all other features constant.

Q4: Can Logistic Regression be used for multi-class classification? A4: Yes. While standard logistic regression is binary, it can be extended using the One-vs-Rest (OvR) strategy, or generalized to Multinomial Logistic Regression (Softmax Regression), which outputs a probability distribution across multiple classes.

Q5: What is Regularization (L1/L2) and why is it used? A5: Regularization adds a penalty term to the cost function to prevent overfitting by discouraging large weights. L1 Regularization (Lasso) adds the absolute value of weights and can drive some weights exactly to zero, performing feature selection. L2 Regularization (Ridge) adds the squared magnitude of weights, keeping all features but shrinking their coefficients.

Practice Problems

1. Problem: Implement a Ridge Regression model (Linear Regression with L2 regularization) using scikit-learn on the Boston Housing or California Housing dataset. Compare its performance to a standard Linear Regression model.

   • Hint: Import Ridge from sklearn.linear_model.

2. Problem: Modify the custom Gradient Descent code above to track and store the MSE cost at every iteration. Plot the cost against the number of iterations to visualize the learning curve.

   • Hint: You will need to calculate np.mean(errors ** 2) in the loop and append it to a list.

3. Problem: Write a logistic regression script to predict whether a patient has a disease based on medical metrics. Once trained, print out the coefficients. Which medical metric has the largest impact on the prediction?

   • Hint: Look at the absolute value of model.coef_.

Summary

• Linear Regression fits a line to data to predict continuous values.
• Gradient Descent is the optimization algorithm used to iteratively tweak model parameters to minimize error.
• Cost Function (MSE/Log-Loss) quantifies how wrong the model is at any given step.
• Logistic Regression passes the linear equation through a sigmoid function to output probabilities for classification tasks.
• Feature scaling and proper evaluation metrics are critical for these models to function correctly in production.