K‑Fold Cross Validation in Machine Learning

K‑Fold Cross Validation is a widely used evaluation method in machine learning that helps determine how well a model can generalize to new, unseen data. Instead of relying on a single train–test split, this technique divides the dataset into K equally sized portions, known as folds. The model is trained and tested repeatedly across these folds, producing a more stable and trustworthy performance estimate.

K - Fold train and test split

Understanding Cross Validation

Cross validation plays an important role in model assessment because it:

• Prevents overfitting — ensuring the model performs well not only on training data but also on unseen samples.

• Provides a more reliable evaluation — results are averaged across multiple runs, reducing bias and variance.

• Maximizes data usage — especially useful when the dataset is small or expensive to collect.

How K‑Fold Cross Validation Works

The K‑Fold procedure can be summarized in three main steps:

1. Divide the dataset into K folds Each fold contains roughly the same number of samples.

2. Iterative training and testing For each iteration:

   • Train the model using K − 1 folds

   • Test the model using the remaining fold Every fold serves as the test set exactly once.

3. Combine the results Compute a performance metric (accuracy, precision, recall, etc.) for each fold and average them to obtain the final score.

Mathematical Representation

Let the dataset be $D$, split into $K$ folds $F_k$.

Training set for fold $k$:

$$D_{\text{train}}(k)=D\setminus F_k$$

Test set for fold $k$:

$$D_{\text{test}}(k)=F_k$$

Overall model performance:

$$\text{Performance}=\frac{1}{K}\sum _{k=1}^K\text{Metric}(M_k,F_k)<strong><br></strong>$$

$$<strong><br></strong>$$

Choosing the Right Value of K 

The value of K influences both computation time and the stability of the evaluation:

• Small K (2–5) Faster but may produce higher variance in results.

• Large K (10 or more) More stable estimates but requires more computation.

• K = n (Leave‑One‑Out Cross Validation) Extremely thorough but computationally expensive.

A commonly used setting is 10‑Fold Cross Validation, which balances accuracy and efficiency for most applications.

Variants of K‑Fold Cross Validation

• Stratified K‑Fold Ensures each fold maintains the same class distribution as the full dataset — ideal for imbalanced classification problems.

• Repeated K‑Fold Runs K‑Fold multiple times with different splits, producing even more stable performance estimates.

• Leave‑One‑Out (LOOCV) Uses one sample per fold; suitable for very small datasets.

Benefits of K‑Fold Cross Validation

• Efficient use of data — every sample is used for both training and testing.

• More dependable evaluation — reduces the chances of overfitting or underfitting.

• Suitable for small datasets — especially when data collection is costly.

For time‑series data, a modified version called Time Series Cross Validation is used, where training always precedes testing in chronological order.

Limitations of K‑Fold Cross Validation

• High computational cost — the model must be trained K times.

• Risk of data leakage — preprocessing must be done carefully to avoid contaminating the test folds.

• Not ideal for time‑series — unless adapted with time‑based splits.

Source: https://www.geeksforgeeks.org/machine-learning/k-fold-cross-validation-in-machine-learning