Train-Test Split Explained: Preventing Data Leakage in Machine Learning
Understand why a model must learn preprocessing only from training data, how pipelines help, and when random splitting is the wrong choice.
Why can a high model score be misleading?
A model can appear accurate when information from its evaluation data leaks into training. The key question is whether the model could use the same information at the moment a real prediction is made.
Suppose you predict whether a support ticket will be escalated when it is opened. A field recording the final escalation decision is an invalid input even if it is available in a historical spreadsheet. It reveals the answer. A field added after resolution can leak future information too.
What does a train-test split protect?
Training data is used to fit the model. Held-out test data is used for a final evaluation of the chosen workflow. Use a validation set or cross-validation for model and parameter selection; repeatedly choosing settings based on the test score compromises the holdout.
A random split can suit independent samples. For forecasting, train on earlier observations and evaluate later ones. If several rows belong to one customer, patient or device, keep related groups separated when the intended question is performance on unseen groups.
A small pipeline example
This teaching example uses Python and scikit-learn. The six synthetic rows are too few for a meaningful accuracy claim. The code demonstrates where preprocessing is fitted, not a production model or placement outcome.
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X = [[1, 5], [2, 7], [3, 6], [8, 2], [9, 1], [10, 3]]
y = [0, 0, 0, 1, 1, 1]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=2, random_state=42, stratify=y
)
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print(len(X_train), len(X_test), len(predictions))Expected sizes: 4 training rows, 2 test rows and 2 predictions. A fixed random_state makes this split reproducible for the same environment. The pipeline fits the scaler using training data, then applies the fitted transformation during prediction.
What should you avoid?
Do not fit a scaler on the full dataset before splitting. Do not perform feature selection using all target labels. Do not allow duplicate or related records to cross the split unknowingly. Do not interpret a tiny demonstration score as evidence that a model is reliable.
How should you evaluate a real project?
Write down when the prediction happens, which fields exist then, and what counts as an error. Compare with a simple baseline. For an imbalanced classification problem, inspect the confusion matrix and class-specific performance rather than accuracy alone. Keep the test set untouched until the workflow is chosen.
Practice: list five fields in a ticket dataset, then mark which ones are available at ticket creation. Explain why resolution_time is unavailable at that point. Next, choose whether the task needs a random, temporal or grouped split and justify the choice.
Learning and sources
Data Science course.
scikit-learn common pitfalls.
train_test_split reference.
AI Leaf Global Learn provides classroom learning in Mudichur/Tambaram, Chennai, and online learning across India. Confirm current batch and syllabus details with the team. Prepared by AI Leaf Global Learn; the code is reviewed against documentation, not represented as executed in a live ML environment.