Feature Stores: The Package Registry for ML Features
Your training pipeline computes 'average amount' as 30-day mean. Your API computes it as 7-day mean. Same name, different values. Feature stores fix this.
Your training pipeline computes “average transaction amount” as the mean of the last 30 days. Your inference API computes it as the mean of the last 7 days.
Same feature name. Different values. Your model is silently wrong.
This is training-serving skew. The number one silent killer of ML models in production.
The Problem
ML features get computed in two places:
| Context | How Features Are Computed | Problem |
|---|---|---|
| Training | Batch job on historical data, saved to CSV | Code written by data scientist |
| Serving | API computes on the fly per request | Different code, different logic |
Two separate implementations. They drift apart over time. Nobody notices until revenue drops.
What Is a Feature Store
A centralized system that stores, serves, and manages ML features. One definition. One computation. Used in both training and serving.
| Component | Purpose | Analogy |
|---|---|---|
| Offline Store | Historical features for training | Data warehouse |
| Online Store | Latest features for real-time inference | Redis / DynamoDB |
Same feature definition feeds both stores. Training-serving skew eliminated.
The DevOps Parallel
You don’t copy-paste library code into every project. You publish it to a package registry. Every team pulls from the same source.
Feature stores are package registries for ML features. Compute once. Register. Every model pulls from the same store.
When Do You Need One
| You DON’T need one if | You DO need one when |
|---|---|
| One model in production | Multiple models share features |
| One team owns features | Multiple teams compute independently |
| Training/serving use same code | You’ve caught skew bugs before |
Start simple. A shared feature library works for small teams. Graduate to a full feature store when feature reuse becomes a bottleneck.
Tools
| Tool | Best For |
|---|---|
| Feast | Open source, Kubernetes-native |
| SageMaker Feature Store | AWS-native teams |
| Vertex AI Feature Store | GCP-native teams |
This is Part 15 of the MLOps for DevOps Engineers series. For weekly updates, join the newsletter.