Feature Stores & Model Registry, In Full Depth

Experiment Tracking ended with a trained model registered under a name. Two problems are still open between that model and production. The model was trained on features computed by one piece of code, and in production the features will be computed by another, so they can quietly disagree. And of the many registered versions, something has to decide which one serves traffic, check it before it does, and record why. This page covers both: the skew and leakage bugs that feature stores exist to prevent, how a feature store works, a running Feast example, and the model registry that controls which version is live.

1. Two Systems That Must Agree

A model sees the world only through its features. During training, those features are usually computed in a batch job: Python or SQL over months of historical data in a warehouse, with no time pressure. During serving, the same features must be computed for one user, now, within a latency budget of a few milliseconds, often by a different service written in a different language by a different team.

That arrangement produces two distinct bugs. The first is that the two computations disagree about what a feature is. The second, subtler one, is that the training computation uses information that wouldn't have been available at prediction time. Both make a model look good in evaluation and do worse in production, and neither raises an error.

The Training-Serving Skew Problem

Training-serving skew is any difference between the features a model was trained on and the features it receives in production. Typical causes:

• Different code. The training pipeline computes "average transaction amount over 7 days" in SQL with AVG(), which ignores nulls; the serving service computes it in Java and treats missing amounts as zero.
• Different windows. Training uses calendar days; serving uses a rolling 168 hours. Close, but not the same number.
• Different freshness. Training uses values as of the transaction; serving uses a value last updated six hours ago by a nightly job.
• Different preprocessing. A scaler or vocabulary was refit after training, so serving maps inputs differently.

The model has no way to know. It receives numbers in the expected range and returns confident predictions; they are just worse. Google's "Rules of Machine Learning" lists skew among the most common sources of production quality problems, and its advice is the principle a feature store implements: compute features with the same code in both places, or log the features used at serving time and train on those.

The limitation of discipline alone: asking two teams to "keep the logic in sync" works until the first change made to only one side. The fix has to be structural, which is where the second bug also comes in.

2. Point-in-Time Correctness

The second bug happens entirely inside training. A training row is a labelled event: "this transaction, at this time, turned out to be fraud". The features attached to it must be the values that were known at that time, because those are the only values the model will have when it predicts in production.

The easy way to build a training set gets this wrong. Here are three labelled events and a feature table of daily snapshots, joined two ways:

import pandas as pd

features = pd.DataFrame({
    "user_id":           [42, 42, 42, 7, 7],
    "feature_time":      pd.to_datetime(["2026-10-01", "2026-10-05", "2026-10-09", "2026-10-02", "2026-10-08"]),
    "avg_txn_amount_7d": [31.0, 48.5, 220.0, 12.0, 15.5],
})
labels = pd.DataFrame({
    "user_id":    [42, 42, 7],
    "label_time": pd.to_datetime(["2026-10-06", "2026-10-10", "2026-10-03"]),
    "is_fraud":   [0, 1, 0],
})

# Wrong: join today's latest value onto every historical label
latest = features.sort_values("feature_time").groupby("user_id").tail(1)
naive = labels.merge(latest[["user_id", "avg_txn_amount_7d"]], on="user_id")

# Right: for each label, the latest feature value at or before the label's time
pit = pd.merge_asof(labels.sort_values("label_time"), features.sort_values("feature_time"),
                    left_on="label_time", right_on="feature_time", by="user_id", direction="backward")

print("naive:\n", naive.sort_values("label_time").to_string(index=False))
print("point-in-time:\n", pit[["user_id", "label_time", "is_fraud", "avg_txn_amount_7d"]].to_string(index=False))

naive:
  user_id label_time  is_fraud  avg_txn_amount_7d
       7 2026-10-03         0               15.5
      42 2026-10-06         0              220.0
      42 2026-10-10         1              220.0
point-in-time:
  user_id label_time  is_fraud  avg_txn_amount_7d
       7 2026-10-03         0               12.0
      42 2026-10-06         0              48.5
      42 2026-10-10         1              220.0

Look at user 42's row on October 6. The naive join gave it 220.0, a value computed on October 9, after the fraud spree that the 220.0 reflects. In training, the model learns that a high 7-day average goes with this user, using a number from the future. In production, it will never have that number in time. This is label leakage through time, and models trained with it show excellent offline metrics that collapse on deployment.

The correct operation is an as-of join: for each label, take the most recent feature value whose timestamp is at or before the label's timestamp. Formally, for an entity $e$ with label time $t$, the feature value used is

$x_e(t) = v_{e,\,j^*} \quad\text{where}\quad j^* = \arg\max_j \{\, \tau_{e,j} : \tau_{e,j} \le t \,\}$

with $\tau_{e,j}$ the timestamps of that entity's feature values. Getting this right by hand for dozens of features, each with its own update schedule, is tedious and easy to get wrong, which is one of the main reasons feature stores exist.

Both bugs have the same structural fix: one definition of each feature, with timestamps, served to training by an as-of join and to production by a fast lookup.

3. How a Feature Store Works

Feature Stores

A feature store is a system that holds feature definitions and feature values, and serves them consistently to training and to production. It has four parts.

A registry of feature definitions. Each feature belongs to a feature view: a named group of features about one entity (a user, a merchant, a device), computed from one source, with a timestamp column. The definition is code, kept in git and reviewed like code.

An offline store holds the full history of feature values, typically as tables in a warehouse or lake (BigQuery, Snowflake, Redshift, Parquet or Iceberg files). It answers the training question: "give me these features, as of each of these timestamps, for these entities". That's the as-of join from section 2, run at scale.

An online store holds only the latest value per entity, in a low-latency key-value database (Redis, DynamoDB, Bigtable, or SQLite for local work). It answers the serving question: "give me these features for user 42, now", in a few milliseconds.

Materialization copies the latest values from the offline store into the online store on a schedule, so the two hold the same numbers. For features that must be fresher than any schedule allows, a streaming job writes to both directly.

The guarantee follows from the structure: training and serving read values produced by one definition, so they agree by construction rather than by two teams staying in sync. The offline store's timestamps make point-in-time joins automatic.

Notice what a feature store doesn't do. In most designs it doesn't compute complex features itself; a batch or streaming job does that and writes the results in. The store's job is to make sure that whatever was computed is served identically to both sides.

4. A Working Feature Store with Feast

Feast is the most widely used open-source feature store. Here is a complete local setup using Feast 0.67, with Parquet files as the offline store and SQLite as the online store. The repository configuration, feature_store.yaml:

project: fraud
registry: data/registry.db
provider: local
online_store:
  type: sqlite
  path: data/online.db
offline_store:
  type: file
entity_key_serialization_version: 3

The feature definitions, definitions.py:

from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32

user = Entity(name="user", join_keys=["user_id"])

user_stats = FileSource(path="data/user_stats.parquet", timestamp_field="event_timestamp")

user_features = FeatureView(
    name="user_features",
    entities=[user],
    ttl=timedelta(days=30),
    schema=[Field(name="avg_txn_amount_7d", dtype=Float32)],
    source=user_stats,
)

data/user_stats.parquet holds the same five snapshots as section 2. Running feast apply registers the definitions and creates the online table:

Created feature view user_features
Created sqlite table fraud_user_features

Now use the store from both sides:

import pandas as pd
from feast import FeatureStore

store = FeatureStore(repo_path="repo")

# Training: one row per labelled event, each with the time the prediction would have been made
labels = pd.DataFrame({
    "user_id": [42, 42, 7],
    "event_timestamp": pd.to_datetime(["2026-10-06", "2026-10-10", "2026-10-03"], utc=True),
    "is_fraud": [0, 1, 0],
})
train = store.get_historical_features(entity_df=labels, features=["user_features:avg_txn_amount_7d"]).to_df()
print(train.sort_values("event_timestamp").to_string(index=False))

# Serving: load the latest values into the online store, then look one up
store.materialize_incremental(end_date=pd.Timestamp("2026-10-23", tz="UTC"))
online = store.get_online_features(features=["user_features:avg_txn_amount_7d"], entity_rows=[{"user_id": 42}]).to_dict()
print(online)

 user_id           event_timestamp  is_fraud  avg_txn_amount_7d
       7 2026-10-03 00:00:00+00:00         0               12.0
      42 2026-10-06 00:00:00+00:00         0               48.5
      42 2026-10-10 00:00:00+00:00         1              220.0
{'user_id': [42], 'avg_txn_amount_7d': [220.0]}

The training frame matches the hand-written as-of join exactly: user 42's October 6 row gets 48.5, not the future 220.0. The online lookup returns 220.0, the latest value, because that's what the model should see if user 42 transacts today. One definition produced both.

Two details worth noticing:

• ttl=timedelta(days=30) bounds how far back the as-of join will look. A label with no feature value in the previous 30 days gets a null rather than a stale value from a year ago, and the online store treats older values as expired.
• materialize_incremental copies only values newer than the last materialization. In production a scheduler runs it every few minutes or hours, which determines how fresh the online features are.

5. Freshness: How Old Can a Feature Be?

Every online feature has an age: the time since its value was computed. A nightly job gives features up to 24 hours old. A materialization every 15 minutes gives up to 15 minutes, plus the time the upstream job took.

Whether that matters depends on what the feature measures. A user's country barely changes, and a day-old value is fine. "Number of transactions in the last 10 minutes" is useless if it's an hour old, because fraud happens in bursts and the burst is the signal. A rough rule: a feature's maximum acceptable age should be small compared with the window it describes.

Freshness also has a training-side consequence that is easy to miss. If production features are typically 30 minutes old, a perfectly point-in-time-correct training set, which uses values exactly as of each label, is slightly too fresh. The model learns from features fresher than it will ever get. Teams that care about this either log the actual served features and train on those logs, or deliberately shift the as-of join back by the typical serving delay.

That completes the data side. The second half of the problem is the model itself: which trained version is serving, and how it got there.

6. The Model Registry

A model registry is the system of record for trained models that may be deployed. Experiment Tracking showed the mechanics: a registered model has a name, each registration creates a numbered version linked to its training run, and aliases such as @champion point at the version in use. This section is about what a registry is for in a team.

One answer to "what is serving right now?" The serving system loads models:/fraud-detector@champion, and the registry is the only place that alias is set. There is no copy of a model file on a server that someone updated by hand.

Lineage. Each version links to the run that produced it, the run records the code commit and the data version, and the data version can be fetched. When an incident report references a specific prediction, that chain answers which model made it, how that model was built, and from what.

Metadata that travels with the model. Each version carries its signature (input and output schema), its evaluation results, tags such as validation_status, and a description. A model card, covered in Legal, Licensing & Governance, belongs here too.

Access control and audit. Who may register, who may move @champion, and a log of every change with who made it and when. In regulated settings, that log is part of what an auditor asks for.

If you learned the registry from older material, it will describe fixed stages (Staging, Production, Archived). MLflow deprecated those in version 2.9 in favour of aliases and tags, and managed registries such as Databricks Unity Catalog never used them. The idea of moving a version through checks is unchanged; the mechanism is now a movable label.

7. Promotion: The Gate Between Training and Traffic

Training finishing should never, by itself, change what serves traffic. Promotion is a separate, deliberate step, and the registry is where it's enforced.

A typical gate for moving @champion to a new version checks, in order:

1. Schema. The new version's signature matches what the serving system sends. A renamed column fails here, not in production.
2. Quality against the current champion. On the same held-out evaluation set, the challenger must match or beat the champion on the primary metric, and must not regress beyond a set tolerance on important slices: per region, per customer segment, per device type. A model that improves the average by hurting one segment badly should not pass silently.
3. Operational limits. Latency at the 99th percentile, memory, and model size within what the serving infrastructure can hold.
4. Feature availability. Every feature the model uses exists in the online store with acceptable freshness. A model trained on a feature that serving can't provide is the skew problem again.
5. Approval. Automatic for low-risk models that pass every check; a named human for high-risk ones.

CI/CD & ML CI/CD automates checks 1 to 4 as a pipeline triggered by registration. Passing the gate usually doesn't send a version straight to 100% of traffic either; Deployment Strategies covers shadow and canary releases that test it on live traffic first.

8. Tools

Feature stores:

Model registries: MLflow Model Registry (open source, and the basis of the managed registries on Databricks and elsewhere), Unity Catalog models on Databricks, SageMaker Model Registry, Vertex AI Model Registry, Azure ML's registry, and the W&B Registry. The concepts on this page (versions, aliases, lineage, gated promotion) map onto all of them; the main difference is which identity and permission system they plug into.

9. When Not to Use a Feature Store

And a registry is worth having from the first model that serves real users, even if it is just MLflow on one machine. The cost is small, and the first incident without lineage is expensive.

Remember

Next: Deployment Strategies covers how a promoted version reaches real traffic safely, through shadow, canary and A/B releases.

Sources

• Feast, Documentation, including point-in-time joins
• M. Zinkevich, Rules of Machine Learning: Best Practices for ML Engineering, Google (rules 29 and 31 on training-serving skew)
• D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015
• MLflow, Model Registry