add preparedata plugin to latency based scorer to consume prefix states #2005

kaushikmitr · 2025-12-16T00:43:54Z

This pull request introduces refactoring and simplification of the prediction-based routing logic in the inference pool scheduler, focusing on introducing the PrepareRequestData plugin to get the prefix cache score and updating the prediction-based routing helm chart config.

Core logic and data structure simplification:

The sloRequestContext struct now stores all prediction results in a single predictionsForScheduling slice, replacing the previous separate maps for TTFT and TPOT values. All code paths and tests have been updated to use this unified structure.
The generatePredictions and scoreWithoutPredictions functions now rely exclusively on precomputed prefix cache scores from the SLO context, removing the need to pass and use the CycleState object for these calculations.
The PrepareRequestData method is introduced to precompute and populate prefix cache scores in the SLO context, further decoupling data preparation from scoring and prediction logic.

Prediction-based scheduling flow and configuration:

The "prediction-based scheduling off" feature and related code paths (including the NoLatencyRoutingProfileName and associated logic) have been removed, consolidating the routing flow and simplifying the profile handler's logic.
The SLOAwareProfileHandler.Pick method is simplified to always return all profiles unless all have already been executed, removing conditional execution based on headers.
The default value for samplingMean in the latency scorer configuration is increased from 100.0 to 1000.0.

Helm Chart:

epp-config.yaml is simplified to pick prediction based routing when enabled or pick default if not.

netlify · 2025-12-16T00:44:00Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`283a4e3`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/69496e4b59132c00083afc44
😎 Deploy Preview	https://deploy-preview-2005--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

kaushikmitr · 2025-12-16T00:45:55Z

@ahg-g this PR simplifies the helm chart config to pick latency aware routing if enabled or switch to default if not.

config/charts/inferencepool/templates/epp-config.yaml

pkg/epp/scheduling/framework/plugins/multi/slo_aware_router/slo_aware_profile_handler.go

pkg/epp/scheduling/framework/plugins/multi/slo_aware_router/preparedata_hooks.go

ahg-g · 2025-12-19T04:25:26Z

@kaushikmitr the naming is a bit all over the place, we use slo / predictor / router etc. I thought we agreed on predicted-latency as the name of the plugin. We can change that in a follow up PR, but I think we should address it soon since the plugin name will be part of the "config api" and it is user facing.

kaushikmitr · 2025-12-19T22:55:43Z

@kaushikmitr the naming is a bit all over the place, we use slo / predictor / router etc. I thought we agreed on predicted-latency as the name of the plugin. We can change that in a follow up PR, but I think we should address it soon since the plugin name will be part of the "config api" and it is user facing.

yes we need to clean up two things. The naming of the plugin (predicted latency) and renaming TPOT everywhere (including docs) to ITL

ahg-g · 2025-12-20T20:50:54Z

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

 		matchLen := state.PrefixCacheServers[ServerID(pod.GetPod().NamespacedName)]
 		pod.Put(approximateprefix.PrefixCacheMatchInfoKey, approximateprefix.NewPrefixCacheMatchInfo(matchLen, total))
 	}
+	// Store the state in plugin state for later use.


why do we need to change anything in the prefix plugin?

The prefix scorer does not consume the prefix state in the same way as the predicted latency one?

its a bug. We are not writing in the plugin state of the prefix-cache-scorer plugin during preparadata but only during scoring. This change will fix that. Also talked to @rahulgurnani about this. We can track it in a separate PR too.

This is not addressed yet

my comment was in pending state, see my response above

ahg-g · 2025-12-22T15:41:21Z

@kaushikmitr the naming is a bit all over the place, we use slo / predictor / router etc. I thought we agreed on predicted-latency as the name of the plugin. We can change that in a follow up PR, but I think we should address it soon since the plugin name will be part of the "config api" and it is user facing.

yes we need to clean up two things. The naming of the plugin (predicted latency) and renaming TPOT everywhere (including docs) to ITL

can we open an issue to track this work pls?

pkg/epp/scheduling/framework/plugins/multi/slo_aware_router/scorer.go

kaushikmitr · 2025-12-19T22:50:38Z

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

 		pod.Put(approximateprefix.PrefixCacheMatchInfoKey, approximateprefix.NewPrefixCacheMatchInfo(matchLen, total))
 	}
+	// Store the state in plugin state for later use.
+	p.pluginState.Write(request.RequestId, plugins.StateKey(p.TypedName().String()), state)


@ahg-g @kfswain talked to @rahulgurnani figured we need to write to plugin state in the preparadata step to ensure that running scorer is not needed to have the plugin state populated. I tested both with and without the feature gate.

PreRequest extension for the prefix plugin is executed even if the prefix scorer is not configured to run, right?

Yes, I validated that

pkg/epp/scheduling/framework/plugins/multi/slo_aware_router/slo_aware_profile_handler.go

kaushikmitr · 2025-12-20T21:26:32Z

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

 		matchLen := state.PrefixCacheServers[ServerID(pod.GetPod().NamespacedName)]
 		pod.Put(approximateprefix.PrefixCacheMatchInfoKey, approximateprefix.NewPrefixCacheMatchInfo(matchLen, total))
 	}
+	// Store the state in plugin state for later use.


its a bug. We are not writing in the plugin state of the prefix-cache-scorer plugin during preparadata but only during scoring. This change will fix that. Also talked to @rahulgurnani about this. We can track it in a separate PR too.

kaushikmitr · 2025-12-22T16:24:00Z

@kaushikmitr the naming is a bit all over the place, we use slo / predictor / router etc. I thought we agreed on predicted-latency as the name of the plugin. We can change that in a follow up PR, but I think we should address it soon since the plugin name will be part of the "config api" and it is user facing.

yes we need to clean up two things. The naming of the plugin (predicted latency) and renaming TPOT everywhere (including docs) to ITL

can we open an issue to track this work pls?
done
#2032

ahg-g · 2025-12-22T17:41:01Z

/lgtm
/approve

Thanks!

k8s-ci-robot · 2025-12-22T17:41:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, kaushikmitr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

add preparedata plugin to latency based scorer to consume prefix states

cab03a9

k8s-ci-robot requested review from ahg-g and danehans December 16, 2025 00:44

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 16, 2025

kaushikmitr added 2 commits December 16, 2025 03:17

remove unused method

dbdee3e

fix ineffectual assignment to prefixCacheScore

afa1cfb

ahg-g reviewed Dec 19, 2025

View reviewed changes

kaushikmitr added 2 commits December 19, 2025 22:43

remove two profiles from helm and router specific handler

3eabf90

simplfy code preparedata

44e0aea

ahg-g reviewed Dec 20, 2025

View reviewed changes

kaushikmitr added 2 commits December 22, 2025 04:41

fix preparedata hook bug

ab1d333

fix scorer test

5dba944

ahg-g reviewed Dec 22, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/multi/slo_aware_router/scorer.go Outdated Show resolved Hide resolved

kaushikmitr commented Dec 22, 2025

View reviewed changes

create rand number once in scorer when instantiated

283a4e3

k8s-ci-robot assigned ahg-g Dec 22, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 22, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 22, 2025

k8s-ci-robot merged commit 4c1b1ed into kubernetes-sigs:main Dec 22, 2025
12 checks passed

add preparedata plugin to latency based scorer to consume prefix states #2005

add preparedata plugin to latency based scorer to consume prefix states #2005

Uh oh!

Conversation

kaushikmitr commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

kaushikmitr commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahg-g commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaushikmitr commented Dec 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahg-g commented Dec 22, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaushikmitr commented Dec 22, 2025

Uh oh!

ahg-g commented Dec 22, 2025

Uh oh!

k8s-ci-robot commented Dec 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kaushikmitr commented Dec 16, 2025 •

edited

Loading

netlify bot commented Dec 16, 2025 •

edited

Loading

kaushikmitr commented Dec 16, 2025 •

edited

Loading

ahg-g commented Dec 19, 2025 •

edited

Loading