Results
Precomputed findings from two experiments: 20 seeds per scenario for each main comparison, 5 seeds for the sensitivity check, model typesafe/jev-1.13, pinned in both. Hover a badge for the hypothesis it tests. Numbers are seeded and reproducible from the configs in experiments/config/.
The verdict: it depends on the data
- Experiment 1: Jev lost. Every rule router beat it in every scenario, including R0, which always picks the same PSP.
- Why: the data was too noisy. Each PSP's per-segment numbers came from a handful of past payments and were off by ~7.6 points, more than the real differences between PSPs. Jev takes the numbers at face value; the rules won by ignoring them.
- Experiment 2: with 20× more data, Jev roughly ties the rules. It wins when a PSP degrades or is new, loses when one segment is thin, and mostly matches a simple statistics formula. Every difference is a few tenths of a point.
Either way, Jev still costs money and latency per decision, where rules are free and instant. Not a strong alternative to rules in these tests; a credible one with good data.
Routers
- R0
- Naive floor: always picks the same PSP. Exists only to show there's room for any router to matter — beating this proves nothing.
- R1
- Hand-tuned rules: an ordered rule list reflecting what a payments team would plausibly write from PSP contracts and dashboards.
- R2
- Tuned rules: R1's rule structure, but every parameter chosen by grid search on a held-out training slice, then frozen. Intended as the strongest rule baseline; in practice the tuning picked up noise and R2 ended up level with R1, so beating either means beating a solid rule set.
- R3 Jev
- Jev (R3): a probabilistic decision model. Sees the exact same eligible PSPs and observed stats as every other router, nothing more.
Scenarios
- Stable
- Nothing is wrong — the baseline. Tests whether Jev roughly matches tuned rules when nothing is drifting, stale, or thin.
- Drift
- PSP Beta's true acceptance quietly drops 8 points mid-stream; observed data only catches up after a lag. Tests whether Jev adapts faster than static rules.
- Thin data
- One PSP's history for a specific segment is capped at ~25 attempts. Tests whether Jev handles very little evidence better than rules.
- Cold start
- One PSP's entire observed history is capped at ~20 attempts, as if it just launched. Same test as thin data, for a whole PSP rather than one segment.
- Conflicting objectives
- The highest-acceptance PSP is also the most expensive, and amounts skew larger. Tests how routers handle a real accept-vs-cost trade-off.
- Eligibility mix
- The payment-method mix shifts so hard eligibility exclusions happen more often. Tests whether a noisier eligible-candidate set changes which router wins.
Experiment 1: the original test
Authorization rate by scenario and router
95% bootstrap confidence intervals (shown as whiskers), 14,000 decisions per bar.
R2 beats Jev's authorization rate in every scenario (0.70–1.36 points), significantly so (non-overlapping CIs) in 4 of 6. H3 predicted Jev might have an edge in thin_data/cold_start; instead the gap there is as large or larger than on stable.
Read across all four bars, too. R0, R1 and R2 are within about half a point of each other. R2's tuning didn't make it reliably better than R1 (it's below R1 in 4 of 6 scenarios). Jev is below all three, including R0, which ignores the data entirely. The next section explains why.
Why Jev lost: the observed data was noisier than the signal
A diagnosis made after Experiment 1's results, using no new calls: the recorded answers are replayed, and each router is scored by expected authorization, the true probability that its chosen PSP authorizes. That removes coin-flip luck. "Follow the data" picks whichever PSP has the highest observed segment acceptance.
In every scenario, following the data scores below R0, which ignores it: the numbers are too noisy to trust. Jev mostly follows them, and doesn't discount small samples, so it inherits the noise. Jev's preference for acceptance over fee (see the trade-off probe and calibration chart below) explains its cost results, not this authorization gap.
Experiment 2: the same test with 20× more data
Pre-registered before any Jev call (docs/hypotheses-v2.md). One change: each PSP's observed numbers come from 20× more past payments. Median segment sample 8 → 152; segment rate error 7.6 → 2.2 points. Same world, scenarios, transactions, seeds and model. R2 retuned on the new data. 84,000 live calls, all successful.
Expected authorization rate with good data
Expected authorization (true approval probability of each choice), 14,000 decisions per bar, 95% paired bootstrap CIs. "Jev, Exp. 1" is the same model on the same transactions with Experiment 1's data. The statistics reference blends segment and overall acceptance by sample size; it's a yardstick, not a competitor.
| Scenario (Jev minus…, points) | Jev, Exp. 1 | R1 | R2 | Statistics ref. |
|---|---|---|---|---|
| Stable | +1.05 [+1.01, +1.09] | +0.00 [-0.02, +0.03] | +0.25 [+0.22, +0.28] | -0.04 [-0.06, -0.03] |
| Drift | +1.05 [+1.00, +1.09] | +0.27 [+0.24, +0.31] | +0.35 [+0.31, +0.39] | -0.13 [-0.14, -0.11] |
| Thin data | +0.81 [+0.77, +0.86] | -0.39 [-0.42, -0.36] | -0.14 [-0.17, -0.11] | -0.10 [-0.11, -0.09] |
| Cold start | +1.00 [+0.97, +1.04] | +0.08 [+0.06, +0.10] | +0.27 [+0.24, +0.29] | +0.70 [+0.67, +0.73] |
| Conflicting objectives | +1.22 [+1.18, +1.26] | +0.20 [+0.17, +0.22] | -0.05 [-0.07, -0.03] | -0.02 [-0.03, -0.01] |
| Eligibility mix | +1.05 [+1.02, +1.09] | +0.00 [-0.02, +0.03] | -0.18 [-0.20, -0.16] | -0.04 [-0.05, -0.03] |
More data lifts Jev by about a point everywhere. It now ties R1 on stable, beats both rule sets on drift and cold start, and loses on thin data. Against the simple statistics formula it is level or slightly behind everywhere except cold start. Realized (coin-flip) authorization rates can't separate any of these differences; their CIs all overlap.
Experiment 2: hypothesis verdicts
| ID | Hypothesis | Supported if | Verdict | Evidence |
|---|---|---|---|---|
| V2-H1 | More data helps Jev. | Jev (v2) − Jev (v1) is positive with CI above zero in at least 5 of 6 scenarios. | ✅ Supported | Jev gained 0.81 to 1.22 points, CI above zero in 6 of 6 scenarios. |
| V2-H2 | With good data, Jev no longer loses to the rules on stable traffic. | On Stable, the CI of Jev − R1 includes zero or is above it. | ✅ Supported | Stable: Jev − R1 = +0.00 [-0.02, +0.03]. |
| V2-H3 | With good data, Jev beats static rules when a PSP drifts. | On Drift, Jev − R1 and Jev − R2 are both positive with CI above zero. | ✅ Supported | Drift: Jev − R1 = +0.27 [+0.24, +0.31], Jev − R2 = +0.35 [+0.31, +0.39]. |
| V2-H4 | Cold start still hurts Jev. | On Cold start, Jev − R1 is negative with CI below zero. | ❌ Not supported | Cold start: Jev − R1 = +0.08 [+0.06, +0.10]. |
| V2-H5 | Jev adds nothing beyond simple statistics. | In no scenario is Jev − statistics reference positive with CI above zero. | ❌ Not supported | Jev − statistics reference: Stable -0.04, Drift -0.13, Thin data -0.10, Cold start 0.70, Conflicting objectives -0.02, Eligibility mix -0.04 points. |
Caveats: on drift Jev beats the rules but only ties R0, because this world degrades the PSP that "always Alpha" mostly avoids. And a router that knows the truth still beats R1 by only ~0.5 points, so every win and loss here is small.
More detail from Experiment 1
Net cost per authorized transaction
Disagreement rate between Jev and R2, and who wins on disagreement
Jev decision cost and latency
Data richness: does more observed history change the outcome?
Levels 1 (overall only), 2 (+ segment stats), 3 (+ last 20 same-segment outcomes). Same 20 seeds.
Authorization rate is statistically indistinguishable across all three levels in both scenarios, but the underlying choice shifts a lot from level 1→2 and comparatively little from 2→3 — level 2's summarized segment stats capture most of what matters.
Sensitivity check: does the conclusion hold under a perturbed world design?
Stable scenario only, 5 seeds per variant (fewer than the main results — a robustness check, not a main-table number).
| Variant | Jev auth | R2 auth | Gap (R2−Jev) | Conclusion flipped? |
|---|---|---|---|---|
| segment_half | 92.5% | 93.6% | 1.09pp | no |
| segment_double | 94.2% | 95.1% | 0.94pp | no |
| drift_0.04 | 92.9% | 93.3% | 0.46pp | no |
| drift_0.12 | 93.5% | 93.7% | 0.26pp | no |
| lag_25 | 93.8% | 94.6% | 0.80pp | no |
| lag_100 | 92.5% | 93.9% | 1.43pp | no |
| window_75 | 93.2% | 93.5% | 0.31pp | no |
| window_300 | 93.4% | 93.9% | 0.57pp | no |
| chargeback_fee_25 | 93.1% | 93.6% | 0.51pp | no |
None of the 9 variants flipped the direction: R2 beat Jev in every one.
Probe: responsiveness
One PSP's observed acceptance swept from below to above a fixed competitor's value (alpha at 0.945), holding everything else fixed. Mean of 3 repeated calls per step.
Probe: trade-off
alpha (higher acceptance, higher fee) vs beta (lower acceptance, lower fee), a 1-point acceptance gap. Small amount (€10) and large amount (€800), 5 repeats each.
| Amount | Choices across 5 repeats |
|---|---|
| small | alpha, alpha, alpha, alpha, alpha |
| large | alpha, alpha, alpha, alpha, alpha |
Jev picked the higher-acceptance, pricier option every time, at both amounts — no small-amount-prefers-cheap effect at this acceptance gap.
Probe: irrelevance
5 harmless variations (shuffled PSP order, renamed display names, reordered metric glossary, an added unrelated field), 3 repeats each.
| Variant | Choices across 3 repeats |
|---|---|
| baseline | alpha, alpha, alpha |
| shuffled_psp_order | alpha, alpha, alpha |
| renamed_display_names | alpha, alpha, alpha |
| reordered_metric_glossary | alpha, alpha, alpha |
| unrelated_field_added | alpha, alpha, alpha |
Every variant stayed consistent with the baseline choice — a clean pass.
Probe: repeatability
Same state, 20 independent live calls.
Calibration: when Jev states probability p for a PSP, how often is it actually the best option?
Derived from the main comparison's 228,746 (PSP, probability) samples — "best" means the ground-truth best PSP among the eligible set for that transaction (section 9.1's rule), not simple authorization.
Not a calibration curve in the usual sense — the rate falls as stated probability rises through most of the range. This traces back to the trade-off probe: because most eligible PSPs here authorize, "actually best" is usually decided by cost, while Jev weights acceptance far more heavily than cost. High Jev confidence often loses to a cheaper alternative on this ground truth.