← Back to Home

Prediction Scorecard

How accurate Creek Intelligence's forecasts have been lately. Experimental — self-grading, updated daily.
Updated 2026-10-02 06:30 UTC · scoring window: last 45 days

Every forecast the system makes is recorded and later graded against what the river actually did — this page is that report card. It currently scores 294 physics crest predictions, 4370 empirical likelihood forecasts, and 0 recession countdowns (0 still maturing). Updated daily; treat it as experimental.

Where the truth comes from: USGS keeps revising its provisional gauge readings for months. The gauge archives these grades are checked against were last re-pulled from USGS on 2026-10-02 05:20 UTC (trailing 120 days, with approval status), and every settled grade whose readings are not yet USGS-approved is re-graded nightly; a grade that moves keeps its first verdict on the record. Last pass (2026-10-02 06:05 UTC): rise: 491 checked, 0 verdicts moved last night, 183 re-graded so far, 161 final; recession: 727 checked, 0 verdicts moved last night, 806 re-graded so far, 1145 final; physics: 1905 checked, 0 verdicts moved last night, 107 re-graded so far, 192 final; ponca: 968 checked, 0 verdicts moved last night, 678 re-graded so far, 0 final; empirical: 4827 checked, 0 verdicts moved last night, 2 re-graded so far, 0 final. USGS is retiring the data service these gauge feeds use; its replacement has been running alongside it since 2026-09-04: 663 hourly checks, 139204 readings compared between the two services with 0 value differences, and 117553 readings compared against what the live feeds stored with 10 differences. 569 readings were seen by the new service before the old one and 27 the other way round (timing, not disagreement). No gauge series is currently returning empty.

Physics Rise Predictors

Predicts the crest a gauge will reach from rainfall + antecedent moisture. Since 2026-09-08 every hourly run is a claim — a rise, or no rise — and both are graded: detection (rises the engine called) and false alarms side by side, then how close the crest and its timing were when a called rise happened.

Sample: 294 records · 5 events · 42 on the new grading · last 45 days · truth as of 2026-10-02 05:20 UTC · events counted on v2 records only (fixed 18-h horizon, from 2026-09-04)

New grading from 2026-09-04: each prediction is checked over a fixed 18-hour window after it was issued (before, the window closed 2 h after the predicted peak, so a later crest was never seen). It is graded on three separate questions: did the gauge rise at all, how far off was the crest when it did, and how far off was the timing. Predictions whose crest was not above the current level are no longer archived as claims.

Claims (from 2026-09-08): every hourly run is archived as a claim — a rise, or no rise (quiet / rain too light) — and graded 18 h later against the gauge. Detection = rises the engine called, out of every rise that happened (a no-rise claim followed by a rise is a miss). False alarms = rise predictions the gauge did not follow. Events = episodes of consecutive hourly runs.

Rise calls by confidence (from 2026-09-09): three copies of the physics engine run every hour, from cautious to eager. The eager one is served, and each rise call is labelled by the most cautious copy that also calls a rise — HIGH (all three agree), MEDIUM (medium and eager), LOW (eager only). Quoted is the verification rate the card showed for that label (leave-one-year-out replay, 2014–2026, June–September figure in summer); Live is what the gauge has done since.

Empirical Forecast Engine

Each hourly forecast is a calibrated probability that the gauge rises to the next level within the promised window (12 h on the fast creeks, 30 h on the slow ones). Scored as probabilities: Brier = mean squared error (0 is perfect); skill (BSS) = improvement over always forecasting the long-run rate (>0 is skill); obs/forecast = how many rises happened per rise forecast (1.0 is calibrated); reliability = what actually happened when the engine said X%. Probabilistic scoring began 2026-09-02; earlier records are not comparable and are not shown.

Sample: 4370 records · 8 events · 77 claims / 4293 no-rise calls · last 45 days · truth as of 2026-10-02 05:20 UTC · events = claim episodes (issuances >= 8 % within 3 h of each other); non-claims do not form episodes

"No rise indicated" calls (forecast below 8%) are graded separately on their miss rate: 4293 such hourly calls in the window, 0.0% of which were followed by a rise to the target level. Real claims (8% and above): 77, of which 0 verified.

All basins pooled: 4370 forecasts, 2 rises, Brier 0.00146, skill None, obs/forecast None, AUC None.

Reliability (when the engine said X%, how often did the rise come?)

Recession Countdowns

Predicts how long until a falling river drops to each threshold. HIT% = reached the level near the predicted time; MAE = average timing error (hours). Newly launched — most predictions are still maturing.

Sample: 0 records · 0 events · last 45 days · truth as of 2026-10-02 05:20 UTC · events = recession episodes per gauge/target

All 0 recession predictions are still maturing — a countdown can only be graded once enough time has passed to see whether the river reached the level. First results expected early July 2026.

Ponca AI Rainfall Event Analysis

When rain arms it, predicts how big the Ponca gauge will get — a class (Fizzle/Moderate/High/Flood), a typical-peak band, and a flood-risk %. Scored per armed EVENT since 2026-09-02: the first call and the most-informed ('mature', most rain so far) call of each event are graded against the crest the event actually reached; the flood-risk % is scored as a probability (Brier, 0 is perfect; 'shown' is the number on the card, 'raw' the analog match alone). After the crest the card hides the % unless a second, higher crest is coming, so post-peak calls are graded only on whether a higher crest came. Events, not 15-minute re-issues, are the sample size. The class words are the analog library's own storm-outcome groups (an internal grouping the card never shows), not the Ponca levels on the gauge card.

Sample: 348 records · 6 events · last 45 days · truth as of 2026-10-02 05:20 UTC

Last 45 days — 6 armed event(s), 348 fifteen-minute calls, 0 flood(s).

All time since 2026-06-08 — 16 event(s), 2 flood(s).

Reliability of the shown flood-risk % on rising calls (348 fifteen-minute rows — rows within one event are near-duplicates, so this shows calibration shape, not sample size): 0-10%: said 1% → 0% flooded (n=348).

Recent events — first call, mature call, actual

Radar Nowcast (Ponca card)

While rain is falling, the Ponca card reads the last 40 minutes of weather radar, moves the picture forward, and says roughly how much more rain lands on the creeks above Ponca in the next two hours (or that the rain is about done). Each radar frame's estimate is graded against the rain the next two hours actually delivered, using the card's own hourly rain accounting. Bias = estimate minus actual (negative means radar under-called); 'more coming' = the card said 0.1 in or more was on the way; 'about done' = it said little was left. The 'past storms like this went on to a real rise' perspective line is graded against whether the event rose at all.

Last 45 days — 328 radar frames graded across 7 rain event(s) (42 not gradable).

Verified = the call was right when made; caught = share of the real ≥0.1″ two-hour periods the card had called in advance.

By storm motion (estimates ≥0.1″): <8 mph (near-stationary): n=14, bias 0.056″, actual÷est 1.21; 8-20 mph: n=31, bias 0.05″, actual÷est 0.85; >=20 mph: n=13, bias 0.102″, actual÷est 0.57. Slow-moving storms grow over the watershed and deliver more than radar advection suggests; fast movers pass through and deliver less.

“Past storms like this went on to a real rise” line (2 lines in 2 events): “about half the time” (51%): said 1 times, the event rose 0% of them; “more often than not” (61%): said 1 times, the event rose 0% of them.

All time since 2026-07-11 — 706 radar frames graded across 12 rain event(s) (42 not gradable).

Verified = the call was right when made; caught = share of the real ≥0.1″ two-hour periods the card had called in advance.

By storm motion (estimates ≥0.1″): <8 mph (near-stationary): n=20, bias 0.067″, actual÷est 0.95; 8-20 mph: n=50, bias 0.062″, actual÷est 0.85; >=20 mph: n=22, bias 0.081″, actual÷est 0.39. Slow-moving storms grow over the watershed and deliver more than radar advection suggests; fast movers pass through and deliver less.

“Past storms like this went on to a real rise” line (24 lines in 6 events): “more often than not” (63%): said 20 times, the event rose 0% of them; “about half the time” (53%): said 4 times, the event rose 0% of them.

Buffalo Rise Engine (per-gauge nowcast)

For each Buffalo mainstem gauge, predicts a coming rise (slight / moderate / large) from local rain + upstream propagation, with a timing window. Graded on whether the gauge actually rose, and within the predicted window. Predictions issued in the last 45 days, with the all-time totals beside them; predictions only fire during rain events, so this fills in over time.

Sample: 235 records · 11 events · last 45 days · truth as of 2026-10-02 05:20 UTC

Predictions issued in the last 45 days: 235 graded records, 11 events (4 rose). All-time: 652 graded records, 48 events (23 rose, 48%).

Since 2026-09-03 the local-rain layer serves a rise probability and a typical rise band (25th–75th percentile, cfs) instead of a size word. "Band held" = the share of real rises that landed inside the band (target about 50%); Brier = mean squared error of the probability (0 is perfect, 0.25 is a coin flip).

A "rise" here means peak >= 1.25 x v0 and peak - v0 >= clamp(0.10 x v0, 10, 30) cfs — the same rule the engine uses to decide a predicted rise has arrived. Records graded before 2026-09-03 used a fixed +30 cfs floor, which read some real low-water rises as misses; the 45-day window mixes both vintages until mid-October. An event = a run of records less than 3 h apart.

Typical actual rise by predicted category: slight: ~28.1 cfs (n=13), moderate: ~28.0 cfs (n=3). (Categories come from rainfall, so this is how they map to real gauge rises — calibration that accrues over time.)

Recent rise predictions — predicted vs. actual

Downstream Propagation (Ponca → Pruitt → St. Joe)

Retired 2026-09-02. The Ponca card's own downstream model duplicated the wave tracker's reaches with weaker validation, so the card now narrates the wave tracker's bands and arrival windows for Pruitt, St. Joe and Harriet, and those are graded in the Routed Waves section below. The four events this section had graded included two synthetic host tests, which have been removed.

Retired — graded under Routed Waves below.

Routed Waves (wave tracker)

When an upstream gauge crests, the wave tracker predicts the crest size and arrival window at each downstream gauge. Every confirmed wave is graded after its window closes: did a real rise arrive, was the crest inside the predicted band, and did it arrive inside the window. Waves are rare events — this fills in slowly.

Sample: 1 records · 1 events · last 90 days · truth as of 2026-10-02 05:20 UTC · one grade per confirmed wave-target

Waves graded: 1 (significant: 1); crest inside the likely band 1/1; inside the wide band 1/1; arrived inside the window 0/1; median predicted/actual crest 1.0×. Alerts issued: 0; floods with no alert: 0.

Watershed Alerts (Q-bucket triggers)

The rain-triggered WATCH/WARNING words on the calibrated creeks (Upper Buffalo, Richland Main, Upper Cossatot, Upper Big Piney, Mulberry), graded against the creek's own gauge: an alert verifies when the creek reached its floatable floor within 24 h; a rise with rain and no alert is a miss. Same recipe as the leave-one-year-out numbers on the Watersheds page.

Sample: 31090 records · 6 events · last 90 days · truth as of 2026-10-02 05:20 UTC · records = 15-min Tier-1 decisions; events = alert episodes (+ rises with no alert)

Alert episodes: 6 (verified 1, false alarms 5, still open 0); rises with no alert: 1; median lead from first WARNING 3.2 h.

Rises with no alert: Upper Cossatot 2026-07-15 18:17Z (rain 0.77" of 1.20")

Neural Net Predictions (Ponca & Boxley)

A neural network projects the level hour by hour from 48 h of 1 km radar rainfall (Ponca 6 h ahead, Boxley 4 h). Every hourly issuance is graded against what the gauge actually did: MAE = average error of the median; IN-BAND% = how often the likely range contained the truth (target ~50%); SKILL = error reduction vs assuming the level doesn't change, on hours where it moved.

5 hour(s) in the window had no gauge data (USGS down): 5 re-served the previous projection with a note, the rest were offline.

558 issuances graded (model pixel-v5 / engine v3, 2026-09-08 → 2026-10-01).

Ponca (6 h)

800 cfs (Ponca: experienced only): 0 of 558 scored windows crossed (mean served probability 0.1%). Flood (1,600 cfs): 0 of 558 scored windows crossed (mean served probability 0.1%).

Boxley (4 h)

800 cfs (Ponca: experienced only): 0 of 558 scored windows crossed (mean served probability 0.8%). Flood (1,600 cfs): 0 of 558 scored windows crossed (mean served probability 0.2%).

All-time: 1923 issuances graded since 2026-07-10 (all models pooled).

Neural Net Predictions (Cossatot)

The Cossatot pixel net projects the level hour by hour from 48 h of 1 km radar rainfall (6 h ahead); crossing probabilities come from its companion gradient-boosted ensemble. Graded the same way as the Buffalo net: MAE = average error of the median; IN-BAND% = how often the likely range contained the truth (target ~50%); SKILL = error reduction vs assuming the level doesn't change, on hours where it moved.

8 hour(s) in the window had no gauge data (USGS down): 8 re-served the previous projection with a note, the rest were offline.

556 issuances graded (model cossatot-pixel-v6b / engine v3, 2026-09-08 → 2026-10-01; low-flow hold in 275).

Cossatot (6 h)

Floatable (240 cfs): 0 of 556 scored windows crossed (mean served probability 0.2%). High water (1,270 cfs): 0 of 556 scored windows crossed (mean served probability 0.3%).

All-time: 1874 issuances graded since 2026-07-14 (all models pooled).

Data health

Every grade above is checked against a USGS gauge feed. A single parameter can go silent while its site keeps reporting (USGS stops publishing discharge when the stage falls below its rating's lowest measured point); this table shows the age of the newest USGS reading per gauge and parameter, and whether a stage-derived stand-in is being served in its place.

21 series checked at 2026-10-02 06:30 UTC — 0 silent, 0 lagging.

All graded gauge series are current.

Level numbers: all 172 copies of the Boxley, Ponca and Hailstone levels (configs, countdown tables, forecast code and the last pages served) match the one shared list (registry 2026.9.28.1).

Coverage

What the scorecard grades today, and what is still being wired into the loop: