Method and evaluation
What is forecast
Coldfront forecasts the ARE consultation incidence of the Robert Koch Institute: the estimated doctor visits for acute respiratory illness per 100,000 inhabitants and week. The forecast reaches 1 to 4 weeks beyond the last observed week, for Germany and for each federal state. Every forecast contains a 50 % band and an 80 % band.
The models
Five models are compared. Depending on the evaluation result, either the ensemble or the calendar-aligned seasonal baseline is shipped (see below).
- Seasonal baseline (plain): the value of a calendar week is the distribution of the same week in earlier, cleaned seasons, using a window of ±1 week. This was the baseline of the first protocol.
- Seasonal baseline (calendar-aligned): like the plain baseline, but the earlier values are first adjusted for the care-seeking calendar effect and aligned to the target week. It is the stricter reference of the second protocol and is shipped on fallback.
- Persistence: the last observed week is carried forward unchanged. It is the simplest reference.
- Regional model: linear quantile regression on the logarithmic growth, pooled over all federal states, with calendar adjustment. The feature groups are chosen in tuning.
- Ensemble: weighted average of the quantiles of the seasonal baseline and the regional model. The weight of the regional model is chosen from a grid in tuning (start value 0.5).
Spread factor: the bands of the ensemble are widened by a factor k, which is chosen together with the ensemble weight on the tuning seasons. The regularisation of the regression (value 1.0) is a start value.
Feature groups
| Group | Content |
|---|---|
| dynamics | Level compared with the previous year and growth over the last 1 and 2 weeks |
| season | Position in the yearly cycle (calendar week) |
| holidays | Share of vacation periods at the origin and in the target week |
| weather | Weekly maximum and minimum temperature (DWD) and their trend |
| wastewater | Wastewater load for SARS-CoV-2, influenza and RSV: level and growth (AMELAG) |
Why there is no forecast for the whole season
How strong a season is depends on the mix of pathogens, immunity and vaccination rates. These quantities cannot be derived from the data. Coldfront therefore extends the observed activity by 1 to 4 weeks. It does not make statements about the season level or the timing of a peak far beyond that.
Evaluation protocol
The protocol (version 1) was fixed before the first evaluation. It was amended once afterwards (version 2, see below). What follows is the state of version 2.
- No random split: the evaluation rolls forward. At each origin only data are used that would have been known at the time, with the publication lag of each signal.
- Tuning and test: the earlier seasons are used for tuning, the most recent one was the test season. Under version 1 this test season was evaluated once and was untouched at that time. Under version 2 it is descriptive only; the next untouched test season is 2026/27.
- The weeks from 2020-W10 to 2022-W39 are excluded, because testing, closures and quarantine rules distorted the series.
- Metrics: MAE (mean absolute error of the median), WIS (weighted interval score, lower is better, scores the median and both bands together) and the coverage of the 50 % and 80 % bands.
- Breakdowns by federal state, forecast horizon and wave phase. Phases: low below 30 % of the seasonal maximum, peak from 85 % of the maximum, otherwise rise or decline according to the direction of the smoothed curve.
- Comparisons use a block bootstrap: blocks of 4 weeks, 1,000 draws, fixed random seed. Reported are the mean WIS difference, the 95 % interval and the number of seasons in which the model is better.
- Peak timing (new in version 2): per forecast horizon, see the definition in the section on peak timing.
- Publication rule: a model is shipped only if its mean WIS is at least 10 % below the calendar-aligned seasonal baseline. The 10 % are a start value without empirical justification. Otherwise this baseline is shipped.
Amendment of the protocol (version 2)
After the first evaluation, a diagnosis was carried out on the tuning seasons only (not on the test season). It found a large, regular care-seeking calendar effect in the ARE data that the version 1 model could not explain, a flawed peak-timing metric, and an inconsistency in the lags of the wastewater features. The protocol was therefore amended and frozen again.
- Calendar adjustment: closed weekdays lower the number of doctor visits by a factor of about 0.81 each, the week between the years by a factor of about 0.4. GrippeWeb, which is not based on doctor visits, shows no such dip. The effect is estimated on data before 2023-W36 only and is applied in the regional model and in the seasonal baseline.
- Consistent lags: the availability lags of the features are now applied identically in training and prediction.
- Joint choice: the ensemble weight and the spread factor are chosen together from grids on the tuning seasons.
- New peak metric: peak timing is scored per horizon (definition below); the old metric, pooled over horizons, was flawed and has been replaced.
- Stricter reference: the publication rule compares with the calendar-aligned seasonal baseline. The plain baseline of the first protocol is still reported.
- Vacation periods now cover the season 2012/13 onward, so the Christmas dips of earlier winters enter training with their calendar features.
Status of the test season: the season was evaluated once under version 1 and was untouched at that time. Under version 2 it is evaluated a second time, for description only. It is no longer an independent test. Only the tuning seasons were used for the choices made under version 2.
Results (protocol version 2)
The tables show the evaluation of season 2025/26 under version 2 with 3,536 scored forecasts (region, origin week and horizon). Chosen on the tuning seasons: feature set dyn-season-holiday, weight 0.75 for the regional model and k = 1.10.
Protocol 1 and protocol 2 side by side
| Model | WIS (v1) | WIS (v2) | MAE (v1) | MAE (v2) | Coverage 50 % (v1) | Coverage 50 % (v2) | Coverage 80 % (v1) | Coverage 80 % (v2) |
|---|---|---|---|---|---|---|---|---|
| Seasonal baseline (calendar-aligned) | – | 189.68 | – | 263.91 | – | 32.6% | – | 53.5% |
| Seasonal baseline (plain) | 203.94 | 203.94 | 285.58 | 285.58 | 34.3% | 34.3% | 56.0% | 56.0% |
| Persistence | 227.11 | 227.11 | 307.11 | 307.11 | 47.9% | 47.9% | 78.3% | 78.3% |
| Regional model | 149.63 | 129.02 | 214.20 | 183.04 | 48.7% | 50.0% | 79.9% | 79.6% |
| Ensemble | 155.79 | 124.38 | 214.51 | 175.60 | 60.1% | 55.0% | 89.0% | 84.6% |
Both columns refer to the same season. The v1 result was the one-time evaluation; v2 is descriptive. A dash means the model did not exist under v1. The differences come from the amendments above, not from new data.
Publication decision
The ensemble is shipped. Its mean WIS is 34.4% below the calendar-aligned seasonal baseline. The threshold of 10% is a start value.
This decision rests on a single, already known season and is therefore provisional.
Comparisons with uncertainty
| Comparison | Mean WIS difference | Relative improvement | 95 % interval | Seasons better | n |
|---|---|---|---|---|---|
| Ensemble versus calendar-aligned seasonal baseline | -65.29 | 34.4% | from -94.42 to -41.47 | 1 of 1 | 3536 |
| Ensemble versus plain seasonal baseline | -79.56 | 39.0% | from -117.53 to -50.07 | 1 of 1 | 3536 |
| Ensemble versus persistence | -102.73 | 45.2% | from -161.88 to -55.81 | 1 of 1 | 3536 |
| Regional model versus calendar-aligned seasonal baseline | -60.66 | 32.0% | from -98.70 to -28.32 | 1 of 1 | 3536 |
A negative WIS difference means that the first model is better. The intervals rest on the block bootstrap over the weeks of a single season. One season is a thin basis: the interval shows the spread within the season, not the uncertainty between seasons.
By wave phase
| Phase | Seasonal baseline (calendar-aligned) | Seasonal baseline (plain) | Persistence | Regional model | Ensemble |
|---|---|---|---|---|---|
| low | 141.96 | 189.39 | 182.85 | 71.10 | 80.17 |
| rise | 199.27 | 213.13 | 264.14 | 149.14 | 134.26 |
| peak | 214.89 | 212.47 | 329.74 | 240.37 | 205.01 |
| decline | 208.75 | 203.93 | 197.06 | 121.50 | 124.79 |
By forecast horizon
| Weeks ahead | Seasonal baseline (calendar-aligned) | Seasonal baseline (plain) | Persistence | Regional model | Ensemble |
|---|---|---|---|---|---|
| 1 | 189.68 | 203.94 | 162.49 | 107.81 | 108.37 |
| 2 | 189.68 | 203.94 | 216.36 | 122.85 | 119.99 |
| 3 | 189.68 | 203.94 | 248.45 | 135.80 | 129.40 |
| 4 | 189.68 | 203.94 | 281.16 | 149.61 | 139.76 |
By federal state and for Germany
At national level the ensemble is thus about equal to the calendar-aligned baseline; the gain lies mainly in the federal states.
| Region | Seasonal baseline (calendar-aligned) | Seasonal baseline (plain) | Persistence | Regional model | Ensemble | Ensemble versus baseline |
|---|---|---|---|---|---|---|
| Germany (national) | 89.44 | 101.37 | 186.16 | 95.80 | 90.01 | -0.6% |
| Baden-Württemberg | 183.97 | 194.71 | 199.25 | 116.28 | 107.82 | +41.4% |
| Bavaria | 241.61 | 246.89 | 211.32 | 131.25 | 124.59 | +48.4% |
| Berlin | 123.33 | 144.20 | 180.51 | 93.65 | 88.25 | +28.4% |
| Brandenburg | 256.50 | 273.63 | 276.04 | 165.12 | 158.08 | +38.4% |
| Bremen | 171.87 | 182.58 | 181.75 | 101.98 | 96.94 | +43.6% |
| Hamburg | 157.77 | 169.46 | 151.93 | 87.79 | 90.40 | +42.7% |
| Hesse | 158.09 | 178.40 | 189.08 | 100.11 | 97.58 | +38.3% |
| Lower Saxony | 183.54 | 190.82 | 192.08 | 105.11 | 105.47 | +42.5% |
| Mecklenburg-Western Pomerania | 263.94 | 287.02 | 413.44 | 248.20 | 233.13 | +11.7% |
| North Rhine-Westphalia | 165.17 | 181.91 | 219.38 | 118.00 | 113.90 | +31.0% |
| Rhineland-Palatinate | 242.10 | 263.58 | 226.78 | 117.88 | 124.64 | +48.5% |
| Saarland | 214.97 | 222.92 | 209.54 | 124.57 | 124.18 | +42.2% |
| Saxony | 177.93 | 186.93 | 202.20 | 109.52 | 103.20 | +42.0% |
| Saxony-Anhalt | 134.91 | 150.65 | 262.27 | 145.36 | 133.94 | +0.7% |
| Schleswig-Holstein | 223.85 | 240.54 | 309.67 | 191.04 | 183.57 | +18.0% |
| Thuringia | 235.49 | 251.30 | 249.52 | 141.65 | 138.78 | +41.1% |
The last column shows the advantage of the ensemble WIS over the calendar-aligned baseline (positive: ensemble better). The row for Germany is highlighted.
Peak timing
Definition (version 2): for each region, season and forecast horizon, the peak is searched in the winter window from calendar week 45 to calendar week 15, as the maximum of the centred 3-week mean. The week between the years and the week after are removed. The peak of the median forecast is compared with the peak of the actual values; reported are the median absolute timing error in weeks and the share of cases within ±1 week.
Persistence lags by about h weeks by construction, because it repeats the last week; its timing error serves as the reference for "nothing learned".
At horizon 1 the ensemble finds the peak within one week in only 53% of the cases (median offset 1.0 weeks). At horizons 2 to 4 the median timing error is between 7 and 9 weeks. This is a real weakness: the forecast is not suited to predict the date of the peak.
The old peak metric of the first protocol pooled all horizons and weeks, was discontinuous and has been replaced. Its values are not comparable.
| Model | Scored cases | Median timing error (weeks) | Share within ±1 week | Of which ambiguous | Mean relative height error |
|---|---|---|---|---|---|
| Seasonal baseline (calendar-aligned) | 68 | 7.0 | 29% | 20 | 13.9% |
| Seasonal baseline (plain) | 68 | 8.0 | 41% | 20 | 11.9% |
| Persistence | 68 | 3.0 | 24% | 20 | 0.3% |
| Regional model | 68 | 7.0 | 24% | 20 | -10.1% |
| Ensemble | 68 | 7.0 | 22% | 20 | -6.6% |
Ambiguous means: the actual curve has a second peak of at least 90 % of the maximum, so the timing is not clear-cut there.
| Model | Weeks ahead 1 | Weeks ahead 2 | Weeks ahead 3 | Weeks ahead 4 |
|---|---|---|---|---|
| Seasonal baseline (calendar-aligned) | 7.0 | 7.0 | 7.0 | 7.0 |
| Seasonal baseline (plain) | 8.0 | 8.0 | 8.0 | 8.0 |
| Persistence | 1.0 | 2.0 | 3.0 | 4.0 |
| Regional model | 1.0 | 7.0 | 8.0 | 10.0 |
| Ensemble | 1.0 | 7.0 | 8.0 | 9.0 |
| Model | Weeks ahead 1 | Weeks ahead 2 | Weeks ahead 3 | Weeks ahead 4 |
|---|---|---|---|---|
| Seasonal baseline (calendar-aligned) | 29% | 29% | 29% | 29% |
| Seasonal baseline (plain) | 41% | 41% | 41% | 41% |
| Persistence | 94% | 0% | 0% | 0% |
| Regional model | 59% | 35% | 0% | 0% |
| Ensemble | 53% | 24% | 12% | 0% |
Choice of features, weight and spread (tuning)
Tuning on the seasons 2023/24, 2024/25. For every feature set the best ensemble weight and the best spread factor are searched jointly. The set with the lowest mean WIS of the ensemble is chosen. The chosen candidate is dyn-season-holiday with weight 0.75 for the regional model and k = 1.10.
For comparison on the same tuning seasons: WIS of the calendar-aligned seasonal baseline 249.66, of the plain baseline 269.15, of persistence 254.99.
| Candidate | Feature groups | Weight regional model | Factor k | WIS ensemble | WIS regional model | n |
|---|---|---|---|---|---|---|
| dyn | dynamics | 0.60 | 1.10 | 169.33 | 193.63 | 6976 |
| dyn-season | dynamics, season | 0.75 | 1.10 | 161.75 | 170.30 | 6976 |
| dyn-season-holiday (chosen) | dynamics, season, holidays | 0.75 | 1.10 | 155.55 | 160.18 | 6976 |
| dyn-season-weather | dynamics, season, weather | 0.75 | 1.10 | 160.03 | 167.40 | 6976 |
| dyn-season-wastewater | dynamics, season, wastewater | 0.75 | 1.00 | 166.60 | 176.88 | 6976 |
| all | dynamics, season, holidays, weather, wastewater | 0.75 | 1.10 | 160.24 | 167.55 | 6976 |
| Factor k | WIS ensemble | Coverage 50 % | Coverage 80 % |
|---|---|---|---|
| 1.00 | 156.06 | 43.0% | 72.1% |
| 1.10 (chosen) | 155.55 | 47.0% | 76.2% |
| 1.20 | 155.77 | 50.3% | 79.8% |
| 1.35 | 157.21 | 55.5% | 83.9% |
| 1.50 | 159.82 | 59.7% | 87.9% |
| 1.75 | 166.58 | 66.9% | 92.2% |
| 2.00 | 175.65 | 72.8% | 95.2% |
Real data versions
The RKI corrects ARE values retroactively. The public version history of its data (since September 2023) allows reconstructing what was actually known at each origin. The table compares the WIS with today's corrected data and with the data versions of the time.
| Model | WIS, corrected data | WIS, data version of the time | Change | Cells |
|---|---|---|---|---|
| 2023/24: 56 of 56 origins with a real data version, 0 fell back to the corrected state | ||||
| Seasonal baseline (calendar-aligned) | 269.72 | 273.50 | +1.4% | 3440 |
| Persistence | 240.54 | 241.15 | +0.3% | 3440 |
| Ensemble | 149.40 | 157.53 | +5.4% | 3440 |
| 2024/25: 56 of 56 origins with a real data version, 0 fell back to the corrected state | ||||
| Seasonal baseline (calendar-aligned) | 230.14 | 230.73 | +0.3% | 3536 |
| Persistence | 269.05 | 267.23 | -0.7% | 3536 |
| Ensemble | 161.54 | 164.25 | +1.7% | 3536 |
| 2025/26: 56 of 56 origins with a real data version, 0 fell back to the corrected state | ||||
| Seasonal baseline (calendar-aligned) | 189.68 | 189.68 | 0.0% | 3536 |
| Persistence | 227.11 | 236.08 | +3.9% | 3536 |
| Ensemble | 124.38 | 129.85 | +4.4% | 3536 |
Note: in both variants the values to be hit are today's corrected values. Revisions of wastewater and weather data are not reproduced. A positive change means that the WIS is worse with the data versions of the time.
Comparison with other forecasts (national)
For Germany as a whole, forecasts of other groups are available in the RESPINOW-Hub (RESPINOW-Hub forecast archive (KITmetricslab)). Their counts were converted to cases per 100,000 with the fixed population ratio 831.4. Only cells are compared for which all models shown have a forecast (panel "core": models covering both seasons). Forecasts of the other groups are marked "external" below.
RespiCast was not reachable and is therefore not part of the comparison.
| Model | n | MAE | WIS | Coverage 50 % | Coverage 80 % |
|---|---|---|---|---|---|
| Seasonal baseline (calendar-aligned) | 156 | 294.77 | 196.34 | 39.1% | 83.3% |
| Persistence | 156 | 350.00 | 258.47 | 49.4% | 81.4% |
| Ensemble | 156 | 200.75 | 149.55 | 71.2% | 91.7% |
| KIT-LightGBM (external) | 156 | 244.83 | 170.66 | 36.5% | 69.9% |
| KIT-MeanEnsemble (external) | 156 | 239.80 | 183.45 | 61.5% | 85.9% |
| KIT-TSMixer (external) | 156 | 278.58 | 193.91 | 55.1% | 87.8% |
| KIT-hhh4 (external) | 156 | 363.60 | 265.82 | 51.3% | 82.1% |
Scored cells per season: 2024/25: 88; 2025/26: 68. In total 156 cells from 39 origin weeks.
| Comparison | Mean WIS difference | Relative improvement | 95 % interval | Seasons better | n |
|---|---|---|---|---|---|
| Ensemble versus KIT-LightGBM (external) | -21.11 | 12.4% | from -57.42 to 12.61 | 2 of 2 | 156 |
| Ensemble versus KIT-MeanEnsemble (external) | -33.89 | 18.5% | from -74.60 to 2.60 | 2 of 2 | 156 |
| Ensemble versus KIT-TSMixer (external) | -44.36 | 22.9% | from -85.41 to -6.17 | 2 of 2 | 156 |
| Ensemble versus KIT-hhh4 (external) | -116.27 | 43.7% | from -177.36 to -60.51 | 2 of 2 | 156 |
In 2 of 4 comparisons the 95 % interval includes zero, so the difference is not statistically established there.
Forecasts: RESPINOW-Hub, KITmetricslab (MIT licence); Prognosen der Teams KIT-*, MPIDS-*. Source: RESPINOW-Hub forecast archive (KITmetricslab), Licence MIT.
Why this is not a fair contest
- The external forecasts were produced in real time with the data available then and contain nowcasts for the still incomplete last week. The Coldfront figures here use today's corrected data. This favours Coldfront.
- For Coldfront, season 2024/25 was used for tuning (in-sample), and season 2025/26 was already used under protocol 1.
- The same WIS with the quantiles 0.1, 0.25, 0.5, 0.75 and 0.9 is computed for all models. This is not the WIS of the RESPINOW-Hub dashboard, which uses more quantiles.
- The sample is small: a few dozen origin weeks, and the horizons of one origin are strongly dependent. The intervals are wide and partly include zero (see above); a difference is established only where the interval excludes zero.
Ongoing check
Archived forecasts are compared with the actuals that arrive later, every week. If the ensemble scores worse than the baseline over at least 8 evaluated runs, the baseline replaces it automatically. Currently 0 runs are scored.
Limits
- Few seasons: the evaluation rests on one descriptive season, and on about three seasons for the federal states overall. Differences between models are therefore uncertain; the intervals show this.
- First publications: the evaluation scores against today's corrected values, not against the values published first. A live forecast only had the data of its time; the comparison with data versions above reproduces this in part.
- Comparison with other initiatives: it exists for the national level only and only against the RESPINOW models, is not fair and has a small sample. The federal states are not compared.
- Peak timing: from horizon 2 the ensemble misses the peak by many weeks. The forecast does not predict the date of the peak.
- Start values: regularisation, the 10 % threshold, the grids for weight and spread and the activity levels have not been checked empirically.
- No statements about individuals or the medical situation: the forecast describes trends at population level, not diagnoses and not recommendations.