Building an Open-Source Hospital Bed-Demand Forecasting Benchmark
Hospital bed-demand forecasting sounds like a straightforward machine-learning problem:
past hospital activity
↓
forecasting model
↓
future bed demand
But comparing models is difficult.
Different projects often use different hospital datasets, prediction targets, time horizons, preprocessing rules, and evaluation metrics. A model tested on one hospital cannot be directly compared with a model tested on another hospital.
So I started thinking about a simple question:
Can we build a shared, reproducible benchmark for hospital bed-demand forecasting?
Why not just publish one hospital dataset?
Hospital data can contain sensitive information and is often difficult to share publicly.
Even when data can be released, a benchmark based on one hospital may represent only one type of patient flow, capacity setup, or operating process.
Instead, I think a useful starting point is to build the benchmark using synthetic hospital-flow simulations.
The goal is not to claim that synthetic data perfectly represents a real hospital.
The goal is simpler:
Give every forecasting model the same scenarios, the same baselines, and the same evaluation rules.
Synthetic hospital scenarios
A simulator could generate things such as:
patient arrivals
admissions
length of stay
discharges
available beds
occupancy over time
Because the simulator is controlled, we can also create different situations.
For example:
Normal week
Seasonal surge
Higher admission rate
Slower discharge rate
Temporary staffing constraint
Change in available capacity
Every model would receive the same generated data and predict future bed demand.
Start with simple baselines
Before testing complex models, the benchmark should include simple forecasting methods.
For example:
Seasonal naive
Prediction for Monday 10 AM = occupancy from last Monday at 10 AM
Historical average
Prediction = average occupancy for that hour and weekday
A new model should always be compared against these simple baselines.
Evaluate more than one number
Mean Absolute Error (MAE) is useful, but it does not describe every part of a capacity forecast.
A benchmark could report several metrics:
Metric | Question |
MAE | How far were the predictions from the true values? |
Threshold detection | Did the model detect when occupancy crossed a chosen capacity level? |
Lead time | How early did the model detect the upcoming threshold? |
Pinball loss | How good were probabilistic or quantile forecasts? |
Scenario response | Did forecasts change when the simulated hospital conditions changed? |
The last one is especially useful with simulations.
For example, if we create two otherwise identical scenarios but increase the discharge rate in one of them, we can compare how the forecasts behave.
This does not prove that a model understands hospital causality. It simply gives us another controlled test.
Keep the benchmark open
The project could use three simple contribution types:
Scenario
Baseline
Metric
Someone could contribute a new hospital-flow scenario.
Someone else could add a forecasting baseline.
Another contributor could add an evaluation metric.
Over time, the benchmark could grow into a shared test suite for hospital bed-demand forecasting.
The goal is not to replace evaluation on real hospital data.
It is to create a common starting point where forecasting methods can be tested under the same conditions.
That would make experiments easier to reproduce, compare, and improve.