Validation Protocol · draft
How the Scenario Simulator and Capacity Estimator will be tested.
A proposed, pre-specified study to find out whether the simulator's verdicts agree with experienced practitioners and where it is wrong. Nothing here has been run.
1. Question
Do the simulator's verdicts (comfortable, tight, at risk, not achievable) and its identified critical chain agree with independent experienced clinical programming leads on standardized scenarios, and is its effort estimation useful once calibrated on a team's own actuals?
2. Design
- Scenarios. 20 fully synthetic scenarios (no real study data) covering the three simulator types, written and frozen before any reviewer sees the tool. Include easy, borderline and deliberately misleading cases (for example, a plan that fails only because of a fixed external-data wait).
- Reviewers. Target 8 to 12 programming or biostatistics leads with at least 8 years' experience, recruited from outside the author's organization and clients. Report how they were recruited.
- Blind first. Each reviewer records their own verdict, estimated required working days, critical-chain item and top action for every scenario before seeing the tool's output. Then they see the output and record agreement and reasons.
- Independence. The author does not score or adjudicate. Reviewers disclose any relationship with the author or Vita Global Sciences. Reviewers are not compensated for any particular outcome.
- Estimator check. With one or more teams willing to share anonymized past actuals (counts and hours only), calibrate on an earlier period and test on a later one. Compare out-of-sample error with the team's original estimates.
3. Pre-specified measures
| Measure | How computed | Pre-set interpretation |
|---|---|---|
| Verdict agreement | Weighted kappa (linear weights) between each reviewer and the tool on the 4-level ordinal verdict; also pairwise reviewer-reviewer kappa | Tool-reviewer agreement is judged against reviewer-reviewer agreement, not against perfection |
| Critical-chain match | Share of scenarios where the reviewer names the same item as the tool | Report with 95% interval |
| Direction of error | Share of disagreements where the tool is more optimistic vs more pessimistic than the reviewer | Optimistic errors are weighted as more serious |
| Effort error (estimator) | Median absolute percentage error and bias, out-of-sample, vs the team's own original estimates | Useful only if it beats the team's historical estimates |
| Interval coverage | Share of actual outcomes falling inside P10 to P90 | Report; target near 80% |
| Tool errors | Every case where a reviewer judges a tool statement incorrect | All listed, each traced to a rule ID |
4. Analysis and reporting rules
- The protocol, scenarios and measures are fixed and dated before recruitment. Changes after that are listed as deviations.
- All disagreements are reported with reviewer reasons, grouped by rule ID. Rules that repeatedly disagree with reviewers are changed or removed, and the change is recorded in the changelog.
- With about 10 reviewers and 20 scenarios the study can show serious disagreement but cannot prove accuracy. The report will say so.
- Results describe agreement with expert judgment on synthetic scenarios. They do not show that using the tools improves real project outcomes; that would need a separate, prospective comparison.
5. Data and ethics
Scenarios are synthetic. Reviewers submit only their own judgments. Team actuals are counts and hours with no study, sponsor or person identifiers, shared only with the owner's written permission and stored without client names. Check client confidentiality terms and, if affiliated with an institution, whether ethics review is needed before collecting reviewer data.
6. Materials
- Reviewer response sheet (CSV)
- Scenario Simulator and Capacity Estimator (the tools under test)
- Evidence governance: no adoption or impact claims without evidence
7. Want to take part?
Independent reviewers and teams willing to share anonymized actuals are welcome via the contribute page. Taking part does not imply endorsement.