Ask an innovation agency how its ecosystem is doing and you will receive a dashboard. Ask what on that dashboard would change if the agency had never existed, and the room goes quiet. Ecosystem measurement is where good intentions meet bad statistics: inputs are counted as outcomes, activity is reported as impact, and the numbers that would actually answer a minister’s question arrive three years after anyone asked it. This article proposes a measurement architecture that a public auditor would accept and a founder would recognise — and explains why most current dashboards are neither.
The idea in brief. Ecosystem KPIs fail for three reasons: they count inputs (events, funds announced, founders trained) as if they were outcomes; they ignore the time it takes for a genuine outcome to become visible; and they claim attribution without a comparison. The fix is a logic model that separates inputs, outputs, outcomes and impact; a small KPI set with explicit time horizons and definitions fixed in advance; cohort-based tracking of ventures over years; and evaluation designs — often free, using oversubscription — that let an agency say what it caused rather than what it witnessed. Global indices are useful for benchmarking a country’s position, not for judging a program.
Why ecosystem dashboards lie
The dashboards are not dishonest. They are optimised for the wrong thing. A public agency must report quarterly; a venture takes years to reveal whether it is real; and so the reporting system fills with whatever is available quarterly. Events held, applications received, founders trained, mentors enrolled, memoranda signed, funds announced: all of these are countable within the reporting window, all are within the agency’s control, and none of them is an outcome. The result is a measurement culture that Charles Goodhart’s and Donald Campbell’s well-known observations predicted — when a measure becomes a target, it stops being a good measure — applied to a whole policy field.
Three distortions recur. The first is input inflation: an agency that is judged on founders trained will train more founders, more briefly. The second is announcement accounting: a fund “launched” is reported before a rupee or dirham is deployed, and a memorandum with a university is reported before a single project begins. The third is survivorship display: the ten successful alumni are photographed; the two hundred who quietly closed are not counted, so no one knows whether ten out of two hundred is good.
The scholarly literature on entrepreneurial ecosystems has been pointing at this problem for a decade. Erik Stam’s 2015 “sympathetic critique” in European Planning Studies argued that ecosystem thinking risks becoming a list of ingredients without a theory of how they combine into outcomes, and proposed distinguishing framework conditions and systemic conditions from the outputs (entrepreneurial activity) and outcomes (value creation) they are supposed to produce. Ben Spigel’s 2017 work in Entrepreneurship Theory and Practice made a related point: ecosystems are relational — what matters is how cultural, social and material attributes reinforce one another — which means counting the attributes in isolation tells you little. Measurement that ignores these distinctions is not measuring the ecosystem; it is measuring the agency’s diary.
A logic model that separates inputs from outcomes
The most useful tool in this field is also the oldest and least glamorous: the program logic model, popularised for grant-makers by the W.K. Kellogg Foundation. It forces every metric into one of five columns, and the discipline of assigning each number to a column is most of the cure.
| Column | Definition | Ecosystem examples | Who controls it |
|---|---|---|---|
| Inputs | Resources committed | Budget, staff, fund capital, incubator space | Agency |
| Activities | What the agency does | Cohorts run, grants awarded, events held, mentors matched | Agency |
| Outputs | Immediate, countable results of activities | Founders completing programs, ventures incorporated, patents filed | Agency and participants |
| Outcomes | Changes in participants’ condition | Ventures operating at 24 months, revenue, employment, capital raised, licences executed | Participants and market |
| Impact | Changes in the ecosystem or economy | Startup formation rate, high-growth firm share, private capital availability, exports, tax base | Everyone; attribution partial |
The rule that follows is simple and rarely observed: an agency may be held accountable for inputs, activities and outputs; it may claim contribution to outcomes; and it may only describe impact. Dashboards that place “startups created” in the same list as “events held” without labelling the columns are the source of most confusion between agencies and their auditors, and most of the political trouble that follows.
The time-to-signal problem
Every genuine outcome has a latency: a period after the activity during which the outcome is not yet observable, however real the program’s effect. Ignoring latency produces two opposite errors — declaring victory on early proxies, and declaring failure before the outcome could possibly have appeared. The chart below is an illustrative model of how long typical ecosystem indicators take to become believable; the exact months vary by sector, but the ordering is stable.
Three implications for agency design. First, no program should be judged on outcome KPIs before its latency has elapsed — and the latency should be written into the program’s charter so that a change of minister does not reset expectations. Second, leading indicators must be chosen for their predictive relationship to outcomes, not their availability: the number of customer interviews a cohort has conducted predicts twelve-month revenue far better than the number of workshops it attended. Third, reporting cadences should differ by column: activities monthly, outputs quarterly, outcomes annually by cohort, impact every three to five years with external evaluation.
Leading indicators worth trusting
Because outcomes arrive late, an agency needs early signals it can defend. The test for a leading indicator is not whether it is available but whether it has been shown — in the agency’s own cohort data or in the research — to predict the outcome it stands in for. Five candidates pass that test more often than most:
- Customer conversations completed during the program, verified by notes or recordings. Ventures that have spoken to fifty potential buyers behave differently from ventures that have spoken to five.
- Paid or piloting customers at exit — the single best predictor of twelve-month revenue in most cohorts.
- Founder commitment: the share of founders working full-time on the venture by program end. Part-time ventures rarely convert.
- Team completion: a co-founder or first hire with the complementary skill the venture lacked at intake.
- Qualified demand for the next stage — applications to follow-on programs, investor meetings taken, corporate pilots requested — as a signal that outsiders see what the program sees.
Each of these can be reported at program end, which is when a minister wants a number. Reported honestly as leading indicators, they buy the program the time its outcomes need.
What the global indices measure — and what they do not
Governments across India and the Gulf pay close attention to global rankings, and rightly: they shape investor perception and offer comparable benchmarks. But each index measures a specific thing, and none of them measures whether a particular program worked.
- The Global Innovation Index, published by WIPO, combines roughly eighty indicators of innovation inputs (institutions, human capital, infrastructure, market and business sophistication) and outputs (knowledge, technology and creative outputs). It is a national-systems measure, slow-moving by design, and it rewards data availability. A country’s rank can rise because its statistics improved.
- The Global Entrepreneurship Monitor (GEM) surveys adults directly, producing rates of total early-stage entrepreneurial activity (TEA), established business ownership, and attitudes. It measures how many people are trying, not how well the system supports those who try; high TEA is common in economies with few formal jobs.
- The Global Entrepreneurship Index developed by Zoltán Ács, Erkko Autio and László Szerb, whose 2014 paper in Research Policy introduced the “national systems of entrepreneurship” framing, deliberately combines attitudes, abilities and aspirations with institutional quality — and is explicit that a system’s weakest component constrains the whole.
- Startup Genome and StartupBlink rank cities and countries on startup output, funding and reach, using commercial databases; they are the most startup-specific and the most sensitive to funding cycles and data coverage.
- National indices such as NITI Aayog’s India Innovation Index bring the same logic to sub-national comparison and are useful for state-level policy competition.
The trajectories of the corridor’s two largest economies illustrate both the value and the limits of these instruments. India’s climb in the Global Innovation Index — from the eighties in 2015 to the top forty by the early 2020s — is a genuine signal of a system improving on many fronts at once. The UAE’s steady position in the low thirties reflects strong institutions and infrastructure alongside a smaller knowledge-output base. Neither line tells a ministry whether its founder program produced ventures.
The practical rule: use indices to set context and ambition at the national level, and to identify which pillar is weakest; never use them to evaluate a program, and never set a program’s target in index points.
Attribution versus contribution
The hardest question in ecosystem measurement is also the one auditors increasingly ask: what would have happened anyway? A founder program that selects the most promising applicants and then reports their success is reporting its selection, not its effect. The classic evaluations in this field — Start-Up Chile, Nigeria’s YouWiN! competition, the Togo training experiment discussed in article 31 — are credible precisely because they had a comparison group, and in each case the comparison changed the conclusion.
Agencies rarely need a research partnership to do better. Three designs are available at low cost:
- Oversubscription lotteries. Where a program has more qualified applicants than places, allocating places among qualifiers transparently at random creates a comparison group for free. The qualifiers who did not get in are the counterfactual. This is ethical (all were equally deserving), politically defensible (it is fairer than opaque ranking) and analytically powerful.
- Staggered rollout. Where a program expands region by region, the later regions serve as a temporary comparison for the earlier ones. The comparison weakens over time but is informative in the first two years.
- Matched cohorts. Where neither is possible, ventures can be compared with similar non-participants from the same registry — imperfect, because participants chose to apply, but far better than nothing if the limits are stated.
Where none of these is feasible, the honest word is contribution: the agency states what it did, what participants achieved, and why it believes the two are connected, without claiming the achievement as its own. Auditors accept contribution claims that are labelled as such. They do not accept attribution claims that are not.
A minimum viable KPI set for a national agency
The following set is designed to be small enough to keep, defined tightly enough to audit, and structured so that each column of the logic model is represented. It is a starting point; sector programs add specifics.
| KPI | Column | Definition | Horizon | Type |
|---|---|---|---|---|
| Qualified applicants per place | Output | Applicants clearing the quality threshold ÷ places | Per intake | Leading |
| Completion with evidence | Output | Share completing all milestone gates with verified artefacts | Program end | Leading |
| Customer validation rate | Output | Share of ventures with paying or piloting customers at exit | Program end | Leading |
| Operating at 12 / 24 / 36 months | Outcome | Venture active with a bank account, activity and team, verified | 12–36 months | Lagging |
| Revenue band at 24 months | Outcome | Self-reported band, sample-verified | 24 months | Lagging |
| Employment created | Outcome | Full-time equivalents on payroll, verified via statutory filings where possible | 24–36 months | Lagging |
| External validation | Outcome | Any independent capital, lending, procurement or partnership | 24 months | Lagging |
| Research translated | Outcome | Licences executed or spin-outs incorporated from supported research | 24–48 months | Lagging |
| Cost per operating venture | Efficiency | Program cost ÷ ventures operating at 24 months | 24 months | Lagging |
| Private capital crowded in | Impact (contribution) | Private capital raised by supported ventures per unit public spend | 36 months | Lagging |
| High-growth share | Impact (description) | Share of supported ventures meeting an OECD-style growth definition | 36–60 months | Lagging |
| Comparison-group difference | Attribution | Outcome difference versus lottery or matched comparison | 24–36 months | Evaluation |
Two of these deserve comment. Cost per operating venture is the number that most changes agency behaviour, because it punishes headcount inflation directly: a thousand founders trained cheaply with few survivors costs more per outcome than a hundred trained well. Private capital crowded in is the number that most persuades finance ministries, because it converts a spending line into a leverage ratio; it must be labelled as contribution, since capital would have found some of those ventures anyway.
Cohort accounting: borrow the venture capitalist’s ledger
Venture investors have solved the latency problem for their own purposes with vintage-year accounting: every fund is tracked by the year it started investing, and performance is compared only among funds of the same age. Agencies should do the same. Every intake is a cohort; every cohort has its own survival curve, revenue distribution and capital-raised total; and cohorts are compared at the same age, never in aggregate.
Cohort accounting has three virtues. It makes latency visible — a 2024 cohort simply has no 36-month data yet, and the dashboard says so. It exposes design changes — if the 2023 cohort’s 24-month survival is higher than 2021’s, something changed between them, and the agency can ask what. And it disciplines survivorship display: every venture is in exactly one cohort and stays there whether it thrived or closed. The practical requirement is a registry with founder consent to follow-up written into the intake agreement, a standard survey at fixed intervals, and sample verification against statutory data — company filings, tax registrations, payroll records — so that self-reports are trusted because they are checked.
A composite case: the dashboard with forty metrics and no answer
The following composite draws on patterns from several agency engagements; details are altered.
A national innovation agency in the region had, over six years, accumulated a forty-metric dashboard. It reported events, attendees, memoranda, funds launched, incubators opened, founders trained, mentors enrolled and media mentions — all rising. A new finance minister asked a single question: how many businesses supported by the agency were operating and paying salaries, and how much had that cost per business? Nobody could answer. The data to answer it had never been collected, because no metric on the dashboard had required following a venture beyond the day it left a program.
The rebuild took eighteen months and was mostly plumbing. The agency adopted a logic model and re-labelled every existing metric by column; two-thirds turned out to be activities. It fixed twelve KPIs with written definitions and horizons. It rebuilt its intake agreements so that every future participant consented to follow-up, and it ran a retrospective survey of past participants with sample verification against the company registry. It introduced cohort accounting and — for its flagship program, which was oversubscribed threefold — a transparent lottery among qualified applicants that created its first genuine comparison group. The first honest report showed a 24-month operating rate that was lower than anyone had assumed and a cost per operating venture that was higher. It also showed, for the first time, that the agency’s intensive programs outperformed its light-touch ones by a wide margin. The budget that followed was smaller in total and larger for the programs that worked.
What could go wrong
- Definitions drift. “Operating” means one thing in 2023 and another in 2025. Antidote: a written data dictionary, version-controlled, with changes disclosed.
- Follow-up collapses. Response rates fall to 30 per cent and the survivors respond. Antidote: consent at intake, small incentives, verification against administrative data, and reporting of response rates alongside results.
- Latency is ignored under pressure. A new leadership team demands outcomes from a six-month-old program. Antidote: horizons written into charters and briefed to every incoming decision-maker.
- Index targets. A ministry sets “top 30 in the Global Innovation Index” as a program goal. Antidote: explain what the index measures; set program targets in the outcome column.
- Measurement crowds out delivery. Founders spend more time reporting than building. Antidote: twelve KPIs, not forty; annual outcome surveys, not monthly.
- Attribution overclaimed. A press release credits the agency with every alumni success. Antidote: the contribution/attribution rule, applied by communications as well as analysts.
Questions ministers actually ask
“Can you give me one number?” Yes: operating ventures at 24 months per unit of program cost, by cohort. It is the closest thing this field has to a return on investment, and it is auditable.
“Why can’t we report results in the first year?” Because the results do not exist yet. What can be reported in year one are outputs that predict results — completion with evidence, customer validation, qualified demand — and the report should say which they are.
“Aren’t lotteries unfair?” Among applicants who have all cleared a quality bar, a lottery is fairer than a panel’s preference for polish. It also produces the only evidence that will ever let the program prove its worth.
“How do we compare ourselves with other countries?” Use the global indices for national position and for identifying weak pillars. Use program-level outcomes, not indices, to compare programs.
“What if the honest numbers are bad?” Then the agency learns which of its programs work and moves money towards them — which is the entire purpose of measurement, and the strongest argument an agency can make for its own continuation.
Methodology & data notes
The time-to-signal chart is an illustrative model based on typical venture development timelines; the ordering of indicators is the claim, not the specific months. Global Innovation Index ranks are as published in each annual edition by WIPO and its partners, and index methodology changes between editions mean small year-to-year movements should not be over-interpreted; readers should consult the current edition for the latest figures. The KPI set is a design proposal, not a standard, and should be adapted to sector and mandate. The composite case is assembled from multiple engagements with details altered. Companion articles cover program design (article 31), incubator and accelerator economics (article 37) and agency governance (article 39).
References & further reading
- WIPO — Global Innovation Index, annual editions and methodology
- Global Entrepreneurship Monitor — adult population survey data on entrepreneurial activity
- Ács, Z., Autio, E. & Szerb, L. (2014) — National Systems of Entrepreneurship: Measurement issues and policy implications, Research Policy
- Stam, E. (2015) — Entrepreneurial Ecosystems and Regional Policy: A Sympathetic Critique, European Planning Studies
- Spigel, B. (2017) — The Relational Organization of Entrepreneurial Ecosystems, Entrepreneurship Theory and Practice
- Ewing Marion Kauffman Foundation — research on measuring entrepreneurial ecosystems
- W.K. Kellogg Foundation — Logic Model Development Guide
- OECD — Oslo Manual (innovation measurement) and high-growth enterprise definitions
- Nesta — research on high-growth firms and innovation policy evaluation
- Startup Genome and StartupBlink — startup ecosystem rankings
- NITI Aayog — India Innovation Index
HexGn builds the measurement architecture into every program it designs for governments — definitions fixed at intake, cohort tracking to 36 months, and comparison groups wherever oversubscription allows — so that the report an agency publishes in year four is the one it can defend.