Every publicly funded research organization eventually faces the same test. A renewal comes due, or a major new ask is being prepared, and someone has to show that the organization delivers value for the money invested in it.
Research funders typically say what they are buying, and many are explicit that talent development is one outcome they want, alongside the knowledge the research generates. For instance, the Strategic Science Fund (SSF), which has invested $860 million in 24 research organizations, names talent among its four objectives: to develop, attract and retain world-class research and innovation talent in scientific areas aligned with Canada’s priorities. Its eligibility criteria include “demonstrable impact.” This month, SSF-funded organizations are reaching the halfway point of their five-year funding agreements, which is a useful milestone to begin preparing the evidence of impact: if talent development will matter at renewal, the evidence base needs to be built well before the final year.
Talent appears this way in funding programs everywhere — federal, provincial, philanthropic. And nearly every research organization makes the corresponding claim: we train the highly skilled people this country needs.
The claim may be true, but it is hard to prove credibly. Most organizations rely on evidence that is too thin for a funder to evaluate: participation counts that show activity, and selected success stories that show possibility. But neither shows whether alumni outcomes are strong, unusual, or attributable to the organization’s work.
Which raises a question worth sitting with: when your next funding decision comes, what will you be able to show?
The evidence that answers it well takes time to assemble — considerably more time than most organizations expect.
Three questions worth preparing for
Put yourself in a reviewer’s chair. An organization claims that it trains the highly skilled people a country needs. What would you want to know before agreeing?
Three questions seem reasonable, and a thoughtful reviewer is likely to arrive at all three:
- How have your alumni performed in aggregate?
- How does that performance compare to a benchmark?
- Is the difference because of you?
The conventional answer to all three is the same: a count of how many people came through the programs, illustrated with a few alumni whose careers turned out remarkably well. It is a true answer, and an honest one. It simply doesn’t reach any of the questions.
A headcount measures outputs, not outcomes — it tells a reviewer how many people you trained, not what became of them. And a handful of standout stories is neither aggregate nor comparative. They may be compelling, even inspiring, but they are still selected cases. A reviewer knows this, which is why the stories tend to be received warmly and weighed lightly.
Question one: How have your alumni performed in aggregate?
The first question is the most basic, and it’s surprisingly hard to answer well.
Where do your alumni work now? How quickly have they advanced? Did they stay in the country — or the province, or the state — that funded their training? Did they go on to further degrees? What sectors did they end up in?
Most organizations cannot answer these questions about their own alumni in any systematic way. The information exists, but it is scattered across thousands of individual career records and has not been collected as a body. Surveys are a common tool, and they carry a structural weakness: they often reach the people most willing to reply, who may also be the people with good news to report. A 10% response rate is not automatically a representative 10% sample.
Answering the first question properly means observing outcomes across a large and representative share of your alumni, not just the ones who responded or the few who have remained in touch with you.
Now suppose you produce credible aggregate results. Suppose you can say that 82% of your alumni work in high-tech and research-intensive sectors, or that a quarter of them hold management or executive roles. What does that actually demonstrate?
On its own, not much. Which brings us to the second question.
Question two: How does that compare to a benchmark?
Nothing is impressive without knowing what is typical. A retention rate of 79% sounds strong — until you learn that comparable graduates in the same fields stay in-country at 77%. A finding that 15% of alumni reach executive roles means one thing if the norm is 4%, and something entirely different if the norm is 14%. Numbers presented without a reference point invite reviewers to supply their own, and reviewers may not be generous in that exercise.
This is what benchmarking is for. The idea is straightforward: identify a group of people who studied in the same fields, at the same levels, and during the same years, but who did not have access to your programs or facilities. Then apply the same data collection and analysis steps to that group as well.
Two further points are worth insisting on. The comparison group should be drawn from a genuinely comparable population, not an arbitrary national average. And it should be applied consistently across every measure you report, not only the ones where the gap turns out to be flattering.
Differences that emerge this way can be substantial, and they are differences a participant count would never have surfaced — they exist as findings only because there was something to compare against.
Question three: Is the difference because of you?
Here is where an honest account has to acknowledge a limit.
Suppose the benchmark shows a substantial gap. A sharp reviewer may still ask the hardest question: is this gap the result of what your organization did, or of who came through your doors in the first place?
It’s a fair challenge. Research organizations attract capable, ambitious, self-selecting people. Some of them would have done well anywhere. Matching a comparison group on field, level, and period narrows this concern considerably — it removes the most obvious alternative explanations — though it does not eliminate it entirely.
What helps is evidence of a different kind: the alumni themselves.
When you interview a sample of alumni about their careers, consistent themes may emerge. Alumni may trace professional discipline — precision, quality standards, planning for what can go wrong — back to the demanding requirements of the research they conducted. Others may credit early exposure to managing teams, communicating across specialties, and understanding how complex organizations function for their later moves into leadership. When the same mechanisms recur across a structured sample of interviews rather than appearing once in a memorable anecdote, they become evidence a reviewer can weigh.
This is attribution, not proof of causation, and the distinction matters enough to warrant a separate post. But the pairing is what makes the case stronger than either method alone: the benchmark shows whether alumni outcomes are observably different, and the interviews help explain whether the organization plausibly contributed to that difference.
If you want to attempt this in-house
Plenty of organizations can make a serious start on this themselves. If you’re considering it, here is roughly how the work goes and where it tends to get difficult.
Start with gathering your alumni lists. You need names, the institution they were affiliated with, their degree level, and the year they were associated with you. If you don’t have a list, one can often be reconstructed from publications, program records, and public sources — this takes longer than people expect, but it’s tractable; I’ve done it.
Aim for scale. Not every alumnus posts useful career information online. For that reason, start with a list of at least a few hundred alumni if you can, so the traced records you find are large enough to support meaningful patterns rather than isolated observations. If you start with a list of around 1,000 or more, you can usually segment by field, degree level, and era with much more confidence.
Check your sample against your population. Whatever share of alumni you manage to trace, compare its composition — degree level, field, gender, era — against your full list. If your traced sample over-represents one group, say so in the reporting. This single step might do more for your credibility than any individual finding.
Report the measures you committed to. Including the ones where you don’t come out ahead. A report showing three strong results and one neutral one is more persuasive than four strong results, because it demonstrates the method wasn’t tuned.
Plan for continuity. A common failure is over-reliance on short-term student staff to do the leg work of the analysis; then that person moves on and takes the method with them. Document the process as you go, so your next study builds on this one rather than restarting.
Mind the data rules. Career information should be gathered in ways that respect privacy law and the terms of the platforms it comes from (see some tips in our earlier post). Involve your privacy office early rather than late; if a privacy impact assessment is needed, it takes time you should budget for.
None of this is beyond a capable in-house team. It is, however, more work than it looks, and the pieces that most often defeat it are time to be invested, benchmark construction, and continuity between studies.
What this means before your next renewal
It’s worth asking what your current impact reporting would look like under these three questions. If your answer to the first is a headcount, and you have no answer at all to the second and third, you’re in the same position as many research organizations — which is precisely the opportunity, because a reviewer comparing applications is looking for whatever distinguishes one from another.
The practical difficulty is timing. Evidence of this kind takes months to assemble, not weeks, and that’s before accounting for data gathering, privacy review, and procurement. Organizations that begin when the renewal is already on the calendar may discover that the window has closed.
Which is why the midpoint of a multi-year agreement — not its final year — is when the evidence base is best built. There is time to do it properly, and time to use what you learn.
The larger point is not that every organization needs a full alumni impact study tomorrow. It is that credible talent evidence has a long lead time. If talent development is part of the value you claim, the renewal year is too late to discover what you can and cannot show.
This is the kind of work we do at Alumni Analyzer, and we’re happy to share our experience. If you’re thinking through talent development impact metrics for your own organization, I’m glad to talk it over — whether or not it leads anywhere.