The Evidence the Portfolio Demands and Ignores
Organisations do not reject data because they lack it — they reject it because the data contradicts a decision that has already been made on other grounds entirely.
The Promise That Never Quite Lands
For as long as I have worked in and around complex organisations, the aspiration has remained constant: decisions should be driven by data. Investment cases should be evidence-based. Portfolio governance should allocate resources rationally, directing capital and capacity to the initiatives that offer the strongest returns and withdrawing them from those that do not. The tools have improved enormously. The data available to decision-makers today would have been unimaginable twenty years ago. And yet the gap between the aspiration and the reality has, if anything, widened.
This essay is an attempt to understand why. Not as a critique of any particular organisation or governance framework, but as a reflection on a pattern that has persisted across sectors, through multiple generations of management thinking, and despite successive waves of technology that promised to make the rational organisation finally achievable.
The Longer View
The idea of data-driven decision making is not new. Its roots run through operations research, management science, and the quantitative revolution in business strategy that began in the mid-twentieth century. Each era has brought its own version of the promise: management information systems in the nineteen-seventies, executive information systems in the eighties, business intelligence in the nineties, analytics platforms in the two-thousands, and most recently the surge of interest in artificial intelligence and machine learning as instruments of organisational decision-making.
Each wave has delivered genuine capability. Organisations today can model scenarios, track performance, and visualise complexity in ways that were technically impossible a generation ago. The challenge was never the technology. It was the assumption embedded in every wave: that better information would produce better decisions, because decisions are fundamentally information-processing problems.
The longer view reveals that this assumption is wrong — not because information is irrelevant, but because it mistakes the nature of organisational decision-making. Decisions in complex organisations are not primarily acts of analysis. They are acts of power, negotiation, identity, and institutional survival. Data enters this arena not as the arbiter it claims to be, but as one more instrument in a contest that operates on entirely different rules.
How Portfolio Decisions Are Actually Made
The formal model of portfolio governance is well established. Initiatives are assessed against strategic criteria. Business cases quantify expected returns. A portfolio board evaluates the competing claims on resources and selects the combination that maximises value for acceptable risk. The process is logical, structured, and comprehensively documented in methodology guides.
The reality, as any practitioner who has sat in these rooms can attest, follows a different logic. The pattern I have observed is consistent enough to describe with some confidence:
The pre-decided initiative. A significant proportion of the initiatives that arrive at a portfolio board have already been decided. A senior leader has committed to them, politically or publicly. The business case is constructed not to inform a decision but to justify one that has already been made. The portfolio board’s role is to approve, not to evaluate. The data in the business case is real, but its function is rhetorical.
The orphan initiative. Conversely, initiatives without a powerful sponsor struggle to secure funding regardless of their analytical merit. A well-evidenced proposal from a business unit without political weight will be deferred, deprioritised, or referred for further analysis — a polite mechanism for killing proposals without the discomfort of an explicit rejection.
The sacred programme. Every portfolio contains initiatives that are effectively immune from challenge. They may be legacy commitments, regulatory imperatives, or programmes so closely identified with a senior leader’s reputation that questioning their continuation is career-limiting. The portfolio board allocates resources around these fixed points, optimising only in the space that remains.
The consensus default. When genuine uncertainty exists and no strong political force tips the balance, portfolio boards default to the path of least organisational disruption. This usually means continuing existing commitments and distributing new investment thinly across competing claims, producing a portfolio that satisfies no strategic priority fully but offends no stakeholder group fatally.
None of this is irrational in the colloquial sense. Each of these behaviours is a rational response to the incentives, power structures, and cultural norms that govern organisational life. But they are not the rationality that data-driven decision making assumes.
Why Data Fails to Displace Politics
The persistence of this gap demands explanation. If the data were genuinely compelling, would it not eventually overcome the political dynamics? The evidence of two decades suggests not, and the reasons are structural rather than incidental.
The Measurement Problem
Portfolio decisions require comparing unlike things. A digital transformation programme, a regulatory compliance initiative, and a market expansion investment are not commensurable in any straightforward way. The business case methodology forces them into a common framework — typically net present value or return on investment — but the apparent precision of the numbers conceals enormous uncertainty and the exercise of judgement at every stage of the calculation.
The people who construct business cases know this. The people who evaluate them know it too. The result is a shared understanding that the numbers are indicative rather than definitive, which in turn means that the numbers alone cannot resolve a contest between competing priorities. They narrow the field, but the final determination is made on other grounds.
The Accountability Asymmetry
Data-driven decision making implies that decisions should be revisited when the data changes. If an initiative is underperforming its business case, the rational response is to reduce investment or cancel it. But accountability in organisations is asymmetric: the cost of cancelling a visible initiative is immediate and personal, while the cost of continuing a failing one is diffuse and deferred. The leader who cancels bears the political consequences; the organisation that continues absorbs the financial ones gradually, across budgets and reporting periods.
This asymmetry means that data pointing towards disinvestment is systematically discounted. Not because it is disbelieved, but because acting on it carries costs that the data does not capture.
The Identity Problem
Organisations do not merely allocate resources. They express identity through their investment choices. A technology company invests in innovation not only because the returns justify it but because that is what a technology company does. A public sector body maintains universal service commitments not because the cost-benefit analysis supports them but because universality is part of its institutional identity.
When data challenges an investment that is tied to organisational identity, the data loses. Not because anyone disputes the analysis, but because the decision is not really about the analysis. It is about what the organisation believes itself to be, and no spreadsheet can resolve a question of institutional self-understanding.
The Generative AI Distraction
The recent enthusiasm for generative AI in organisational decision-making deserves a brief note, because it represents the latest iteration of the same foundational error. The argument runs: if human decision-making is biased by politics and psychology, perhaps machine intelligence can provide the objective analysis that humans cannot.
The error is familiar. It assumes that the obstacle to rational decision-making is analytical capacity, when the obstacle is actually the political and cultural context in which analysis is received and acted upon. An AI system that recommends cancelling a programme sponsored by the chief executive will encounter exactly the same institutional resistance as a human analyst making the same recommendation. The quality of the analysis was never the binding constraint.
What the Myth Costs
The myth of the rational organisation is not harmless. It imposes real costs, though they are largely invisible because they are built into the way organisations operate.
It wastes analytical effort. Organisations invest heavily in analytics capability, business case methodology, and portfolio management tools. When the outputs of these investments are systematically overridden by political dynamics, the investment is not wasted entirely — it provides a useful discipline and a common language — but its value is a fraction of what was promised.
It obscures the real decision-making process. By maintaining the fiction that decisions are data-driven, organisations make it harder to address the actual dynamics that determine outcomes. The political negotiation, the stakeholder management, the coalition-building — these are the real mechanisms of portfolio governance, but they operate in shadow because the official narrative does not acknowledge them.
It erodes trust. When people within the organisation can see that decisions are not being made the way the governance process claims, trust in governance erodes. The business case process comes to be seen as theatre — a necessary performance with no bearing on outcomes. This cynicism, once established, is extremely difficult to reverse.
Living With the Gap
I am not arguing that data and analysis are valueless in portfolio governance. They are essential. A portfolio board without analytical rigour is simply a political negotiation with no common factual basis, and that is worse. The data disciplines — business cases, benefits tracking, portfolio analytics — provide the evidential foundation without which governance becomes pure assertion.
What I am arguing is that we should be honest about the limits of this foundation. Data informs portfolio decisions. It constrains them. It provides a language for discussing them. But it does not and cannot determine them, because determination requires resolving contests of priority, power, and institutional identity that data alone cannot adjudicate.
The mature portfolio governance organisation is not the one that has finally achieved data-driven decision making. It is the one that has stopped pretending this is achievable and has instead built governance processes that explicitly acknowledge the political dimension of investment decisions — that create structured space for the negotiation that is happening anyway, while using data to ensure the negotiation is at least anchored in a shared understanding of reality.
This is a less inspiring vision than the rational organisation. It offers no transformative promise and no technological silver bullet. But it has the considerable advantage of describing the world as it actually is, and in my experience, that is the only foundation on which durable governance improvement can be built.