The case for Open Standards

The following fleshes out the argument for a software procurement open standards mandate in NZ and provides more examples.

Open Standards: the rule, not the exception

When our government hires a contractor to build a bridge, they choose from among suppliers who can demonstrate their ability to meet vendor-neutral, royalty-free open standards for engineering quality (strength, resilience) and safety (for workers and public). Any supplier not able to meet such independent standards is not in the running. This is great, as it ensures a minimum objective quality threshold for bridges, and a transparent set of expectations those competing to offer such services to the government can strive to meet: a level playing field.

Except for Software Suppliers

Unfortunately, with software procurement, suppliers have no such requirements. When software suppliers respond to tenders, they are allowed to define their own "proprietary" standards, which are unique to them. By allowing this, the government gives suppliers effective local monopolies: by definition, the winning supplier is the only one who can meet their standards, because they're the ones making them up or changing them at their whim.

Examples of these proprietary standards include network communication protocols, application programmer interfaces (APIs), and file formats. Because these are arcane to anyone who is not an IT specialist, few in government procurement comprehend the nuances of proprietary standards, or their implications.

Implications of Proprietary Standards

This implications of proprietary standards are profound:

1. competing suppliers cannot develop compatible software because
   •  the original vendor does not document them,
   • the cost of reverse engineering them is huge, and is actually illegal in some jurisdictions,
   • the original vendor can change them at their whim, unilaterally ruining any compatibility efforts.
2. the more a government customer has invested in a proprietary solution, the more it will cost to shift to an alternative solution. This is called "lock-in". The perceived "switching cost" to a different supplier's solution are typically wrongly associated by procurement agents with the cost of the new solution rather than the deliberate lock-in created by the incumbent supplier.
3. having chosen a proprietary solution for one aspect of business, procurement agents often explicitly require compliance with that solution's proprietary standard for other solutions, effectively creating an expanding barrier to competing suppliers (most of which are also proprietary) and granting an expanding monopoly to the incumbent supplier.
4. those interacting with government - like businesses, NGOs, and citizens - are effectively forced to adhere to the same proprietary standards to effectively exchange information with government, especially for compliance purposes.
5. over time, proprietary suppliers grab increasingly broad monopolies in government, and other suppliers (large or small, foreign or domestic) are simply unable to compete. This practice has resulted in the largest, most powerful business entities in the history of humanity.

With a single policy decision, however, government could reverse this very undesirable situation, and create a level playing field for software suppliers: by mandating that software comply with relevant open standards.

Competition requires Compatibility

Everyone knows that to compete in the software industry, you need to offer full compatibility with the market leaders. Unfortunately, when the market leader succeeds in embedding proprietary standards in its products, it can thwart any attempts by competitors to offer compatibility, and therefore to compete.

The only viable means for facilitating competition in software is to ensure that the "interfaces" with which people and data interact with software are built to conform with open standards. Although conforming with open standards does not guarantee practical compatibility, it provides an objective metric for judging where problems with compatibility lie, because compliance with an open standard can be assessed by independent parties.

Open, Vendor-Neutral, Royalty-Free Standards exist

In most cases, software suppliers pushing their own proprietary standards will make arguments like "we are such innovators that open standards simply hold us back". To a point, this is true, however this is not a valid reason to accept proprietary standards. It would be equivalent to a bridge builder rejecting compliance with engineering and safety open standards because doing so would disallow their use of untested materials and practices that might endanger workers or the general public. Example might be the widespread use of asbestos or dodgy structural steel...

Many open standards already exist in the software industry, in almost every niche. Here are a couple examples.

Public Transport Ticketing

Consider public transport in NZ. Every region has a public transport system, and in each case, offers residents a transport card (ATHop, BUSIT, Smartride, Snapper, Metrocard, GoCard, etc.) which links them to a proprietary ticking system, each provided by different suppliers. No two ticketing systems are compatible between regions. None. This is a massive inefficiency, and a tremendous barrier to tourists and domestic travelers alike to might want to use public transport around the country. How could governments tendering for these systems been so short sighted?

One way to fix this would be to replace all the various small supplier solutions with a single supplier's solution. That is what the NZTA seems inclined to do. That,  however, is a "winner-take-all" scenario, where the government picks the winner, granting them a national monopoly, and the results in the demise of all the other suppliers. In the longer term, it will be a costly, unfair, monocultural disaster for everyone involved.

The other solution is to adopt an open standard for ticketing, like this one. In other parts of the world, public transport ticketing systems do this, and travel cards can be used throughout entire countries.

Productivity Software

Perhaps the most pervasive proprietary standard we use every day is the set of pseudo-open standards promoted by the Microsoft Corporation. For over a decade, the Microsoft Corporation has enjoyed an effective monopoly in software used by government. It has extracted monoply rents at the expense of the taxpayer and to the detriment of NZ's domestic software industry.

Everyone knows that files ending in DOCX, XLSX, and PPTX are opened with Microsoft Office. People create many of those documents every day. Some competing software claims to offer compatibility, although it is never quite perfect. That is because the MS "open standard" for its formats is not open in a useful sense: it allows the Microsoft Corporation to make changes to the standard unilaterally. This undermines the compatibility efforts made by would be competitors.

This can be remedied rapidly and largely transparently to users by the government mandating, instead, that government IT shift to using Open Document Formats (ODF) by default. ODF files conform to a properly vendor neutral open standard, which no single vendor can change at a whim - in fact there's a just, supplier-neutral process by which changes can be made, and ironically, Microsoft is a member of the OASIS group that started the ODF standard. Although government employees could continue using MS Office - which supports loading and saving ODF formats (but never by default) - this move would open the door to competitive software procurement in future, and a "sinking lid" on the procurement of software depending on proprietary standards.

Financial Implications

It is difficult to speculate on the cost savings that mandated open standards compliance would offer due mostly to the fact that, on the grounds of "commercial sensitivity", the government will not provide any account of its software expenditure for specific software products and suppliers.

We can guess that the government spends, for example, $100 million per year on its Microsoft Select Agreement, which is purely for proprietary software license fees, not associated support or other costs. A shift to open standards would immediately make it possible to replace MS licenses with open source software with no license fees, but similar support costs. So that's a minimum of $100 million taxpayer money saved per year, corresponding to just one proprietary software vendor. It's possible that the full proprietary software license bill for NZ government is an order of magnitude greater than this with all the specialised "enterprise", and custom-developed, proprietary software in use, meaning a cost of more like $1 billion for taxpayers yearly.

With open standards, far more services could be provided by NZ owned software companies, meaning that NZ would receive far more tax revenue, perhaps reducing the cost by the roughly 30% companies (or employees) pay in tax. Moreover, much of the support could be distributed across a broader range of companies creating competitive pressures which would drive prices down: proprietary software suppliers tend to charge more for their services because of their monopoly control of the software and, for example, support provider training and "registered partner"-type relationships.

There is also the ripple effect - if companies and NGOs in NZ were also able to stop paying monopoly rents for software licenses (or if they adopted open source, pay nothing for licenses), while keeping support costs equal, we would probably see at least an additional $1 to $5 billion in savings annually in our economy. This would free up resources that could be invested in real innovation and improving competitive advantages relative to the rest of the world.

If we do not take the step, we will be disadvantaged compared to those other economies, internationally, who do.

Correcting past mistakes

Yes, a shift to mandating open standards for software has costs (although they're probably far lower than most would think), and there are complex "edge cases", but the alternative - persevering with the status quo dominated by proprietary standards, awarding effective monopolies to incumbent suppliers - is far far worse. It amounts to digging ourselves an ever-deeper hole. There are definitely low-hanging fruit that would have a massive impact very quickly, and at very low cost.

Other governments, notably the UK and EU, and individual agencies in many countries, have already realised this. In 2014, the UK made the decision to mandate open standards for software procurement, and they are already enjoying substantial savings, as well as increased participation by domestic suppliers who, in turn, parley those successes into greater export opportunities. Their lead in this area should embolden our government, as they have already demonstrated that it is possible, and both ethically and economically better for all of New Zealand. After all, we have always strived to be an egalitarian society, championing a level playing field for all.

D5 Commitment

In fact, we're already committed to open standards - in 2014, Minister Peter Dunne signed NZ up for the "D5 Charter" which among other good things commits us to adopting open standards. It'd be great to see how NZ is tracking on meeting its commitments as a member of the "Digital 5" which includes the UK as well as Estonia, Israel, and South Korea. To date, the Minister has been "enigmatic" when asked about progress.