Part 2 of a series on toxic technology. In part one I introduced toxic technology, and defined what I mean by the term.
The characteristics of toxic technology can be invisible to people working in organisations. It is hard to directly observe the security flaws, tangled architecture or unreadable code without expertise. But the symptoms of toxic technology are common and visible.
Delivery
Toxic technology makes delivery slower, harder and riskier. This is caused by shortcomings in the the less visible design of technology: the code and architecture. This internal design has a significant effect on the ease and risk of change. Some change can be simple, like adding lego blocks to an already assembled structure. But some change require the existing structure to be pulled apart, or new kinds of lego block fabricated. The most complex changes ripple out into connected systems. Toxicity makes even the most simple changes surprisingly complicated.
There’s no perfect internal design for technology – it’s a balancing act between preparing for the future vs. reacting to immediate needs. Preparing involves making technology more malleable and understandable (known as refactoring), and therefore ready for likely future changes. But teams can be wasteful if they over-prepare, investing in changes to anticipate something that never happens. When the pressure to deliver is high, teams refactor less. Failing to refactor over a long time will make technology brittle and hard to understand – change becomes harder and harder to achieve. If the pain of changing technology is too high, it will become perceived as ‘legacy’.
Where legacy technology exists, its impact on delivery can be hard to avoid. If legacy systems hold data, or process transactions which are important to the organisation, then they become bottlenecks for delivery. It can be very challenging to replace legacy systems, so it is common for new technology to layer on top of the old, accumulating over time. Teams must be wary of the infectious nature of toxic technology – when new technology integrates with legacy technology, some changes can only move at the pace of the slowest system.
Operations
The ease of changing technology is also important for its operation – it improves the ability to manage failure. Teams need to be able to release fixes with confidence, knowing they’re unlikely to cause side effects. To help detect and diagnose failure operators sometimes need to add new ways to observe how the system is behaving. Toxic technology increases the risks of accidently making things worse when trying to deal with an emergency.
Toxic technology makes systems brittle. If changes cause unwanted side effects, small incidents can grow into catastrophic failures. Legacy systems will often grow a reputation for stability, but this reputation is a result of infrequent change. When incidents do occur, it takes a long time to recover and the impact on the organisation and users is more severe.
The organisation
A catastrophic IT outage is a big threat to organisations. It creates an existential threat to commercial organisations, and can cause the loss of trust in a public institution. In 2019 British Airways and TSB experienced outages which impacted customers over several days. By lasting for longer periods of time, compound effects are seen – customers will shop elsewhere and public service users could see impacts on their lives. Modern technology companies, currently less burdened by toxic technology have outages which usually only last a few minutes or hours. Sometimes the press coverage of these is high-profile, but compound effects are avoided as business returns to normal. However, in the public and banking sectors, the loss of critical systems for several days is becoming increasingly normal.
Communications around these kinds of persistent outages state, or imply, that there is a single root cause. The truth is that any major technology failure will have multiple complex causes: a succession of technical and organisational failures, alongside human error and bad luck. But organisations are unlikely to reveal these complex causes publicly, because it begins to expose the depth of toxic technology the organisation relies upon. Where they have accumulated toxic technology at scale, small failures can cascade into catastrophes.
Toxic technology is a cyber security risk. Neglect means that technology is not kept up-to-date, and vulnerabilities emerge. Pressure on delivery means good security practices are routinely deprioritised. Meanwhile, cyber crime is increasing year-by-year, in terms of both frequency and sophistication. Insufficient cyber security, caused by toxic technology, will someday result in the failure of a major corporation or institution.
Challenging new legislation like GDPR is beginning to take effect, with British Airways fined £183million in July 2019 for a data breach. Their rivals Easyjet are being investigated for the unauthorised access of 9 million customers’ personal data. Most organisations being fined so far are established organisations with high volumes of toxic technology. Legislation of technology is expanding further – including accessibility, online platforms, and encryption – meaning toxic technology is also becoming a legal concern.
Where technology is subject to standards, the impact of toxic technology can be critical to doing business. Toxic technology is very likely to breach standards through poor security, or a failure to keep up with changing requirements. PCI-DSS is a standard used by the payment cards industry to protect customers’ financial information. Failing to meet the standard can result in legal action, cost of fraud, loss of revenue and range of other negative impacts.
Cost
Toxic technology places a financial burden on organisations. It makes it harder to be strategic and innovative because resources drain away on tactical or emergency manoeuvres. Only the highest priority work can be done, because the cost and risk of change has risen so high.
Toxic technology can get stuck – becoming both too hard to change and too costly to replace. But even if remaining unchanged, costs increase because of a changing context. Most technology is connected with networks, platforms, APIs and ad-hoc user integrations. Whenever external systems change, there are costs to re-integrating the technology. New security vulnerabilities mean new investment in cyber defences. Changing supporting platforms can trigger expensive migrations. People costs increase as access to niche skills and knowledge becomes harder. Commercial contract renewals are renegotiated at higher prices to support ageing technology. And throughout this process of toxic technology growing in cost, modern replacements tend to become cheaper, but remain out of reach due to the cost of switching.
The simplest way for toxic technology to impact an organisation’s finances, is by buying or building something that’s not needed. The technology industry is full of ‘silver bullet’ solutions to complex problems, particularly in domains like data warehousing and cyber security. Faced with a complex public health and economic challenge like Covid-19, governments across the world have responded by spending significant sums on apps. Whilst these apps sound intuitively useful, and are very annouceable for political leaders, there’s no evidence to show they’re effective. If they’re not scrapped entirely, governments are left with complex sustainability and data privacy challenges.
Users
Pain and frustration are not uncommon to experience when using technology. But design is even more challenging when the medium is brittle and slow-to-change. Technology provides its most painful experiences inside large slow-moving institutions, where toxicity has accumulated over decades. Inside these organisations, staff are mandated to use dire technology in order to go about their daily duties. In August 2019, Dr Dominic Pimenta described his experience as a junior doctor in the UK’s National Health Service:
His experiences are very typical for public servants and administrative staff in large, established organisations across the world.
It is common for users of toxic systems to increase their usability with a layer of spreadsheets and paper-based work-arounds. Whilst this can optimise use for long-term users, it makes it harder for new users to learn. As new staff enter the workforce with the raised expectations of internet-era services, they will be less tolerant of technology which is frustrating and confusing to use. The impact of poor user experience is likely to hit certain groups more than others, with the effect of excluding those with permanent, temporary or situational disabilities.
Service outages, caused by neglect of technology, can erode trust with users, and prevent them from meeting their needs. The impact can range from the inconvenient to the life-threatening. Outages to services such as medical advice, security monitoring, housing, and access to money could have enormous impacts on the lives of users.
Toxic technology is also more vulnerable to cyber attack, which can have a significant impact on users, such as the suicides following the Ashley Maddison breach, or the leak of the HIV status of 14,000 individuals in Singapore.
Institutions such as governments, or monopoly service providers present a big risk to their users if toxic technology accumulates. When the quality of the user experience diminishes, there is no choice of alternative, and users must suffer the consequences.
The symptoms are everywhere
These symptoms are common in larger, more stable organisations. But their causes are systemic, and so start ups and high growth tech companies are not immune. Without bold new approaches to building and sustaining technology, supported by changes to how we fund, staff and govern teams, the outcome is the same: the accumulation of complexity and toxicity.
These systemic causes, and ways to mitigate them are the subject of future instalments.
Continue reading part 3.
Sign up for future instalments
Sign up for email updates below.
Dave Rogers 