Background Image
Show me

Lies, damned lies...and defective data

Helen Mayson

Data powers many processes in an organisation

Data might sound dull, but it’s more important than ever in a fast-moving market, says Nigel Turner, vice president of Information Management Strategy, Trillium Software

The days when maverick leaders relied on gut decision making and thinking on the fly are numbered.  Like it or not, today’s decisions and key business processes are fuelled by data. According to a recent Economist Intelligence Unit report commissioned by Capgemini, some 75% of business leaders surveyed believe their organisations are now data-driven.

As decisions and processes are now so data dependent, then that data had better be good - accurate, complete, consistent and accessible. Defective data results in wrong decisions, poor economic performance and potential penalties from regulators.

So how do you ensure that you have accurate data to support and feed your business? In a world of Big Data, information is pouring into the organisation in multiple formats from a whole range of sources. Yet when it comes to ensuring its veracity, many organisations have a big problem with Big Data.

According to various analyst reports and management surveys:

  • 10 - 30% of the average company’s potential turnover is lost to problems caused by poor data. What might it be costing you?
  • 85% of leaders experience issues due to their inability to analyse and act on data in real-time. Ring any bells?
  • 50% or so of companies are not fully satisfied with their current CRM programmes, often due to a lack of data integrity. Can you relate to that?
  • 65% of corporate fixed asset data is incomplete, inaccurate, or altogether missing.

According to one distinguished analyst group, 33% of organisations rate the quality of data in their organization as ‘poor at best,’ while according to Aberdeen Group, even in ‘best in class organisations’ only 48% of firms are satisfied with their data quality.

Perhaps one of the reasons poor data costs organisations so much money is that all too often nobody’s responsible for ensuring key data is fit for business purpose. Business managers sometimes think of data as a technical problem and dump it on the IT department to resolve. But this is a mistake as data is a business asset, generated by business operations. Those in IT often feel their own competencies are to capture, store and secure it, and make it accessible; they cannot alone control its quality.

Slowly, business leaders are realising their role in data management and data quality.  They are recognising that it’s business managers who need to ‘own’ the data and business managers who must bring passion and determination to improving and managing its quality. Brought together, business managers can talk about what data really matters and where the real challenges are, while IT can advise on the systems and solutions to help support improvements.

The key to ensuring good data is to create a data quality improvement and compliance process. This means ensuring that the data entering corporate systems and processes meets required standards for cleanliness, relevance and timeliness. There are three main data compliance steps:-

  1. Assess the quality of existing data and its degree of reliability and consistency for the business processes it supports.  Get some quantified insight into the quality of your data.  To do this, data profiling approaches enable you to fully understand the issues in your data and to determine what steps need to be taken to remedy them.  Specialist data quality software automates this process, enabling you to incorporate your own data validation rules, both for quality and for its relevance to your specific business needs.
  2. The appropriate data quality software should also convert these rules into processes that transform and correct the data into a common format and content. A standardised and corrected piece of data ensures it will match with associated data coming via other channels and with legacy systems of data collection. This ensures, for example, that associated customer, financial, product, and historical data is linked correctly as well as with any external data appends.
  3. Finally, the same repeatable process created for step 2 can also be embedded into systems to automate the validation and correction of data at point of capture. Business users will then all have a high level of data consistency, quality and reliability serving their specific requirements.

Overall, coherent and integrated attempts to improve data quality in organisations could reap massive rewards. At British Telecom, an enterprise data quality improvement programme delivered benefits in excess of £600 million over 10 years.

In the age of data, and now of Big Data, senior executives should care very much about data quality. The first action to take if you feel your key data might not be all it should be, is a data quality assessment. Discover the truth. No more lies, damned lies or defective data.

About Nigel Turner
Nigel Turner is Vice President of Information Management Strategy at Trillium Software where he is helping current and potential Trillium Software clients start, expand and accelerate their enterprise data quality initiatives.  He spent much of his career at British Telecommunications plc (BT) where he led an internal enterprise wide data quality improvement programme that was praised by Gartner, Forrester, Ovum Butler and others both for its approach and proven benefits.  Nigel is also a part time lecturer at Cardiff University where he teaches data management. He can be reached via or @trilliumsw


Add a comment