Intelligence by design

Today’s data management challenges

With sources and volumes of business-critical data only likely to increase in future, many large organizations have embarked on enterprise-wide data warehouse (DWH) initiatives to create a pool of core historical data to be drawn on by a variety of business users.

But creating an effective data warehouse is considerably more complex than simply channeling data from live systems into a catch-all database for future analysis.

Key challenges include:

• Extraction: Market, transaction and static data from multiple sources must be extracted securely, consistently and comprehensively;
• Accuracy: Data must be cleansed to ensure all errors are identified and addressed before being uploaded to the DWH;
• Performance: Large quantities of data must be uploaded regularly, typically daily, to the data warehouse from live systems and extracted within tight time windows in accordance with end-user needs, which may change at short notice;
• Scalability: DWH projects must be sufficiently robust and flexible to handle future business needs and new as-yet-unspecified data sources and file formats;
• Flexibility: User-friendly, responsive interfaces are required to enable new reports and analyses to be generated, tested and run at short notice, as needs evolve;
• Consistency: Data from multiple sources should be normalized for storage and retrieval purposes, using standardized definitions across sources;
• Synchronization: Data from live systems must be accurately reflected in the DWH, even when backdated changes are made.

Many DWH projects have run into difficulties and have been shelved. Organizations that under-estimate the scope and scale of a DWH implementation project risk wasting valuable investment dollars and internal resources on a facility that is incapable of producing reliable data for business users. Slow delivery of inaccurate data at high cost quickly adds up to a ‘white elephant’ IT project, but perhaps more important is the opportunity cost and the risk of falling behind peers and competitors in terms of analysis and intelligence.