Databases Reference
In-Depth Information
CASE STUDY: STARTING AT THE WRONG END OF THE SPECTRUM
(continued)
If the work to pull only current data from the new system had been
developed first, the loads could have been running that entire time. After two
years, the organization would have the most recent two years of history with no
additional cost or effort.
Two lessons can be learned here:
■
Periodically evaluate the effort and resources being put into chasing
down data problems. Look at the work compared to the impact on
the business. Understand that the cost to research and correct data
problems may be more than what the potential business value, especially
if the data represents a very small percentage of the business.
■
Critically review the historical requirements to support business analysis
and determine the most direct and cost effective way to have that data
loaded.
■
Once the minimal historical requirement is met, decisions can be
made about the cost and value of loading more historical data.
Cleansing at the Source
When possible, the best solution to data quality problems is to address them
at the source or where the data is originally captured. The first step is to
ensure that research is done to determine the root cause of the problem. When
the problem is in the data warehouse, it can be corrected there. More often,
however, the root cause of data quality problems exists in the source systems
and/or business processes.
Addressing the problem at the original source can mean changes need
to be made to core application systems and/or business processes of the
organization. This would be a reasonable alternative if nothing else were
going on in the organization, but reality and experience show that this is often
not a realistic option. Usually, the application support team does indeed agree
with the recommended correction, but it may not be a priority when ranked
against other changes that may be needed just to keep the business running,
and it may not be implemented for 18 months! Changes to business processes
can encounter similar delays. You may find you can change a business process
more easily than changing the systems to capture the data, or you may find
you can change the systems faster than you can change the business process.
You need to understand all the options and take a pragmatic approach to the
problem.
The data warehouse and operational application teams must work together
to determine the impact of the change on the business, and how this may
