Databases Reference
In-Depth Information
create those structures and processes for itself. This would actually be a
reasonable cause to pause the creation of a Market Basket ETL application
until the construction of the data warehouse ETL Metadata structures
and processes is complete. The level of confidence in the completeness and
identity of the data within the Market Basket ETL application is in large
part based on the completeness and identity of the data in the data ware-
house ETL. If the data warehouse ETL is not able to guarantee the com-
pleteness and identity of its data, then the Market Basket ETL application
will most probably not be able to guarantee the completeness and identity
of the data it delivers to the Market Basket application.
Data Quality
Data quality is assessed by one of two methods: looking for something that
should be there, or looking for something that should not be there. The
“looking for something that should be there” method programmatically
calculates the interrelationships within the data (e.g., Summed detail rows
should equal header rows, units multiplied by unit price should equal total
price) and posts an alert when the expected condition has not occurred. The
“looking for something that should not be there” method programmatically
looks for data that is obviously wrong (e.g., negative unit price, null values,
alpha characters in a numeric field) and posts an alert when the unexpected
condition has occurred. As the data was extracted from the original source,
transformed in the ETL application, and then loaded into the data ware-
house, that data should have gone through a data quality assessment.
If the data did indeed go through a data quality assessment on its way
to the data warehouse, then the guarantees provided by those data quality
assessments extend to the data in the data warehouse and now to the data
coming from the data warehouse to the Market Basket application. Even
if that is true, the Market Basket ETL application may need to include
its own data quality assessments. The analysis of the data in the Market
Basket application may expose, or assume, data behaviors or data interre-
lationships that are not already guaranteed by the data warehouse ETL. If
that is the case, then the Market Basket ETL application will need to assess
the data from the data warehouse in the Market Basket ETL application.
If the data did not go through a d ata quality assessment on its way
into the data warehouse, then the Market Basket ETL must provide all
the guarantees for all the data that comes from the data warehouse to the
Market Basket application. Either way, the data quality methods are the
