Databases Reference
In-Depth Information
Incremental loads
Another important advantage of designing an appropriate data architecture, is the
fact that it eases the construction and maintenance of incremental load scenarios,
which are often required when dealing with large data volumes.
An incremental load is used to transfer data from one database to another efficiently
and avoid the unnecessary use of resources. For instance, suppose we update our
Base QVD Layer on a Monday morning, pulling all transactions from the source
system and storing the table into a QVD file. The next morning, we need to update
our Base QVD layer so that the final QlikView document contains the most recent
data, including transactions generated in the source system during the previous day
(after our last reload). In that case, we have two options:
1. Extract the source table in its entirety.
2. Extract only the new and/or modified transactions from the source table and
append those records to the ones we previously saved in our Base QVDs.
The second option is what we call an
Incremental Load
.
The following diagram depicts the process of an incremental load at a general level,
when a Base QVD Layer is used:
The process of performing an incremental load in QlikView varies in complexity
depending on the nature of the source table. At a general-level, we would approach
the task by following these steps:
1. We first query the source database using a
Where
clause with the appropriate
logic so that only
new or updated records
are extracted.
