Information Technology Reference
In-Depth Information
of judgment is frequent in off-the-shelf implementation projects as well
as platform migration projects.
Data enters a system in many ways. It often comes in through appli-
cations that do adequate validation, hopefully. However, when data import
utilities are used to import data directly into the system, the security and
validation logic is bypassed. This happens because sometimes the need
for import utilities is not recognized until it is too late and one does not
have time to build the appropriate import utilities. Sometimes the validation
must be skipped because the data volumes are high. This approach ignores
the security and validation logic, which would otherwise have been
invoked had the data been entered using the data entry screens, or system
APIs (application programming interfaces).
Even if data fields in the two systems match, there may be problems
with duplicate entries (acceptable in the source system but not the
destination due to a different indexing scheme) or mandatory fields. The
latter is a frequent cause of problems. When data is imported at the
database level, optional fields in the source system can be mapped to
fields that are considered mandatory for some business logic in the
destination. Because the data is supposed to have entered through the
screens or APIs (where NULLs would not have been permitted), the
business logic assumes it will always get a non-NULL value. This results
in bad data getting in and a possible system failure.
Sometimes utilities provided with the database are used to handle bulk
loads. Many of these utilities are data movers designed for speed or
throughput. They rarely allow the expression of any validation or security-
related logic. With some tools, clean-up can be a big issue if the import
job terminates abnormally.
By the same logic that imports are important, exports from a system
can be critical for the proper functioning of other systems (Figure A.4).
Files are often used for communicating data across processes. This
method is common in legacy and mainframe systems where one job
outputs a file for another job. Such outputs need not be called exports
because the job is expected to transform an input file to an output file.
The term “export” carries a connotation of being an extra facility, beyond
the main features of the application. The work required to create a simple
comma separated (csv) output should not be underestimated.
Do not assume that off-the-shelf software that has an export facility
will meet one's data integration requirements. While planning to move
data from one process to another, one needs to verify that the destination
process has import facilities and that it can import data in the for mat
exported by the source process.
Going across boundaries is not easy. Both the format and the content
must match. One system's purchase order number may not directly map