Information Technology Reference
In-Depth Information
Table 2.
Performance comparison among different methods
DataSet
Config.
P (%)
R (%)
F (%)
Dictionary-based (SecTag) 38.09 67.44 48.69
Dictionary-based(Training)
47.24 98.88 63.93
Dictionary-based(SecTag+Training) 46.05 93.65 61.74
CRF-based
Set2
(Develop. Set)
94.47
91.87
93.15
CRF-based + Layout Features
94.67 94.15 94.41
Dictionary-based (SecTag)
38.43
69.1
49.39
Dictionary-based(Training)
45.9
95
61.89
Test
Dictionary-based(SecTag+Training)
44.56
89.28
59.45
CRF-based
94.33
90.74
92.5
CRF-based + Layout Features
95.45
92.48
93.94
4
Discussion
As shown in the previous section, CRF-based methods evidently outperformed the
dictionary-based approach. The diversity of section heading names of EHRs is a key
issue that resulted in the huge performance gap between CRF- and dictionary-based
methods. A physician can combine any section heading names in an EHR to form a
new section, or insert any supplemental information in a section heading. For exam-
ple, the bold texts in the two sections “Meds (
confirmed with patient
)” and “DATA
(
08/25/61
):”. These cases cannot be dealt by the dictionary-based method, since there
are no dictionaries that can cover all of these variations. By contrast, the CRF model
is capable of identifying these names, because the sequential labelling formulation
can model the dependency between tokens. In addition, the results showed that the
inclusion of the terminology from SecTag resulted in a decreased precision. This is
caused by the various section headings of different granular levels within the SecTag
content. For example, it includes terms like “toenail exam” and “muscle tone exam”,
which usually does not belong to the topmost section headings.
On the other hand, the results of CRF-based section recognition are not completely
flawless. A comparison of the “CRF-based” configuration with the “CRF-
based+Layout feature” indicates that adding the layout feature enables the CRF model
to recognize section headings that did not appear in the training set. For example,
section headings such as “HCP/FAMILY CONTACT”, “INDICATIONS FOR TPN”,
“Allergies or adverse reactions” “Course on floor” and “Oncology CONSULTATION
NOTE” were not present in the training set, but with their clear layout, the model with
our layout features is able to recognize them.
Finally, this work conducted error analysis on the predicted results of our best con-
figuration “CRF-based+Layout features”, and categorized the errors into two catego-
ries: false negative (FN) and FP cases. The following subsections discuss them
respectively.
Search WWH ::
Custom Search