Textual Signatures: Identifying Text-Types Using Latent Semantic Analysis to Measure the Cohesion of Text Structures - Natural Language Processing and Text Mining - page 114

7.11 Experiment 3

In Experiment 1, we showed that LSA may be able to provide a textual signature
based on the relationships between the abstract of the paper and the sections within
the paper. We will refer to this kind of signature as type same paper (SP). In Ex-
periment 2 we showed that LSA can also produce prototypical signatures indicative
of articles of a similar theme. We will refer to this kind of signature as type same
theme (ST). In Experiment 3, we show that LSA based signatures can also indicate
papers that are differently themed. We will refer to this kind of signature as type
different theme (DT).
Based on the findings from Experiment 2, we predicted that the DT signature
would more closely match that of Experiment 2. However, because the themes of
Experiment 3's abstracts are different from the sections they were being compared
to, we predicted the differences of the LSA scores for the AI and AD comparison
types over the AM and AR comparison types would be less pronounced. To test
this prediction, we randomly replaced the abstracts from Experiment 1 with the
thematically consistent abstracts from Experiment 2.
To examine differences in relatedness of the abstract to each of the text sec-
tions for the DT corpus, we conducted a repeated measures ANOVA on the LSA
cosines including the within-text factors of AI ( M=.289, SD=.138 ), AM ( M =.231,
SD =.141), AR ( M =.234, SD =.143), and AD ( M =.298, SD =.150). There was a main
effect of comparison type, F (3,19)= 9.278, MSE =.002, p< .001. Pairwise contrasts
(see Table 7.4) indicated that the DT signature (like the ST signatures) resulted
in the AR and AM comparisons being significantly different from the AI and AD
comparison types. However, also like the ST signature, the AI comparisons did not
significantly differ from the AD comparisons. Thus, despite the appearance of Fig-
ure 7.1 producing a lower cosine signature, we could not be sure from these results
whether the DT corpus significantly differed from the ST corpus.

Table 7.4. Pairwise comparison of abstract to dissimilarly themed body

Method (AM) Results (AR) Discussion (AD)

Introduction (AI) Diff=.058 (.013)* Diff=.055 (.014)* Diff=-.002 (.013)
Method (AM) Diff=-.003 (.015) Diff=-.060 (.019)*

Results (AR) Diff=-.056 (.017)*

Notes: Diff denotes the average difference between the cosines; * p<.01

To examine whether the SP, ST, and DT corpora were significantly different from
one another, we ran mixed ANOVA, including the within-text factor of comparison
type and the between-text factor of corpora. Because the SP corpus contained 67
papers, and the other corpora were comprised of 20 papers, we randomly selected a
20-paper corpus from Experiment 2a to represent the SP corpus.
As shown in the previous studies, there was a main effect of comparison type,
F (3,171) = 41.855, MSE =.007, p< .001. There was also a main effect of corpus,
F (2,57) = 67.259, MSE =.013, p< .001. A post hoc Bonferroni test between the cor-
pora indicated that there was a significant difference between each of the corpora: SP