Intelligent News Aggregator for German with Sentiment Analysis - Smart Information Systems: Computational Intelligence for Real-Life Applications - page 37

all features into the set and 0.169 higher than the F1-score achieved by the Naive
Bayes baseline with all features. We find that polarity classification is more difficult
for the negative class.
Sentiment Classification . Our sentiment classification approach aims to solve
a three-class classification problem. We evaluate our overall approach by putting
together the results of our two SVM classifiers. Starting with the results of the
subjectivity classifier we pass all quotations classified as subjective to our polarity
classifier for further separation into positive and negative. The overall sentiment
classification results in a macro-average F1-score of 0.51 . Remember, the dataset is
unbalanced and contains 86 % neutral citations. We also conduct experiments pro-
viding Gold Standard answers for one of the tasks in order to evaluate the impact
of the second classifier on the performance of the overall system. First, we simulate
the output of the subjectivity classifier by taking the Gold Standard answers as input
of the polarity classifier (Table 1.7 , Gold Subjectivity Answers Polarity Classi-

fication) and, second, assign Gold Standard answers to quotations marked as sub-
jective by the subjectivity classifier (Table 1.7 , Subjectivity Classification

+

Gold
Polarity Answers). We then measure the effect of each classifier when integrated in
an optimal system. With Gold Subjectivity answers our overall system achieves a
F1-score of 0.86 . Looking at it the other way round, using the subjectivity classi-
fier's output and the Gold polarity answers, our system achieves a F1-score of 0.61,
which is around 0.1 higher than the overall system result but around 0.25 lower than
the system grounded on the Gold subjectivity answers. These results correlate with
the results obtained when testing the classifiers separately. The main error source is
the subjectivity classifier.

+

Table 1.6 Results for the polarity classification part

Positive Negative Macro-averaged Accuracy

Pre Rec F1 Pre Rec F1 Pre Rec F1

NB baseline (all) 0.763 0.859 0.808 0.655 0.500 0.567 0.709 0.680 0.688 0.734

All 0.825 0.930 0.874 0.828 0.632 0.716 0.826 0.781 0.795 0.826

All-bow 0.889 0.901 0.895 0.811 0.790 0.800 0.850 0.845 0.848 0.862

All-bow-valence 0.890 0.916 0.903 0.833 0.790 0.811 0.862 0.853 0.857 0.872

All-bow-
valence-postags

0.863 0.887 0.875 0.778 0.737 0.757 0.820 0.812 0.816 0.835

Bow 0.693 0.986 0.814 0.875 0.184 0.304 0.784 0.585 0.559 0.706

Discourse
markers

0.663 0.775 0.714 0.385 0.263 0.313 0.524 0.519 0.513 0.596

Postags 0.743 0.775 0.759 0.543 0.500 0.521 0.643 0.637 0.640 0.679
Sentiws 0.863 0.887 0.875 0.778 0.737 0.757 0.820 0.812 0.816 0.835
Valence shifters 0.711 0.831 0.766 0.539 0.368 0.438 0.625 0.600 0.602 0.670
Our polarity SVM classifier outperforms the Naive Bayes baseline with an F1-score of 0.86 achieved
on all 109 subjective quotations in our corpus. It uses a feature set consisting of SentiWS terms,
POS tags and discourse markers. As with subjectivity classification the most appropriate features
are the SentiWS features