Document Sentiment Classification

The number of words in the list doesn’t mean its quality. We conducted several experiments to benchmark the produced opinion word lists. We decided not to check the words manually, but to use them in the real-world task, that is sentiment clas- sification. The experiments are performed on the annotated part of ROMIP 2011 da- taset [5]. It contains reviews of books, films and cameras. There are 750 positive and 124 negative review instances.

Counting the number of positive and negative words is the most straightforward way to text sentiment classification [13]. The one with the greater number of opin- ion words wins. The work [17] suggests that it is better to consider the presence of an opinion word in text rather than the number of appearances. We implement both approaches. We will refer to the first as to “Frequency voc” and to the second as to “Binary voc”.

Supervised approaches to text sentiment classification were studied by Pang et al.

[17]. We use a linear perceptron classifier with two types of feature computation: term frequencies and delta TF-IDF. The latter was proposed by Martineau et al. [11] and proven to be efficient for sentiment classification in Russian [16]. The experiment re- sults of these methods were obtained after performing 10-fold cross validation. These results act as a base line of supervised classification that requires an annotated data- set. We compare them with dictionary-based classification that does not require class labels to train, because it has negative and positive words. Therefore, results of super- vised classification are considered as a higher bound for a dictionary based.

table 4. Experiment results

Lexicon Method MicroP MicroR (Acc) MacroR MacroF1

Perceptron 0.84 0.84 0.59 0.60

Perceptron + TfIdf 0.84 0.84 0.62 0.63

Romip-GT Binary Voc 0.76 0.68 0.59 0.58

Frequency Voc 0.79 0.72 0.59 0.59

Romip-GT merged

Binary Voc 0.84 0.80 0.59 0.61

Frequency Voc 0.86 0.82 0.59 0.61

Ulanov A. V., Sapozhnikov G. A.

Lexicon Method MicroP MicroR (Acc) MacroR MacroF1

BL-GT Binary Voc 0.65 0.60 0.62 0.54

Frequency Voc 0.73 0.69 0.59 0.56

BL-GT filtered

Binary Voc 0.78 0.78 0.59 0.58

Frequency Voc 0.77 0.72 0.58 0.58

MR Binary Voc 0.67 0.52 0.50 0.49

Frequency Voc 0.66 0.53 0.51 0.50

The experiment results are represented in Table 4. The binary approach provides the same weight to all of the words. Low performance of the binary approach as com- pared with the frequency approach means that the lexicon is of low quality. It may contain common words that can be found in the text (that rarely speak about subjec- tivity). So we can say that “BL-GT” is rather dirty. “ROMIP-GT merged” gives the best performance among the opinion lexicons. It has the same number of words as “BL- GT filtered”, but the performance of the “ROMIP-GT merged” is higher, so we can say that its quality for sentiment classification is better. It is because the words in “ROMIP- GT merged” were translated with the use of context unlike the words in “BL-GT fil- tered”. “BL-GT filtered” shows better results in manual assessment, but worse results in classification. We can explain this by the fact that “ROMIP-GT merged” contains such words that out of context may seem not opinion words or words that are more often used in user reviews as compared with words from “BL-GT filtered”.

We supposed that the increase in the classification performance could be due to the fact that we used a part of the big dataset ROMIP 2011 to retrieve “ROMIP-GT merged”, and the labeled dataset that was used for classification was also a part of ROMIP 2011.

However, it turned out that the intersection between these parts did not exceed 1%, and it couldn’t lead to the significant increase of the classification performance.

We use our lexicons as a list for feature selection as in [6], and train a linear per- ceptron classifier. It produces nearly the same results both for “ROMIP-GT merged”

and “BL-GT filtered”. This experiment shows that “BL-GT filtered” contains enough words that can be used as classification features. However, it also contains common words that have low weight in the supervised classifier, which does not happen when this lexicon is used in vocabulary classification.

4. Conclusion

We proposed a novel method for opinion lexicon projection from a source lan- guage to a target language with the use of a parallel corpus. The method was applied to different datasets and evaluated against the baseline. The quality of created lexi- cons was evaluated in sentiment classification benchmark. The experiments showed that the lexicons are of high quality. They can be used for sentiment annotation of a corpus in a target language as well.

Out future work is related to enhancement of the method and conducting more experiments. We plan to work with opinion phrases, investigate other translation

Context-dependent opinion lexicon translation with the use of a parallel corpus

options instead of the most probable ones. We will apply our method to other lan- guage pairs, apart from English-Russian. Additionally, it will be interesting to explore how the method can be applied to other tasks, such as subjectivity lexicon projection and, more general, multilingual projection of document features.

References

1. Banea C., Mihalcea R., Wiebe J., Hassan S. Multilingual subjectivity analysis us- ing machine translation. EMNLP ‘08: Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing, 2008.

2. Breck, E., Y Choi, and C. Cardie. Identifying expressions of opinion in context.

In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-2007), 2007.

3. Carmen Banea, Rada Mihalcea, and Janyce Wiebe. Multilingual Sentiment and Subjectivity, in Multilingual Natural Language Processing, editors Imed Zitouni and Dan Bikel, Prentice Hall, 2011.

4. Chatviorkin Ilya, Lukashevich Natalia. Automatic Extraction of Domain-Specific Opinion Words. Proceedings of the International Conference Dialog, 2010.

5. Chatviorkin Ilya, Braslavski Pavel, Lukashevich Natalia. Sentiment analysis track at ROMIP 2011. Proceedings of the International Conference Dialog, 2012.

6. Dang Y., Zhang Y., Chen H. A lexicon-enhanced method for sentiment classifica- tion: An experiment on online product reviews, IEEE 2010.

7. Ding, X., B. Liu, and P. Yu. A holistic lexicon-based approach to opinion mining.

In Proceedings of the Conference on Web Search and Web Data Mining (WSDM- 2008), 2008.

8. Eisele A., Chen Y. MultiUN: A Multilingual Corpus from United Nation Docu- ments. In Language Resources and Evaluation, 2010.

9. Hatzivassiloglou, V. and K. McKeown. Predicting the semantic orientation of ad- jectives. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL-1997), 1997.

10. Hu, M. and B. Liu. Mining and summarizing customer reviews. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2004), 2004.

11. J. Martineau and T. Finin. Delta TFIDF: An Improved Feature Space for Sentiment Analysis. In Proceedings of the Third AAAI Internatonal Conference on Weblogs and Social Media, San Jose, CA, 2009.

12. Liu, Bing. Web Data Mining. Exploring Hyperlinks, Contents, and Usage Data.

2nd ed. 2011, XX, 622 p.

13. Melville P., Gryc W., and Lawrence R. Sentiment analysis of blogs by combining lexical knowledge with text classification. KDD 2009.

14. Mihalcea R., Banea C. and Wiebe J. Learning Multilingual Subjective Language via Cross-Lingual Projections, in Proceedings of the Association for Computa- tional Linguistics (ACL 2007), Prague, June 2007.

Ulanov A. V., Sapozhnikov G. A.

15. Och F. J., Ney H. A Systematic Comparison of Various Statistical Alignment Mod- els, Computational Linguistics, volume 29, number 1, pp. 19–51 March 2003.

16. Pak A., Paroubek P. Language independent approach to sentiment analysis (Limsi participation in romip’11) Proceedings of the International Conference Dialog, 2012.

17. Pang В., Lee L. Thumbs up? Sentiment Classification using Machine Learning Techniques, In Proc. of the conference on the Empirical Methods 2002

18. Robert C. Moore. Fast and Accurate Sentence Alignment of Bilingual Corpora.

In Proceedings of the 5th Conference of the Association for Machine Translation in the Americas on Machine Translation: From Research to Real Users (2002), pp. 135–144

19. Steinberger J., Ebrahim M., Ehrmann M., Hurriyetoglu A., Kabadjov M., Len- kova P., Steinberger R., Tanev H., Vázquez S., Zavarella V. Creating Sentiment Dictionaries via Triangulation. Decision Support Systems, May 2012.

20. Steinberger J., Lenkova P., Kabadjov M., Steinberger R., van der Goot E. Multilin- gual Entity-Centered Sentiment Analysis Evaluated by Parallel Corpora. In Pro- ceedings of the International Conference Recent Advances in Natural Language Processing, 2011.

21. Turney, P. Thumbs up or thumbs down?: semantic orientation applied to unsu- pervised classification of reviews. In Proceedings of Annual Meeting of the As- sociation for Computational Linguistics (ACL-2002), 2002.

StatiStical machine tranSlation with linguiStic language model

Zuyev K. A. (konst@abbyy.com),

Indenbom E. M. (Eugene_I@abbyy.com), Yudina M. V. (Maria_Yu@abbyy.com)

ABBYY, Moscow, Russia

Stemming from traditional “rule based” translation a “model based” ap- proach is considered as an underlying model for statistical machine trans- lation. This paper concerns with training on parallel corpora and application of this model for parsing and translation.