Code-switching behavior is a common phenomenon on social media to express solidarity or establish authority. While past work on automatic code-switching detection depends on dictionary look-up or named-entity recognition, our recurrent neural network model that relies on only raw features outperformed the top systems in the EMNLP’14 Code-Switching Workshop by 17% in error rate reduction.

Final project for the Deep Learning course at CMU.

Abstract

Mixed language data is one of the difficult yet less explored domains of natural language processing. Most research in fields like machine translation or sentiment analysis assume monolingual input. However, people who are capable of using more than one language often communicate using multiple languages at the same time. Sociolinguists believe this “code-switching” phenomenon to be socially motivated. For example, to express solidarity or to establish authority. Most past work depend on external tools or resources, such as part-of-speech tagging, dictionary look-up, or named-entity recognizers to extract rich features for training machine learning models. In this paper, we train recurrent neural networks with only raw features, and use word embedding to automatically learn meaningful representations. Using the same mixed-language Twitter corpus, our system is able to outperform the best SVM-based systems reported in the EMNLP’14 Code-Switching Workshop by 1% in accuracy, or by 17% in error rate reduction.

Download

arXiv arXiv hosted PDF

Citation

1
2
3

Chang, Joseph Chee, and Chu-Cheng Lin.
"Recurrent-Neural-Network for Language Detection on Twitter Code-Switching Corpus."
arXiv preprint arXiv:1412.4314 (2014).

Bibtex

1
2
3
4
5
6

@article{chang2014recurrent,
  title={Recurrent-Neural-Network for Language Detection on Twitter Code-Switching Corpus},
  author={Chang, Joseph Chee and Lin, Chu-Cheng},
  journal={arXiv preprint arXiv:1412.4314},
  year={2014}
}