Code-switching Detection - Approaches and Evaluation: Investigating approaches and evaluation methods for code-switching detection in multilingual text data to identify language switches within sentences

Abstract

Code-switching, the alternation between two or more languages within a single discourse, is a prevalent linguistic phenomenon in multilingual communities. Detecting code-switching in text data is essential for various natural language processing (NLP) tasks, such as machine translation, sentiment analysis, and information retrieval, to ensure accurate language processing. This paper provides a comprehensive overview of approaches and evaluation methods for code-switching detection in multilingual text data. We examine the challenges associated with code-switching detection, including the lack of annotated datasets, the complexity of language mixing patterns, and the need for context-aware detection algorithms.

The paper discusses various approaches used for code-switching detection, including rule-based methods, statistical models, and deep learning techniques. Rule-based methods rely on linguistic rules and patterns to identify language switches, while statistical models utilize probabilistic models to detect code-switching based on lexical and syntactic features. Deep learning techniques, such as recurrent neural networks (RNNs) and transformer models, have shown promising results in code-switching detection by leveraging the contextual information of text data.

Furthermore, we explore evaluation methods for code-switching detection, including accuracy, precision, recall, and F1 score. We discuss the importance of annotated datasets for evaluating code-switching detection systems and the challenges of cross-lingual evaluation in code-switching detection. We also review existing annotated datasets and evaluation benchmarks for code-switching detection to facilitate future research in this area.