Character n-gram application for automatic new topic identification

ÇAĞLAR GENÇOSMAN B., ÖZMUTLU H. C., ÖZMUTLU S.

INFORMATION PROCESSING & MANAGEMENT, cilt.50, sa.6, ss.821-856, 2014 (SCI-Expanded) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 50 Sayı: 6
  • Basım Tarihi: 2014
  • Doi Numarası: 10.1016/j.ipm.2014.06.005
  • Dergi Adı: INFORMATION PROCESSING & MANAGEMENT
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Social Sciences Citation Index (SSCI), Scopus
  • Sayfa Sayıları: ss.821-856
  • Anahtar Kelimeler: Content-ignorant algorithms, The character n-gram method, New topic identification, The Levenshtein edit-distance, Pre-processed spelling correction methods, NEURAL-NETWORK APPLICATIONS, WEB, CATEGORIZATION
  • Bursa Uludağ Üniversitesi Adresli: Evet

Özet

The widespread availability of the Internet and the variety of Internet-based applications have resulted in a significant increase in the amount of web pages. Determining the behaviors of search engine users has become a critical step in enhancing search engine performance. Search engine user behaviors can be determined by content-based or content-ignorant algorithms. Although many content-ignorant studies have been performed to automatically identify new topics, previous results have demonstrated that spelling errors can cause significant errors in topic shift estimates. In this study, we focused on minimizing the number of wrong estimates that were based on spelling errors. We developed a new hybrid algorithm combining character n-gram and neural network methodologies, and compared the experimental results with results from previous studies. For the FAST and Excite datasets, the proposed algorithm improved topic shift estimates by 6.987% and 2.639%, respectively. Moreover, we analyzed the performance of the character n-gram method in different aspects including the comparison with Levenshtein edit-distance method. The experimental results demonstrated that the character n-gram method outperformed to the Levensthein edit distance method in terms of topic identification. (C) 2014 Elsevier Ltd. All rights reserved.