Character n-gram application for automatic new topic identification


ÇAĞLAR GENÇOSMAN B., ÖZMUTLU H. C., ÖZMUTLU S.

INFORMATION PROCESSING & MANAGEMENT, vol.50, no.6, pp.821-856, 2014 (SCI-Expanded) identifier identifier

  • Publication Type: Article / Article
  • Volume: 50 Issue: 6
  • Publication Date: 2014
  • Doi Number: 10.1016/j.ipm.2014.06.005
  • Journal Name: INFORMATION PROCESSING & MANAGEMENT
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Social Sciences Citation Index (SSCI), Scopus
  • Page Numbers: pp.821-856
  • Keywords: Content-ignorant algorithms, The character n-gram method, New topic identification, The Levenshtein edit-distance, Pre-processed spelling correction methods, NEURAL-NETWORK APPLICATIONS, WEB, CATEGORIZATION
  • Bursa Uludag University Affiliated: Yes

Abstract

The widespread availability of the Internet and the variety of Internet-based applications have resulted in a significant increase in the amount of web pages. Determining the behaviors of search engine users has become a critical step in enhancing search engine performance. Search engine user behaviors can be determined by content-based or content-ignorant algorithms. Although many content-ignorant studies have been performed to automatically identify new topics, previous results have demonstrated that spelling errors can cause significant errors in topic shift estimates. In this study, we focused on minimizing the number of wrong estimates that were based on spelling errors. We developed a new hybrid algorithm combining character n-gram and neural network methodologies, and compared the experimental results with results from previous studies. For the FAST and Excite datasets, the proposed algorithm improved topic shift estimates by 6.987% and 2.639%, respectively. Moreover, we analyzed the performance of the character n-gram method in different aspects including the comparison with Levenshtein edit-distance method. The experimental results demonstrated that the character n-gram method outperformed to the Levensthein edit distance method in terms of topic identification. (C) 2014 Elsevier Ltd. All rights reserved.