Using conditional probabilities for automatic new topic identification

Purpose – One of the most important dimensions of search engine user information seeking behaviour is content‐based behaviour. One of the main elements in developing a personalised intelligent search engine is new topic identification. The purpose of this study is to perform automatic new topic identification in search engine transaction logs using conditional probabilities of new topic arrivals. Design/methodology/approach – Sample data logs from FAST (currently owned by Yahoo!) and Excite (currently owned by IAC Search & Media) are used in the study. Conditional probabilities of new topic arrivals and topic continuations given query category are used to estimate new topic arrivals. Findings – The findings of this study show that the conditional probability approach reduced overestimation of topic shifts, increasing some performance measures to their highest ever value compared to previous studies. A straightforward procedure such as the conditional probability approach can be as successful as, and for some measures more successful than, more complex methods applied in previous automatic new topic identification studies. Originality/value – A straightforward procedure that can enable fast automatic new topic identification, a problem not yet solved, and an important step towards personalised search engines.