A new text representation method for clustering based on higher order Markov model

The ordinal relations in word sequence and character sequence can reflect the latent information about writing style, genre features and topic. Thus, the ordinal relations are important information and should be considered for text clustering. However, the ordinal relations were often neglected in the traditional methods of text clustering. In view of that the ordinal relations can be statistically characterized by the transition probabilities of the higher order Markov model, in this paper, a new method based on higher order Markov model was proposed for text representation. In the new method, all transition probabilities of a higher order Markov model are used as features of text, and the order is identified by maximizing the average Markov-Shannon entropy (MME). The experimental results imply the new text representation method performs better than traditional method.