Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

length normalization

Length normalization is a technique used in information retrieval and text processing to adjust term weights and relevance scores to compensate for variations in document length. Because longer documents naturally contain more words and higher raw term frequencies, standard weighting schemes can unfairly bias retrieval and classification algorithms toward them regardless of their true topical relevance. Length normalization resolves this imbalance by scaling term frequencies or vector magnitudes according to the size of the document, such as dividing by the total word count, applying cosine vector normalization, or penalizing deviations from the average document length across a collection. This adjustment ensures that text similarity comparisons and categorization models evaluate documents based on the proportional significance and specificity of their content rather than sheer volume.

1 item

A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization

A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization

T. Joachims

OrganizationsTU Dortmund University

Why you should read this

Presents a theoretical probabilistic foundation for the heuristic Rocchio and TFIDF classifier, introducing an optimized probabilistic variant, PrTFIDF, that improves text categorization accuracy without adding computational complexity.

The Rocchio relevance feedback algorithm is one of the most popular and widely applied learning methods from information retrieval. Here, a probabilistic analysis of this algorithm is presented in a text categorization framework. The analysis gives theoretical insight into the heuristics used in the Rocchio algorithm, particularly the word weighting scheme and the similarity metric. It also suggests improvements which lead to a probabilistic variant of the Rocchio classifier. The Rocchio classifier, its probabilistic variant, and a naive Bayes classifier are compared on six text categorization tasks. The results show that the probabilistic algorithms are preferable to the heuristic Rocchio classifier not only because they are more well-founded, but also because they achieve better performance.

Added

2026-09-24