Two Words Together

A vectorizer that looks at words one at a time throws their order away: to it, "good, not bad" and "bad, not good" are the same row. Most words don't mind, but not only means something next to the word it denies.

Word pairs: TfidfVectorizer(ngram_range=(1, 2)) makes a column for every single word and for every pair of neighboring words, a bigram: (1, 2) means "terms of 1 to 2 words". A vectorizer's build_analyzer() gives back the function it uses to split a text into terms, so you can see them:

split = TfidfVectorizer(ngram_range=(1, 2)).build_analyzer()
print(split("Never arrived at all"))

['never', 'arrived', 'at', 'all', 'never arrived', 'arrived at', 'at all']

Filler words: stop_words="english" drops a ready-made list of about 300 common English words, like "the", "this" and "would", before the terms are made. That means fewer columns, but look closely at what it drops. A fitted vectorizer's vocabulary_ is a dict from each term to its column, so len(vectorizer.vocabulary_) is its number of columns.

The starting code splits reviews.csv into 75% for training and 25% for testing and defines try_vectorizer(name, vectorizer). It trains a pipeline of that vectorizer and a LogisticRegression, prints the number of columns and the test accuracy, then shows how the vectorizer splits the tricky review "Would not buy this teapot again" and what the model answers. Your code must print:

Words: 108 columns, accuracy 0.983
   ['would', 'not', 'buy', 'this', 'teapot', 'again'] -> positive
Words and pairs: 536 columns, accuracy 1.0
   ['would', 'not', 'buy', 'this', 'teapot', 'again', 'would not', 'not buy', 'buy this', 'this teapot', 'teapot again'] -> negative
Pairs, no stop words: 441 columns, accuracy 0.967
   ['buy', 'teapot', 'buy teapot'] -> positive

Your task: below the first run, try two more vectorizers: pairs, with words and pairs, then trimmed, with pairs and no stop words. Compare the columns, the scores and the tricky review's answer. Which of its words did the stop-word list take away?

Assignment
Follow the Coding Tutorial to make the pairs and trimmed vectorizers and pass each one to try_vectorizer.

Hint
Both new vectorizers take ngram_range=(1, 2); trimmed also takes stop_words="english". The names to pass are "Words and pairs:" and "Pairs, no stop words:".