Find Similar Texts

To find messages like a new one, compare their words. Turn every message into a TF-IDF row of word weights, as in the TF-IDF lesson, then score two rows with cosine similarity: 1.0 when they have the same words in the same mix, 0.0 when they share no word, and in between when they share some. Punctuation and word order don't count, and a shared rare word counts more than a shared common one.

cosine_similarity(a, b) from sklearn.metrics.pairwise gives back a table of scores: one row per row of a, one column per row of b. Compare one row with all of them and you get a single row, with one score per text:

texts = ["red mug", "red pen", "blue cup"]
vectorizer = TfidfVectorizer()
rows = vectorizer.fit_transform(texts)
print(cosine_similarity(rows[0], rows).round(3))
# one row: 1.0 for "red mug" itself, 0.366 for "red pen", 0.0 for "blue cup"

A search text isn't one of the messages, but it can be scored the same way. vectorizer.transform([text]) gives back its TF-IDF row, weighed with the vocabulary the vectorizer learned from the messages, so it has the same columns as their rows; words the messages never use are left out. Don't fit again: a new vocabulary wouldn't line up with the rows.

This lesson's script prints:

Message: my package has not arrived. Please help.
                                             text     topic  score
56   Hi, my package has not arrived? Please help.  delivery  0.962
146            Hello, my package has not arrived.  delivery  0.851
53      my package has not arrived. It is urgent.  delivery  0.796
Search: where is my package
                                     text     topic  score
40        where is my order. Please help.  delivery  0.609
78        where is my order? Please help.  delivery  0.609
162  Hey, where is my order. Please help.  delivery  0.568

Your task: the starting code reads messages.csv, turns every message into a TF-IDF row in rows and gives you show_top3(table), which prints the 3 rows with the highest "score". Score message 5 against every message, put the rounded scores in a "score" column and show the top 3 without message 5 itself. Then turn the search into a question row, score it against every message and show its top 3.

Assignment
Follow the Coding Tutorial to find the three messages closest to message 5, then the three closest to a search text.

Hint
cosine_similarity gives back a table even for one row: [0] takes its first row, one score per message. drop(index=5) leaves message 5 out, since it always scores 1.0 with itself.