Keywords and Vectors Together

An embedding keeps the meaning of a text. A product code like TP-220 has no meaning, so a search by vectors can miss the one chunk that names it. Ask "Is TP-220 dishwasher safe?" and the best match is the mug, not the teapot.

What finds a code is the oldest kind of search: look for the same words. A text first becomes a list of key words: no punctuation, all lowercase, and without common words like is or the. For example:

"Is the Desk Lamp (LP-310) dimmable? Yes, it is!"
→ ['desk', 'lamp', 'lp-310', 'dimmable', 'yes']

The keyword score is the share of the question's key words that a text has too. The question "Is TP-220 dishwasher safe?" has three key words. The text "It is dishwasher safe." has two of them, so it scores 0.667. Ranking by a mix of the similarity and the keyword score is called hybrid search: a chunk with similarity 0.2 and keyword score 1.0 gets half of each, 0.6.

When your program is finished, it prints:

paragraph 27 chunks, 5 of 5
Best score: 0.653
No. TP-220 is the teapot, and its glaze isn't made for the dishwasher: wash it by hand.
Source: teapot.md

Your task: write key_words(text), which returns the words of a text that are worth comparing, and keyword_score(question, text), which returns the share of the question's key words found in the text. Then, in top_matches, store each chunk's keyword score and rank by half the similarity plus half the keyword score.

Assignment
Follow the Coding Tutorial to write key_words and keyword_score, and to mix the keyword score into the ranking in top_matches.

Hint
The list COMMON is already at the top of the file. key_words must return its list. keyword_score divides by at least 1, so a question made only of common words gives 0 instead of an error.