Did Retrieval Find It?

The interview question: "You have a small test set. For each question you know the chunk ids your retriever returned, best first, and the ids that are really relevant. Compute precision@k and recall@k per question, and the hit rate over the set."

The idea: only the top k ids count, because only they reach the prompt. Precision@k is the share of those k ids that is relevant. Recall@k is the share of the relevant ids that made the top k. A question is a hit when at least one relevant id is in the top k, and the hit rate is the share of questions that are hits.

top = ["a", "b", "c"][:2]   # k = 2: ['a', 'b']
relevant = ["b", "c", "d"]
found = 1                    # only 'b' is in both
print(found / 2)             # precision@2: 0.5
print(round(found / 3, 3))   # recall@2: 0.333

The starting code gives you count_found, the four test questions and the prints. With k = 3 your program must print:

Can I get my money back? precision: 0.667 recall: 0.667
Do you ship to Canada? precision: 0.0 recall: 0.0
Is my cracked teapot covered? precision: 0.667 recall: 1.0
Can I pay with PayPal? precision: 0.333 recall: 1.0
Hit rate: 0.75

Your task: inside the loop, count the relevant ids in the top k, compute precision and recall, and count a hit when anything relevant was found.

Assignment
Follow the Coding Tutorial to compute precision@k, recall@k and the hit rate inside the loop.

Hint
Slice the retrieved list with [:k] before counting. Precision divides by k, the same for every question; recall divides by len(relevant[question]), which changes from question to question.