Start learning free

LESSON 68 OF 129 · AI ENGINEERING: RETRIEVAL (RAG)

Keep the Index Fresh

Practice

The shop's documents change: a price goes up, a paragraph is added. The index has to follow. The starting code does it the wasteful way: after one changed sentence it embeds all 27 chunks again. A hosted embeddings model bills every text you send, and a real shop has thousands of chunks.

To embed only what changed, give every chunk a hash: a short fingerprint computed from its text. The same text always gives the same hash, and the smallest change gives a completely different one. Save every chunk with its hash and its vector in index.json. On the next run, a chunk whose hash is in the file takes the saved vector, and only the others are embedded.

Here is the idea on three notes of a café, with letters standing for the hashes:

text now             saved hash   hash now   embed again?
Open 9 to 5.         A            A          no
Closed on Sunday.    B            B          no
Parking costs $2.    C            K          yes

Your program builds the index, the shop changes one sentence of shipping.md (shipping to Canada goes from $15 to $18), and the program builds the index again. When your code is finished, it prints:

Embedded 27 of 27 chunks
The shop changed shipping.md
Embedded 1 of 27 chunks
Best score: 0.661
Shipping to Canada costs a flat $18 and takes 7 to 10 business days.
Source: shipping.md

The check then builds the index four more times:

nothing changed                      →  Embedded 0 of 27 chunks
two paragraphs changed               →  Embedded 2 of 27 chunks
one paragraph added in the middle    →  Embedded 1 of 28 chunks
index.json deleted                   →  Embedded 28 of 28 chunks

Your task: write text_hash, give every chunk the hash of its text, save the index to index.json, write load_vectors to read the saved vectors back, and make build_index reuse a saved vector when a chunk's hash is in the file.


Assignment
Follow the Coding Tutorial to hash every chunk's text, save the index to index.json and embed again only the chunks whose hash is new.


Hint
Only vectors are reused. A chunk's text is always read from the document itself, so the answer has the new price. On the very first run there is no index.json: check that the file exists before you read it.


STEP 1 OF 12
    Ctrl + Enter to check