CodeCite retrieves clauses from the National Building Code of Canada 2020 before generating an answer. This evaluation tests whether a second model improves the order of those retrieved clauses.
The test uses 30 questions, each labelled with an expected clause. Both versions start with the same 30 candidate clauses for each question. One keeps the original ranking; the other uses a second model to rerank them.
for q in questions:
hits = retrieve(client, store, q["question"], k=POOL, chunks=chunks)
reranked = rerank(client, q["question"], hits, top_n=POOL)
dr = first_gold_rank(hits, q["gold"])
rr = first_gold_rank(reranked, q["gold"])
dense_ranks.append(dr)
rerank_ranks.append(rr)
def summarize(ranks: list[int | None]) -> dict:
n = len(ranks)
out = {f"hit@{k}": sum(1 for r in ranks if r and r <= k) / n for k in K_VALUES}
out["mrr"] = sum(1 / r for r in ranks if r) / n
return outResults
| Metric | Embedding only | With reranking |
|---|---|---|
| Expected clause first | 60.0% | 53.3% |
| In the top three | 90.0% | 90.0% |
| In the top five | 93.3% | 90.0% |
| Mean reciprocal rank | 0.760 | 0.709 |
Both versions placed the expected clause in the top three for 27 of 30 questions. The original ranking placed it first more often. Reranking also reduced the top-five score and mean reciprocal rank.
What the results show
These scores measure where the expected clause appears in the results. They do not measure the accuracy of generated answers. The evaluation covers only these 30 questions.
On this test set, reranking provided no improvement. Keeping the evaluation in the repository makes it possible to compare future changes using the same questions.
The published results include the rank for each question, so the individual changes can be checked.