Loading question…
REFERENCE ANSWER —
01
Ask GLM about the answer
Question onlyPYTHON · SCORING CALL
Answer surprisal —
Each token is scored after the question and the answer tokens before it.
↓ Add search results to the same prompt
02
Give GLM the search results
RETURNED RESULTS
↓ Insert the selected results into GLM’s prompt
PYTHON · SAME ANSWER, NEW CONTEXT
Answer surprisal after search—
Retrieval + GLM—
COST PER BIT REMOVED
— ÷ — = —
RETRIEVAL + ONE GLM SCORE
Retrieval—
GLM score—
As each result enters the prompt
Answer surprisal · bits03
Read the logprobs
PYTHON · LOGPROB EXTRACTION
SELECTED REQUEST · ANSWER TOKENS
Exact request & prompt
Prompt sent to GLM
Production source & question data
bench/prompts.py · bench/scoring/fireworks.py
04