SKILL 1 OF 9 · AI FEATURES
LLM evaluation
Checking that a model scores and responds the way a teacher would, before a learner sees anything. A rubric, a set of answers with known scores, and a loop that measures the gap.
WHERE IT SHOWS UP
PROJECT 01Beeline
PROJECT 02 · USEDLearning MeterSEE BELOW ▾PROJECT 03JUMP Math Lab
PROJECT 04The Artificial Leaf
HOW I DO IT
The evaluation loop
- 01RubricFour CEFR?CEFRThe Common European Framework of Reference for Languages, the scale from A1 to C2 used to describe how well someone uses a language.-aligned bands.
- 02Anchor answers?ANCHOR ANSWERAn example answer for each rubric band, given to the scorer so it can see what each level looks like.An example answer at every band, for every activity.
- 03Test setAnswers with known scores, real and synthetic.
- 04Score twiceEvery answer marked twice, at low-creativity settings.
- 05MeasureFalse passes?FALSE PASSWhen the AI said an answer met the level but a human grader said it didn’t. The error that matters most for learners., agreement, latency, cost. Then tune and run again.
PROJECT 02 · LEARNING METER · KIDS&US · 2023–2024FROM P. 9–14
The Learning Meter, before launch
A learner speaks or writes inside a game and an LLM?LLMA large language model: an AI system trained on large amounts of text that can read and write language, used here to score answers and write feedback. scores the answer against a teacher’s rubric. The loop above decided what shipped.
THE TEST SET · ≈4,000 ANSWERS
REAL≈1,300
SYNTHETIC≈2,700
Synthetic near-misses test the edges of each band.FALSE PASSES · FIRST PROMPTS → TUNED
FIRST PROMPTS58%
TUNED27%
Anchor answers at every band and synthetic near-misses cut false passes.VALID OUTPUT100%of responses came back as a score plus feedback.
LATENCY≈1–2 stypical response, tracked alongside accuracy.
HARD RULEAnswers not in English score 0A rule written into every prompt.
COSTRepeat answers cachedsame answer, same score, instantly.
TOOLS
Azure OpenAIGPT-3.5 and GPT-4PythonJupyter notebooksGoogle ColabVS CodeEmbeddingsCosine similarityElevenLabs speech-to-textMurf AI text-to-speech
RELATED SKILLS
WORK WITH MEAI features, from design to evaluationWhere AI helps, rubrics and test sets, feedback people can act on
ASK ME FOR A DAY RATE ▸