◂ CONTENTS PROJECT 02 · KIDS&US · P. 9–14 CHECKPOINTS 4/4

PROJECT 02 · EVALUATION BEFORE LAUNCH

Learning Meter

Speaking and writing activities inside Animal Planet by Kids&Us, the gamified learning app, for learners aged 8 to 12 learning English as a second language. A learner speaks into a game and ten words come back telling them what to fix, marked by AI against a teacher’s rubric. Getting those ten words right meant building the thing underneath: rubrics, anchor answers?ANCHOR ANSWERAn example answer for each rubric band, given to the scorer so it can see what each level looks like., a test set, and an evaluation loop that ran before anything reached a learner.

IN THE PRESSKids&Us mejora la participación de su alumnado en un 40% con la IA de MicrosoftMicrosoft Source EMEA, 16 January 2025

ROLE · EDTECH SPECIALIST & PROJECT MANAGERREMOTE · BARCELONA & BILBAO · 2023–2024
PROBLEMLearners needed feedback on their spoken and written English, and checking an AI score could not fall to teachers.
MY PART
  • Rubric bands?RUBRIC BANDA level on the marking scale used to judge a learner’s spoken or written answer. and promptsAI FEATURE
  • Evaluation loop before launch≈4,000 ANSWERS
  • Game and story designWHOLE APP
  • Activity designWHOLE APP
OUTCOMEFalse passes in the evaluation
Before58%
After27%
Over about 4,000 answers. Live in all 650 schools; participation up 40% in A/B tests.
THE JOB COVERED · APPROXIMATE SHARE OF MY TIME
UX AND UIStoryboard, gameplay and activities
AI FEATURESRubric bands, prompts, test bench, feedback wording
METRICSA/B tests in the PlayZone
DELIVERY AND CONTENTActivity authoring, QA, five direct reports
MORE ABOUT THE ROLE ▾
ROLEREMOTE · BARCELONA & BILBAO · 2023–2024
JOB TITLEEdTech Specialist & Project ManagerProduct lead, AI Learning Meter in Animal Planet by Kids&Us
QUESTThe Learning Meter end to end, from the CEFR?CEFRThe Common European Framework of Reference for Languages, the scale from A1 to C2 used to describe how well someone uses a language.-aligned rubric bands?RUBRIC BANDA level on the marking scale used to judge a learner’s spoken or written answer. and the prompt design to the evaluation loop that ran before launch. Live in all 650 schools.
TEAMFive direct reports, and matrix-managed digital, art, development and editorial specialists.
JUMP TO THE SCORE SCREEN ▸
WHO IT WAS FORMainly Kids&Us franchise schools and their teachers. Families too, since learners were expected to use the app at home as well.
AGREED AT THE STARTThe Learning Meter needed a human in the loop. Teachers already carry a heavy load, so that check could not fall to them.
THE PATH

Four checkpoints before a learner saw a score

  1. CHECKPOINT 1 ✓The rubric I wroteFour bands, each with anchor answers for every activity.
  2. CHECKPOINT 2 ✓The rubric, runningOne answer followed through the pipeline, first prompts against tuned.
  3. CHECKPOINT 3 ✓The test set≈4,000 answers with known scores: ≈1,300 real, ≈2,700 synthetic.
  4. CHECKPOINT 4 ✓Before and afterDid it get more trustworthy? The score screen below.
ROUGH WORK · SKETCHED FIRST · 5 SKETCHES ▾

SCORE SCREEN · FIRST PROMPTS VS TUNED

one block = 5%
Wrong answers wrongly passedLower is better. The number I cared about most.
FIRST ≈58%
TUNED ≈27%
Exact agreement with the expected scoreThe same score as the one set by a person.
FIRST ≈44%
TUNED ≈56%
Within one bandOff by at most one step.
FIRST ≈66%
TUNED ≈75%
100%of responses came back as a valid score plus feedback
≈1–2 Stypical response, with rare peaks near 10 s
ROUGH WORK · SKETCHED FIRST · 1 SKETCH ▾
DESIGN DECISIONS

Trade-offs I designed around

RISKQuality varies from run to runEvery answer in the test set was scored twice, and model settings stayed low-creativity, so drift showed up as a number before launch.
RISKFalse praiseAnchor answers at every band and synthetic near-misses in the test set cut false passes?FALSE PASSWhen the AI said an answer met the level but a human grader said it didn’t. The error that matters most for learners. from 58% to 27%.
RISKThe wrong languageAnything not in English scores 0, a hard rule in every prompt.
RISKLatencyFeedback is read mid-game, so speed was tracked alongside accuracy.
RISKCost and consistencyLearners give many of the same answers. Matching them first made scoring instant and cheaper, and the same answer always got the same score.
RISKWhat the learner readsTen words or fewer, in the first person, naming one fix, and strict on the score.
SHIPPED · IPHONE AND IPAD

Animal Planet by Kids&Us, on an iPad

The app the Learning Meter sits inside. A learner travels the world map mission by mission, with the caravan as a hub for audios, challenges and the Play Zone.

Animal Planet by Kids&Us on an iPad: the world map with missions 1 to 4 on the continents.
WORLD MAPMissions 1 to 4, placed on the continents.
Animal Planet by Kids&Us on an iPad: the caravan hub, with Audios, Challenge 1 and the Play Zone.
THE CARAVANThe hub, with Audios, Challenge 1 and the Play Zone.
Animal Planet by Kids&Us on an iPad: the Mission 2 map.
MISSION 2 · MAPThe route through one mission.
Animal Planet by Kids&Us on an iPad: a Mission 2 challenge.
MISSION 2 · CHALLENGEOne challenge inside the mission.
IN THE APP

What a learner plays

An Animal Planet activity intro screen titled Recipe Rush on a patterned teal background. A card reads: first, complete the recipe, then follow the steps by using the correct utensils and carrying out the correct actions. A Play button sits below it, and a drag-and-drop icon marks the activity type in the corner.
AN ACTIVITY OPENSThe task in a learner’s own level, and the activity type marked before they start.
The Recipe Rush activity. A torn recipe card taped to a mint green patterned background reads: chop the onion and fry it in some olive oil. A knife, a frying pan, a pot boiling on a hob, a wooden spoon and an oven are laid out below it.
THE TASK ITSELFThe instruction stays on screen while a learner picks the utensils and carries out the actions in order.
An Animal Planet listening activity. Three restaurant fronts sit across a patterned teal background: a sushi restaurant, an Italian one and a burger bar. A character avatar with a speaker bubble sits below them, and a multiple-choice icon marks the activity type in the corner.
A DIFFERENT TYPEAudio plays from the characters, and a learner answers by choosing. The corner icon carries the change.
The Kids&Us Activity Maker. Block categories run down the left: game objects, audio, video, position, appearance, custom logic, grading, arrays, waits and comments. The canvas holds three columns of blocks: an objects definition listing hearts, a test button and a fail screen; a configuration column setting variables and binding audio assets; and a start program column with the loop that runs them.
THE ACTIVITY MAKERBlock categories, then three columns: the objects a screen contains, the configuration that sets them up, and the start program that runs them. Recipe Rush, above, was built here.
ROUGH WORK · SKETCHED FIRST · 3 SKETCHES ▾
BONUS ROUND

What moved engagement

I redesigned the gamified loops from engagement data and A/B tested the redesign. The test grew from my BETT 2024 write-up, which proposed the playzone as a place to find out what works. Animal Planet by Kids&Us went on to reach more than 25,000 learners in its first rollout.

+40%learner participation, A/B tested
3–4 → 7–8activities a week
25,000+learners in the first rollout, reported by Microsoft (2025)
An illustrated world map of islands with mission markers on it, a learner's avatar in the middle, a settings button and a character icon in the corners, and a small robot companion at the bottom right.
THE WORLD MAPMissions open from an illustrated map of the course.
A treemap of Mission 4. Three challenges, each split into blocks labelled Adaptive and Fixed, with adaptive blocks broken down by level 0.0, 1.0 and 2.0.
INSIDE A MISSIONEach challenge mixes adaptive activities, at levels 0.0, 1.0 and 2.0, with fixed ones.

PRODUCT SCREENS AND ARTWORK ARE KIDS&US’S

ROUGH WORK · BEHIND THIS PROJECT · 9 SKETCHES ▾
ROUGH WORK · BEHIND THIS PROJECTFLIP TO THE ROUGH WORK ▸
LOOKING BACK

What I would do differently

My approach has become more creative since. I would revisit the four score bands with that in mind, and treat how a band looks and feels to a learner as a design question in its own right.

STAGES CLEARED · 6/6

Where I worked across the lifecycle

  1. 01 · DISCOVER✓
    • Speech and writing tasks for learners around eight
    • Rubric bands
    • Discovery into activity gamification
  2. 02 · PROTOTYPE✓
    • Prompt databases
    • Anchor answers per activity
  3. 03 · PILOT✓
    • A test set of about 4,000 answers
    • Real answers plus synthetic edge cases
    • 10 representative franchise schools
  4. 04 · VALIDATE✓
    • Evaluation loop before launch
    • Every answer graded twice
  5. 05 · BUILD✓
    BEYOND THE BRIEF
    • Handoff to engineers
    • Caching for repeat answers
  6. 06 · MEASURE✓
    BEYOND THE BRIEF
    • False passes, agreement, latency, cost

From a rubric to the accuracy numbers that decided what shipped.