◂ PROJECT 02 · LEARNING METER ROUGH WORK · BACK OF P. 9–14 9 SKETCHES

PROJECT 02 · ROUGH WORK

Behind Learning Meter

My own boards, schemas and test charts, made while I planned the course and built the scoring. Elements are redacted.

MYWAY · STORYBOARD, GAMEPLAY AND ACTIVITIES

the course, its missions and what each activity asks
A flow that starts from the learning objectives. The left branch leads to learning outcomes, scope and sequence, lesson plans and the teacher’s guide. The middle branch runs through the course narrative and asset maps to the physical storyboard, the assets and the student book. The right branch is MyWay: challenge objectives, missions, narrative and programming notes, the activity guide and the digital storyboard. Numbered circles mark the largest workloads.
SKETCH 1 OF 9How a course gets built A flow that starts from the learning objectives. The left branch leads to learning outcomes, scope and sequence, lesson plans and the teacher’s guide. The middle branch runs through the course narrative and asset maps to the physical storyboard, the assets and the student book. The right branch is MyWay: challenge objectives, missions, narrative and programming notes, the activity guide and the digital storyboard. Numbered circles mark the largest workloads.FED INTO ·What a learner plays ▸
A Miro board for Mission 2, set at B2. Each of its three challenges lists its learning objectives with their weighting, then the lexical, grammatical, textual and conversational content that covers each one: restaurants and cooking with quantifiers and a recipe, animals with tenses and chat messages, jobs with modals.
SKETCH 2 OF 9Mapping a mission A Miro board for Mission 2, set at B2. Each of its three challenges lists its learning objectives with their weighting, then the lexical, grammatical, textual and conversational content that covers each one: restaurants and cooking with quantifiers and a recipe, animals with tenses and chat messages, jobs with modals.FED INTO ·What a learner plays ▸
A schema. A box reading activity level differentiation, levels zero, one and two, branches to two lists. Content design elements: content focus, distractor complexity, support mechanisms, audio focus, skill complexity. Game design elements: template layering, guidance and hints, user prompts, tested items, duration, screen count.
SKETCH 3 OF 9What a level changes A schema. A box reading activity level differentiation, levels zero, one and two, branches to two lists. Content design elements: content focus, distractor complexity, support mechanisms, audio focus, skill complexity. Game design elements: template layering, guidance and hints, user prompts, tested items, duration, screen count.FED INTO ·What a learner plays ▸
A notebook cell that loads the activity tracker and draws a treemap of AP1 activities, nested by mission, challenge, learning objective, type and difficulty level.A notebook cell that counts activities at each of fifteen production stages, from waiting and activity ideation through game design, assets, creation and testing to activity ready, and prints the completion percentage.A notebook cell that pivots the activities by challenge, learning objective, type and level, and prints a line per challenge such as twelve adaptive at level 0, twelve at level 1 and six fixed.A notebook cell that compares every activity name with every other and lists pairs above a similarity threshold, with their codes and status.A notebook cell that writes an overview for each mission: its learning objectives and challenges, the total of fixed, adaptive and PlayZone activities, and the distribution by level.
SKETCH 4 OF 9Tracking the activities Colab notebooks I wrote to keep the AP1 course in balance: a treemap of every activity by mission, challenge, learning objective, type and level; counts of adaptive and fixed activities per challenge; the production pipeline from activity ideation to ready; a check for duplicate and near-identical activity names; and a script that writes the mission summary report.FED INTO ·What a learner plays ▸

LEARNING METER · THE AI SCORING

the system, the prompt and the tests behind the score
A diagram of one activity, drawn in 2023. Mini-activities run in order. A speaking or writing answer goes through speech-to-text to the Learning Service, which prompts ChatGPT and stores the result. The activity then computes a weighted grade, sends a report and saves the score to the learner’s user model. Sample tables beside it show the arithmetic.
SKETCH 5 OF 9The first Learning Meter A diagram of one activity, drawn in 2023. Mini-activities run in order. A speaking or writing answer goes through speech-to-text to the Learning Service, which prompts ChatGPT and stores the result. The activity then computes a weighted grade, sends a report and saves the score to the learner’s user model. Sample tables beside it show the arithmetic.FED INTO ·Four checkpoints ▸
The 2024 version on Azure OpenAI. The Learning Service filters the sentence, checks a cache, builds the final prompt from a prompt database and a question database, checks the response format and saves everything for review. On the right sits the prompt testing suite: an evaluation set of real and synthetic answers with expected scores goes through evaluate, validate and quantify, and loops back to the prompts until the results are acceptable. Logos and a partner name are redacted.
SKETCH 6 OF 9The prompt, and how I tested it The 2024 version on Azure OpenAI. The Learning Service filters the sentence, checks a cache, builds the final prompt from a prompt database and a question database, checks the response format and saves everything for review. On the right sits the prompt testing suite: an evaluation set of real and synthetic answers with expected scores goes through evaluate, validate and quantify, and loops back to the prompts until the results are acceptable. Logos and a partner name are redacted.FED INTO ·Four checkpoints ▸The score screen ▸
The scoring prompt: grade a young learner’s English answer on a set of score options, reply in JSON with a score and an explanation, and explain mistakes in ten words or fewer.The test loop and its output. For each answer it prints the expected score, the predicted score, the Learning Meter’s feedback and the cost of the query.Confusion matrices of expected against predicted scores, one for each version of an activity.
SKETCH 7 OF 9The test bench we ran together The notebook used to test the scoring before launch. It holds the scoring prompt, sends each answer in the test set to Azure OpenAI, prints the expected score beside the predicted one with the cost of each query, and builds a confusion matrix for every activity version.FED INTO ·Four checkpoints ▸The score screen ▸
Absolute score error for each expected score: 0, 60, 80 and 100.Mean, minimum and maximum latency per query, in seconds.Mean cost per query, in US dollars.Answers that came back in the expected format, against those that did not.
SKETCH 8 OF 9Reading the test runs Four charts from one run over the evaluation set: how far each score landed from the expected one, how long each query took, what each cost, and whether every answer came back in a format the app could read.FED INTO ·The score screen ▸
Notebook cells defining cosine similarity, computing a centroid for each activity, version and expected score, and measuring the distance from every embedding to every centroid.
SKETCH 9 OF 9Scoring by distance A second approach to scoring. Each answer becomes an embedding, a centroid is computed for every score group within an activity, and a new answer takes the score of the nearest centroid.FED INTO ·Four checkpoints ▸