COMPUTER VISION / FROM PIXELS TO PLAY

Pixels to Play

Build from image numbers to convolution and learning. Follow each main idea, then open a worked example, formula or Python explanation when you want to go further.

THE LECTURE / FIVE KEY IDEAS

From pixels to a model that learns.

Images are numbers. Brightness and color values become the model’s input.
Filters find local patterns. The same weights slide across the image to make feature maps.
A CNN learns its weights. Forward pass → loss → backward gradients → update → try again.
Check new examples. Better training results alone do not guarantee better predictions.
Use the ideas in a game. Draw and guess, return to the formulas, then test an early prediction and make a video effect.

Start with image data ↓

Overview first

Worked examples, formulas and Python are available in the expandable sections.

Summary Convolution Train the CNN Animate backprop Camera game RPS demo Video demo Roles and evaluation Technical reference Review roadmap

Code language

Choose Python or JavaScript for the code examples. Python is shown first.

01 / START WITH THE INPUT

Images are numbers

One pixel stores brightness or color; an image arranges those values in rows and columns.
A tensor shape records the number of images, height, width and channels.
The symbol CNN receives a small grayscale image and produces Ring, Plus and X scores.

Worked examples: pixels, shapes and model outputs

01 / THE WHOLE SYSTEM

One camera supports several different tasks.

Today we turn pixels into predictions and predictions into a game. The useful question is what each component receives, what it returns, and how we know it worked.

Image measurementsLearned featuresTask outputInteraction rulesEvaluation

Task	Output	Our example
Classification	One score per class	Ring, plus or X in our new CNN lab
Object detection	Boxes, classes and scores	A sports ball in a video frame
Landmark estimation	Coordinates of named points	21 wrist and finger landmarks
Early prediction	A guess about a later move	RPS from motion before a cutoff

Predict first: which of these outputs would an air-drawing brush actually need?

Technical walkthrough: what learns, and what we program

The small CNN below learns new weights during this lecture. MediaPipe and COCO-SSD reuse pretrained weights. Our pinch threshold, palm dwell, ball association, smoothing, game scoring and graphics are application code. The RPS early predictor fits class centroids from eight numerical features.

These are separate models and responsibilities. Improving a camera gesture rule does not update the vision weights. A visual effect can look convincing while following the wrong object, so evaluation needs a separate human reference.

Professional decision: define the required output before choosing an architecture. A product designer defines the action and failure behavior. A vision engineer chooses measurements. A data scientist defines labels and the test.

02 / INPUT DATA

An image is a tensor of measurements.

A color image has height, width and color channels. A training batch adds another dimension. Our small CNN reads grayscale images with shape [batch, 16, 16, 1].

One toy pixel: 0 = black, 1 = white
One color pixel: [red, green, blue]
Tensor layout used here: [number of images, height, width, channels]

Nearby pixels contain useful structure. A convolution checks local patterns while sharing the same weights across positions.

Technical walkthrough: dimensions, preprocessing and memory

A 640 × 480 RGB frame contains 921,600 color values. Uint8 storage takes about 0.92 MB, while float32 takes about 3.69 MB, before extra tensors. A 30-frame/second stream would contain about 27.6 MB/second of uncompressed uint8 RGB values. Those are raw-array estimates, not encoded video file sizes or measured browser throughput.

Our symbol generator directly produces float32 values in [0,1]. The ball detector wrapper accepts a canvas and performs its own model preprocessing. Apply the normalization expected by the specific model. Dividing every model input by 255 without checking its contract can break inference.

Changing width and height independently can squash a round ball. Ball Remix preserves the frame aspect ratio, places it inside a square, then maps detected boxes back by subtracting the padding.

Engineering check: record shape, dtype, color order, resize rule and model version. A correct neural network with the wrong input contract still gives bad predictions.

Optional context: the complete system and its outputs

START HERE / THE COMPLETE PICTURE

The whole story, before the details.

Computer vision turns an image into useful measurements. A game turns those measurements into an experience. We will connect both sides: understand what a neural network calculates, watch how learning changes it, and build interactions whose successes and failures we can explain.

1 · PixelsImage or decoded video frame 2 · CNN featuresLearned local patterns 3 · Model outputScores, boxes or coordinates 4 · Rules and playBrush, prediction or video effect 5 · EvidenceErrors, coverage and useful behavior

Each arrow changes the representation. Pixels describe brightness and color. Convolution combines nearby values using shared weights; activation and pooling prepare useful features. Later layers turn features into an output. Our application interprets that output and decides what to draw, when to accept a prediction, or which ball to follow. A visible failure can originate at any of these stages.

First identify the question the model answers.

The same camera can support different tasks. Knowing the output makes the formulas and tests meaningful: a class score cannot locate a fingertip, and a box in one frame does not establish an object's identity across a video.

Task	What it returns	Our example
Classification	One score per possible category	Ring, plus sign or X
Object detection	Boxes, categories and scores	A sports ball in a video frame
Landmark estimation	Coordinates of named points	Wrist and finger joints
Early prediction	A guess about a later outcome	RPS from observations before a deadline

Learning changes the weights.

Forward → loss → backward → update → forward again. The forward pass calculates scores. A label lets us calculate a loss. Backpropagation follows the calculation backward, multiplying local derivatives and adding shared paths. The optimizer uses those gradients to change the parameters. A fresh forward pass then measures the effect of that update.

The network itself is feed-forward. Repeating this training cycle creates the learning loop. Training data, the loss and the optimizer together determine what the model can learn.

Using the model calculates an answer.

New input → forward pass → output → application action. During ordinary inference, the learned parameters stay fixed. The model does not run a backward pass simply because a new camera frame arrives. The application may still remember earlier coordinates, smooth movement, maintain a game timer, or associate detections between frames.

That memory belongs to the interaction or tracking logic. It does not automatically train the CNN or prove that the next answer will be correct.

Three kinds of work cooperate in this workshop.

Our 1,283-parameter symbol CNN learns from generated ring, plus and X images. You can inspect its untrained behavior, train it, compare held-out predictions and examine learned activation maps. The hand and ball models are pretrained: they already learned from data before this workshop. We use their estimated landmarks, gesture categories or detection boxes. The smaller early-RPS classifier separately learns class centers from motion features.

Application logic connects those outputs to play. A pinch ratio controls painting, an open-palm dwell enables erasing, a cutoff excludes future hand observations, and tracking links ball detections through time. These rules are inspectable Python explanations in the lecture and working browser code in the demos. Keeping the roles clear helps you locate a bug and explain your own contribution.

One complete learning step, with real calculated numbers.

The separate 15-parameter teaching network starts with a true-class probability of 32.30% and loss 1.13009. Keep its input image fixed, calculate all gradients, and apply one simultaneous SGD update. A new forward pass raises that probability to 53.01% and lowers loss to 0.63460. You will trace why each weight changes. This example explains local optimization on one image; it does not measure human-camera accuracy or performance on unseen images.

Use the ideas to make something people can play.

Filter Detective makes pixel arithmetic visible. Air-art drawing and guessing turns finger coordinates into a shared game, with gesture erasing, hidden camera preview and surprise variations. The final RPS exercise tests what evidence existed before a reveal. The Video Trails take-home project uses Python, CSV coordinates and a deque to render a complete video. The optional Ball Remix comparison shows a pretrained CNN detector, appearance tracking and artwork.

Your explanation should connect a decision to evidence: what entered the system, what was learned, what you programmed, what failed, and how you measured it. Model comparison, data labeling, interface design, game rules and visual effects all offer substantial ways to contribute.

02 / REUSE THE SAME LOCAL CALCULATION

Filters turn pixels into features

Multiply the pixels in a patch by the matching filter weights, then add.
Slide the same weights across the image: the responses form a feature map.
ReLU keeps positive responses; pooling summarizes nearby activations.

Worked examples: convolution, ReLU and pooling

03 / CONVOLUTION

A small filter produces one response per patch.

Multiply matching image values and weights, then add the products. Slide the same filter to the next patch.

Vertical-edge filter

-1  0  1
-1  0  1
-1  0  1

A dark-to-bright patch

0  0  1
0  0  1
0  0  1

Each row contributes 1. The response is 3. Reverse the boundary: the response becomes −3.

A uniform patch gives zero because positive and negative terms cancel. This filter detects a local pattern, rather than recognizing an entire object.

Technical walkthrough: channels, bias, stride and padding

Z[u,v,o] = b[o] + Σ K[a,d,c,o] × X[uS+a−P,vS+d−P,c]

Output size = ⌊ \frac{N + 2 P - F}{S} ⌋ + 1

Learned parameters = (F × F × input channels + 1) × output channels

The sum covers kernel rows, columns and input channels. Each output channel has its own filter and bias. Weight sharing means the same filter checks every location. In the common CNN convention, the displayed kernel is not flipped: the operation is technically cross-correlation.

The calculator assumes a square input/kernel, equal stride and symmetric padding, dilation 1 and a bias per output channel. Our fixed-filter playground deliberately omits bias and learning: its displayed kernel has only nine weights.

Tensor and parameter calculator

Image side Kernel Stride Padding Input channels Filters

Changing input image size changes activation memory. It does not change the number of convolution weights. Changing the channel or filter count changes both.

04 / FILTER DETECTIVE

A response map makes the arithmetic visible.

Open the filter playground. Predict the vertical-edge response before changing anything.
Compare vertical and horizontal boundaries. Click a response cell to inspect the nine products.
Draw a symbol in the pixel grid. Compare an edge filter with blur.
Turn ReLU off. Explain a negative response to a partner.

8 × 8 input3 × 3 valid filter6 × 6 responseReLU2 × 2 max pool3 × 3 map

ReLU replaces negative values with zero. Max pooling keeps the largest value in each block. These operations prepare features for a later decision.

Technical walkthrough: activation, pooling and receptive field

ReLU is max(0,z). Multiple linear operations without a nonlinear activation can collapse into a single linear map. ReLU lets a network represent more complex relationships. It also removes the negative edge response, so different learned filters may respond to opposite orientations.

Pooling shrinks spatial maps and can reduce sensitivity to small shifts. It does not guarantee that a model ignores all position changes. Strided convolutions offer another way to shrink maps. Flattening changes shape into one vector without learning weights.

A receptive field is the input region that can affect an output. For a sequence of layers, maintain jump j and receptive field r: start at 1 and 1, then use r_new = r + (F−1)j and j_new = jS. Padding changes edge behavior and coordinate alignment.

Ablation question: what changes if pooling is removed? Compare parameter count, runtime and held-out errors rather than assuming a larger model is better.

03 / THE CENTRAL LEARNING LOOP

How a CNN learns

Forward: calculate class scores; compare them with the correct label to get a loss.
Backward: calculate how each weight affects that loss.
Update: the optimizer changes weights; a fresh forward pass checks the effect.

ForwardLossBackwardUpdateForward again

Training changes weights. Using a trained model normally runs only the forward calculation.

1 · Watch gradients move backward 2 · Train Ring / Plus / X

1 · Values forward, gradients backward

Follow one fixed image through a small CNN. Values move forward to the loss; gradients move backward to show how the weights affect that loss. An optimizer changes the weights, then the same image runs forward again.

Next: train Ring / Plus / X →

This animation uses a 15-parameter example so every update can be inspected. The next training experiment uses a separate 1,283-parameter CNN. Change the label or learning rate in the detailed calculation.

2 · Train the Ring / Plus / X CNN

05 / TRAIN A CNN HERE

This network learns its filters while you watch.

Predict how an untrained model will behave. Then train the CNN, inspect its held-out confusion matrix, draw a new symbol, and compare a learned activation map.

Local TensorFlow.js. No installation or camera needed. The architecture has 1,283 learned parameters. The synthetic data and initializers use seed 314.

Train a real CNN to classify generated rings, plus signs and Xs. This toy training task is separate from the pretrained hand and ball models. The dataset and initializer use seed 314. This small lab explicitly uses the CPU backend for more consistent teaching results. Runtime and floating-point differences across devices still prevent a guarantee of identical accuracy.

Ready. 384 training images, 96 validation images and 120 held-out test images. No downloads or camera permission.

WATCH THE NETWORK LEARN

Forward → loss → backward → update

The schematic follows the architecture used by this CNN. Its arrows and stage highlights explain the calculation at a readable pace; they are not synchronized to individual training operations. The progress, epoch metrics, weights and activation snapshot below come from the actual training run.

Animation pace

Measured training progress

Ready · 384 images ÷ 32 per batch = 12 updates per epoch. 12 epochs = 144 optimizer updates.

Train to see actual loss, training accuracy and validation accuracy.

Teal → forward values · Purple ← backward gradients · Orange = parameter updates. Feature stacks and moving arrows are schematic; tensor shapes match the implemented model.

Explore the saved training epochsNo snapshots yet

Same feature-map brightness scale across epochs

Epoch 0 is the seeded, untrained network. Snapshots 1–12 become available as each epoch finishes. Scrubbing changes the display without changing the trained model.

Train or inspect the untrained CNN to begin.

Fixed training image

The same image is used for every snapshot so that weight changes can be compared.

Actual first-layer filter

Filter

Actual activation map

Snapshots appear after initialization and each complete epoch.

Train or inspect the untrained CNN to see actual weights. Gradient arrows illustrate the chain rule; they do not display measured gradient magnitudes.

This model uses Adam, which scales the update using its running gradient estimates:

θ_{t + 1} = θ_{t} - η \frac{{\hat{m}}_{t}}{\sqrt{{\hat{v}}_{t}} + ε}

η is the learning rate (0.01 here); m̂ and v̂ are bias-corrected running estimates of gradients and squared gradients; ε stabilizes division.

Live epoch history

The test set remains separate until training finishes.

A held-out generated image

Your hand-drawn test

Drawing needs a mouse or touch input. Keyboard-only route: use Next held-out example and compare its generator label with the CNN guess. This tests generated images, not your own drawings.

A real first-layer activation map

Learned filter

Inspect the untrained CNN or train it, then compare filters. Bright cells show stronger responses after ReLU. Each map uses its own brightness scale.

Every layer, tensor shape and parameter count

Convolution parameters: (kernel height × kernel width × input channels + bias) × output channels. Dense parameters: (input units + bias) × output units. Pooling and flattening add no learned parameters.

CNN code · Python / JavaScript

The Python example trains the matching 1,283-parameter architecture using PyTorch. It includes the seeded symbol generator, independent training/validation/test splits and the full training loop. JavaScript shows the actual generator and TensorFlow.js model used by the browser lab. The frameworks initialize and train independently, so their learned weights and results can differ.

CNN code language Download source

Python setup: python -m pip install numpy torch (Mac: use python3). PyTorch reads images as [batch,1,16,16]; the browser uses [batch,16,16,1]. PyTorch’s CrossEntropyLoss receives raw class logits, with softmax applied when displaying predictions.

The browser training loop uses model.fit(trainX, trainY, {epochs:12, batchSize:32, shuffle:false, validationData:[validationX, validationY]}), then evaluates the 120 held-out test images. Inspect its UI/training implementation in cnn-lab.mjs. Both implementations use seed 314; generated-data scores describe this toy generator.

Technical walkthrough: what each line learns

The first 3 × 3 convolution transforms one grayscale channel into eight feature maps. With a bias per filter it has (3×3×1+1)×8 = 80 parameters. Same padding keeps 16 × 16 resolution.

Pooling gives 8 × 8 × 8. The next valid convolution has twelve 3 × 3 filters and (3×3×8+1)×12 = 876 parameters, producing 6 × 6 × 12. Pooling gives 3 × 3 × 12. Flattening makes 108 values. The three-output dense layer has (108+1)×3 = 327 parameters. Total: 1,283.

p_{k} = \frac{e^{s_{k}}}{\sum_{j} e^{s_{j}}}

Cross-entropy for one labeled image: L = −log(p[true class])
Generic gradient update: weight ← weight − learning_rate × gradient

A true-class score of 0.8 gives loss about 0.223. A score of 0.2 gives loss about 1.609. Backpropagation applies the chain rule to compute gradients for the weights. Adam uses moving averages of gradients and squared gradients to adapt its updates. The lab uses Adam with learning rate 0.01.

A batch contains 32 images. One epoch visits the 384 training images once, giving twelve batch updates. Twelve epochs give 144 updates. Validation measures a separate set after epochs and does not supply weight-update gradients. The 120-image test set is evaluated after training.

Repeatedly changing settings based on the test score turns that set into a validation set. Keep another test set for a final claim. A seeded run makes comparison easier, but GPU kernels and browser versions can still change numerical details.

Python implementation: PyTorch CNN and training loop. Browser source: training integration · JavaScript. API reference: TensorFlow.js training guide.

3 · Next: draw and guess →

04 / PLAY FIRST, THEN CONNECT THE CONTROLS

Try the camera game

A pretrained hand model estimates finger positions; our pinch and palm rules turn them into drawing and erasing.
Open the playground and play a few draw-and-guess rounds. Try a variation and rotate artists.
After playing, return to the formulas to explain why the controls and learning steps work.

1 · Open the camera playground

After playing: hand landmarks, brush rules and failures

07 / AIR-ART PARTY

The game makes model behavior visible.

Open the camera playground. Start the camera and locate the fingertip points.
Pinch to paint. Separate to lift. Hold an open palm briefly to erase, then pinch to return to the pen.
Choose one cooperative group or equal turns for teams. View the secret prompt, hide it, then start drawing.
Try Surprise mix, hidden camera preview, or a harder prompt bank. Rotate artists after each short turn.

Play together as one group, or give each team equal turns. Mouse and touch controls are available when camera tracking is unavailable.

Technical walkthrough: rendering, game gates and stable layout

The drawing board is 800 × 600 pixels. Brush widths are 7, 16 or 30 board pixels. Eraser width is three times the selected brush width. Pointer coordinates scale from the displayed board rectangle to the board’s internal coordinates.

Drawing and erasing rules · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/runtime_examples.py; download the complete file for imports, shared helpers and runnable examples. This Python example calculates stroke settings and alpha compositing. The live page renders through the browser canvas.

Drawing and erasing rules language Download complete source

Loading Python functions…

Loading Python functions…

The game allows brush strokes in free drawing or the timed playing phase. A pause or ready screen lifts the brush. Losing a hand clears the previous stroke point, so reacquiring it does not join a line across the gap.

Mirror mischief flips horizontal control again. Vanishing ink removes alpha every 100 ms while painting is allowed. The prompt banks are editable JSON with Easy objects, Medium scenes and Hard ideas. Selection cycles through the chosen bank. It does not ask an AI to invent a prompt.

Hiding the camera preview changes presentation while tracking continues. Stop camera closes capture. The eraser button keeps a fixed visible label and border thickness. Reserved status-message height prevents model feedback from moving the canvas.

Frontend/UX experiment: count accidental erases and action delay with two palm-dwell settings. Record whether missing-hand recovery leaves an unwanted stroke.

06 / PRETRAINED HAND VISION

Landmarks turn an image into a controller.

Palm detection / hand crop21 landmarksGesture categoryOur brush rule

MediaPipe Gesture Recognizer supplies the hand points and a supported gesture label. The drawing app reads index tip 8 to aim, then compares thumb–index distance with palm width.

pinch ratio = \frac{d (4, 8)}{\max (d (5, 17), 0.001)}

Paint when ratio < 0.35

Technical walkthrough: coordinates, mirroring and the eraser state machine

The drawing rule uses 2D distances in normalized image coordinates. Landmarks are model estimates. World landmarks are a separate coordinate output, rather than calibrated camera-depth measurements. Palm-width normalization helps with apparent scale but perspective and occlusion still matter.

Cursor and brush rules · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/hand_functions.py; download the complete file for imports, shared helpers and runnable examples.

Cursor and brush rules language Download complete source

Loading Python functions…

Loading Python functions…

Observed input	Application state
Pinch	Pen immediately, paint enabled, reset palm dwell
Open_Palm score ≥0.6 for less than 350 ms	Prepare eraser, keep brush lifted
Open_Palm score ≥0.6 for at least 350 ms	Eraser enabled
Other gesture, low score or missing hand	Reset dwell and lift brush

Air art configures one hand, VIDEO mode and CPU execution. It leaves MediaPipe detection/presence/tracking thresholds at defaults. The RPS page separately sets those three thresholds to 0.6. Neither configuration is a measured accuracy statement.

Python brush rules · Browser drawing integration · JavaScript · Official model pipeline

08 / MODEL DETECTIVE

A failure is useful when the test explains it.

Test a steady palm first. Then change one condition: orientation, hidden fingertips, motion speed or distance. Predict what will happen before trying it.

Failure	Possible layer	Next check
No preview	Capture / browser permission	Camera state and device error
Preview works, no points	Model loading or vision inference	Model status, lighting and hand visibility
Points work, accidental erase	Gesture interpretation	Score threshold and dwell time
Tool switch moves the board	Layout feedback	Control dimensions and wrapping

Technical walkthrough: latency, missing data and lifecycle

Latency includes capture, decoding, model execution, application rules and drawing. Frames per second measures throughput and does not by itself measure action delay. A timestamp near inference start is not necessarily the camera exposure timestamp.

Start/stop uses an epoch token: if an asynchronous request finishes after the user canceled it, the app closes that abandoned stream/model. Stop cancels the rendering loop and stops media tracks. A permission timeout also closes a stream that arrives too late.

Resource lifecycle rule · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/runtime_examples.py; download the complete file for imports, shared helpers and runnable examples. The caller supplies a Python resource adapter. This demonstrates the ownership rule; it does not call browser camera APIs.

Resource lifecycle rule language Download complete source

Loading Python functions…

Loading Python functions…

The hand model runs locally using bundled JavaScript, WASM and weights. Camera access depends on browser and operating-system permissions.

Runtime engineering experiment: compare the model version, browser, input resolution, action delay and restart behavior across devices.

05 / AFTER THE DRAWING GAME

Return to the formulas. Explain what happened.

Connect a filter response to the pixels and weights that produced it.
Trace loss and the chain rule into a weight update; use the animation again when a step needs a picture.
Apply the same reasoning to hand geometry, timing, and choosing a useful model output.

Open the detailed learning calculation Connect the math to hand controls

Then try predicting a move before the hand finishes opening.

Next: play the RPS prediction challenge →

06 / DEMO FIRST / FINAL CHALLENGE

Try predicting the next move

Open the final challenge and play a few rounds.
Compare opening the hand early with keeping a fist until the last moment.
Then open the explanation below to connect what happened to the prediction and its score.

1 · Play the RPS demo

2 · After the RPS demo: how prediction and fair testing work

09 / PREDICTING A LATER MOVE

The cutoff defines what the model is allowed to know.

The visible gesture label recognizes the current hand. Our early RPS predictor uses four finger-straightness values and four change rates to guess a later move.

READY: 600 msMOVE: adjustableFreeze features before revealHOLD: 700 msConfirm and score

The default MOVE lasts 700 ms. The camera path asks the player to confirm the actual result. The large scored computer-hand overlay appears after that confirmation. The early guess itself must already be frozen.

Technical walkthrough: eight features and a nearest-centroid classifier

Finger straightness = \frac{MCP-to-tip distance}{total three-segment finger length}

One frame: [index, middle, ring, little]
One round: [four latest values, four change rates per second]

The predictor prefers world landmarks and falls back to image landmarks. Straightness is a geometric proxy, not a clinical joint angle. A straight chain approaches 1. A folded chain has a shorter endpoint distance relative to its path length.

Use only samples in [cutoff−450 ms, cutoff]. Require at least four samples, at least 100 ms between first and last, and a final sample no more than 150 ms before cutoff. Missing evidence produces no prediction.

Feature window function · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/hand_functions.py; download the complete file for imports, shared helpers and runnable examples.

Feature window function language Download complete source

Loading Python functions…

Loading Python functions…

Training needs at least three complete rounds per move. Fit eight training means and population standard deviations, with a scale floor of 0.05. Standardize each feature, calculate a class centroid, then choose the centroid with the smallest squared Euclidean distance. These distances are not calibrated confidence probabilities.

z_{i} = \frac{x_{i} - μ_{i}}{\max (σ_{i}, 0.05)}

centroid[class] = mean standardized feature vector for that class
guess = argmin_class Σ (z[i] − centroid[class,i])²

The scheduled cutoff is start + 600 + movementMs − leadMs. MOVE ranges from 300 to 2500 ms. The first callback reaching the cutoff freezes the input. If that callback already reaches reveal, the app invalidates the guess instead of predicting late. Scheduling delays can shorten the achieved lead.

Python feature functions · Python training and prediction · Python deadline freeze rule

Deadline freeze function · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/hand_functions.py; download the complete file for imports, shared helpers and runnable examples. This is the one-time freeze rule. A caller owns the round state and supplies the current time.

Deadline freeze function language Download complete source

Loading Python functions…

Loading Python functions…

10 / FINAL RPS EXPERIMENT

A fair score needs failures and baselines.

Open the final challenge. Use the artificial preset for today’s short experiment.
Play fresh rounds. Keep a fist until late, then try opening earlier.
Count correct early guesses and rounds with insufficient tracking separately.
Compare another player. Ask which evidence existed before the cutoff.

Uniform random guesses have 33⅓% expected accuracy across three classes. Always choosing the most common training move is another baseline. Game wins and correct early guesses measure different outcomes.

Technical walkthrough: leakage, coverage and the movement fallback

Split complete rounds, players or recording sessions. Adjacent frames from the same round make a poor independent train/test split. Fit normalization on training data only. Keep post-cutoff frames out of all early features and label the final move after input freezes.

Report both valid predictions / attempted rounds and correct guesses / valid predictions. Excluding missing guesses without reporting them can make a system look better than the full user experience.

The fallback lists fingers whose straightness range changed by at least 0.12 across two or more observed frames. It reports movement without inventing a rock/paper/scissors label.

The preset contains nine artificial motion traces. It demonstrates collection, fitting and evaluation flow. It is not a downloaded human-motion dataset, and its score does not establish human-camera prediction accuracy.

Data-science experiment: collect balanced rounds from one player, freeze the scaler and centroids, then test another player. Report class-by-class errors and rejected rounds.

07 / DEMO FIRST / VIDEO PROJECT

Watch a ball become a video effect

Open the take-home video demo and change the moving ball into your own artwork.
Compare supplied positions, simple measurements and CNN-based estimates.
A CNN can learn useful visual features; drawing a trail or sticker from known coordinates needs no CNN.

1 · Try the video effect Optional · Try the ball detector

2 · After the video demo: detection, tracking and rendering

11 / CNN VIDEO PORTFOLIO

Detection supplies boxes. Tracking supplies continuity.

Open Ball Remix when ready. Analyze the clear soccer clip, play the effect, then compare the ping-pong challenge and alternate CNN.

Decoded video frameAspect-preserving paddingMobileNet + SSDBall associationOptional smoothingReplacement graphic

MobileNetV1 is the current default. Both selectable models reuse pretrained COCO weights. Analysis samples the entire video at 10 Hz and includes a final decoded frame. Playback uses stored positions and interpolation. The orange-ball appearance tracker can supply separately labeled pixel measurements when the CNN misses.

Technical walkthrough: the actual detector call and coordinate transform

The frame is resized to at most 960 pixels wide while preserving aspect ratio. Center it in a black square with side ceil(1.3 × max(width,height)). The square input preserves ball proportions during model resizing. After detection, subtract padding from each box’s x and y origin.

Detector postprocessing function · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/video_functions.py; download the complete file for imports, shared helpers and runnable examples. The detector is an explicit supplied callable. This function does not load the browser COCO-SSD bundle in Python.

Detector postprocessing function language Download complete source

Loading Python functions…

Loading Python functions…

The Python examples separate these stages: detect_balls() handles the application threshold, and remove_padding() handles coordinate correction. The supplied detector must return objects with class, score and bbox fields. The live Ball Remix page loads its bundled TensorFlow.js/COCO-SSD model and supplies the padded browser canvas.

The returned box is [x, y, width, height]. The detector wrapper handles model input resizing, inference, box conversion and suppression of overlapping candidates. Its output class is the broad “sports ball” category.

SSD predicts class scores and offsets for many default boxes across feature maps. Non-maximum suppression reduces redundant overlapping detections. That is different from deciding which ball remains the same object across frames.

MobileNetV1 uses depthwise spatial filters followed by pointwise 1 × 1 channel mixing. Ignoring biases and normalization, a regular F × F layer uses F² Cin Cout weights, while the separable block uses F² Cin + Cin Cout. A 3 × 3, 32-to-64-channel example gives 18,432 versus 2,336 weights. That arithmetic illustrates the block, not the full model parameter count.

The offline seek helper waits for seeked and checks readyState ≥ 2 before reading decoded pixels. Assigning currentTime alone is asynchronous. Offline analysis does not wait for a new compositor presentation: identical or black frames may not trigger requestVideoFrameCallback again. The live preview can use that callback to align overlays with presented video frames.

Actual Ball Remix implementation · COCO-SSD wrapper · SSD paper · MobileNet paper

Technical walkthrough: association, interpolation and the smoothing trade-off

In the CNN-only tracking rule, the first accepted sample chooses the largest ball box. Later samples choose the accepted ball center closest to the previous point; a missing detection resets association. With appearance tracking enabled, a colorful accepted CNN box seeds a color prototype. Matching connected pixels can then supply an appearance position when the CNN misses. A sample with neither source resets association. This remains a simple tracker without a learned appearance embedding, velocity model, distance gate or multiple-object identity management. Inspect the color measurement and source selection.

center = [x_{box} + \frac{width}{2}, y_{box} + \frac{height}{2}]

smoothed = alpha × previous + (1−alpha) × current
between two samples: position(t) = left + fraction × (right−left)

Smoothing affects x/y positions and can add delay. Interpolation requires valid neighboring points and a gap no greater than 0.3 seconds. It does not cross a row with no position from either source. The final point may hold for less than 0.101 seconds. Interpolated coordinates are calculations between measurements, not extra CNN observations.

Smoothing can lag behind motion

Previous-position weight 0.5

The overlay radius is 0.55 × max(box width, box height), multiplied by the size slider. It covers the image in 2D. It does not segment the ball, reconstruct a hidden foot or inpaint the original pixels.

CHOOSE A METHOD / DEFINE THE JOB FIRST

A CNN is one tool. Start with the simplest method that fits.

A video effect combines three separate jobs: find an object, follow its position, and draw something there. Each job can use a different method. If a CSV already supplies the coordinates, Python and OpenCV can draw a trail without finding the ball or running a neural network.

Job	A useful method	What it assumes	What to test
Draw supplied positions	CSV + deque + OpenCV	Frame numbers and coordinates already exist.	Frame alignment, missing rows and the rendered effect.
Locate one colorful ball	Color matching and connected components	The ball has a distinctive visible color.	Changed lighting, blur and similar-colored objects.
Measure local contrast	A hand-designed kernel	The desired pattern can be expressed by a fixed local rule.	Orientation, scale and distracting edges.
Detect varied objects	A trained model such as a CNN	Its training data supports the new inputs.	Localization errors, missed objects and false detections.

The same-looking trail does not prove that two systems solved the same detection problem. Compare the source of the coordinates, accuracy, missing outputs, runtime and behavior under changed conditions. A simpler rule can be the better choice for a controlled task; a learned model can handle patterns that are difficult to specify by hand.

Try the code exercises Explore kernels Build the Python video project

08 / FINISH WITH EVIDENCE

Choose the tool. Explain the evidence.

State what you changed and which problem it solves.
Compare a simple baseline; include errors and missing outputs.
Show a new example and one failure you can explain.

Project options, evaluation measures and portfolio evidence

12 / EVIDENCE AND PROJECT HANDOFF

Each role owns a decision and evidence.

The portfolio project combines an original effect, one technical improvement, a labeled evaluation and a second video. Open the coding project.

Role / interest	Concrete contribution	Evidence to show
ML / vision engineer	Compare models, pooling or association	Held-out errors, runtime and a documented trade-off
Data scientist	Define splits, labels and baselines	Confusion matrix, misses, false detections and coverage
Frontend / runtime engineer	Improve frame synchronization and interaction	Action delay, stable layout, clean stop/restart
Product / design / non-coder	Define the game and failure behavior	User observations, annotated cases and task completion

Show what learned, what you programmed, what failed, and what evidence supports the claim.

Technical walkthrough: localization metrics and a box-overlap experiment

Ball Remix implements normalized center error on labeled visible-ball samples where a detection exists. It separately counts visible-ball misses and detections on absent-ball frames. It does not yet compute average precision, IoU or identity-switch metrics.

Normalized center error = \frac{distance(predicted center, human center)}{image diagonal}

IoU = \frac{intersection area}{union area}

Box overlap is different from confidence

Shift detector box

The IoU calculator is a teaching example added to this lecture. It does not silently replace the take-home app’s metric. For a stronger project, implement and validate box labels, identity errors and latency measurements explicitly.

A wrong ball can still produce high coverage and a high class score. Evaluate on a second clip after freezing your design. Keep the model and media licenses with the project.

AFTER THE MAIN TOPICS / FULL TECHNICAL STUDY

Return to any topic for the complete derivation.

Open the formula reference, worked calculations or technical reference below. Topic links open the exact section for you. Python remains the default in every function panel; the same fullscreen network and training examples remain available.

Review roadmap: inspect the formulas and implementation piece by piece

Choose a topic to inspect its reasoning. Each worked derivation defines its quantities, substitutes numerical values, states its assumptions and links to code. Open a checkpoint to compare your reasoning.

Piece to inspect	What to explain or verify	Open the detail
1 · Notation and shapes	Column vectors, class-by-feature weights, tensor dimensions and filter indices.	Notation · Shapes and parameters
2 · Local calculus	Multiply along one path, add branches, then apply one simultaneous update.	Local chain rule
3 · Batch learning	Where the mean-loss factor enters, broadcast bias and dense matrix-gradient shapes.	Two-example calculation
4 · Reverse the CNN	Shared kernel gradients, input-gradient overlap, activation gates and pooling ties.	Input derivatives · Gates and pooling
5 · Class scores and loss	Every softmax Jacobian path, stable cross-entropy and the normalized-label assumption.	Full loss derivative
6 · Run the update	Follow one parameter in the animation; connect it to NumPy and the PyTorch batch loop.	Animation · Training state and code
7 · Hand and timing contracts	World versus image coordinates, measured features and the two movement windows.	Coordinates · Timing
8 · The portfolio pipeline	Detector suppression, saved records, app thresholds, identity and interpolation.	Detector stages · Coding project

Formula deep dive: from one patch to one gradient

Follow the actual numbers. Each step defines its symbols, calculates a worked example, and connects the equation to code. Change inputs and weights to test your understanding. Open any topic for its complete calculation and Python functions.

MATH 01 / A MAP BEFORE THE SYMBOLS

One image becomes a loss through a chain of functions.

We will calculate forward values, differentiate the same functions, and use those derivatives to change weights. Every numerical example here is a deliberately small teaching example. The separate 1,283-parameter symbol CNN still runs with TensorFlow.js.

X: imageZ = convolution + biasA = ReLU(Z)h: pooled / flattened featuress: class logitsp: softmax scoresL: label-dependent loss

L = CE (softmax (Dense (Flatten (Pool (ReLU (Conv (X)))))), y)

Symbol	Meaning and shape
X, K, b	Input image tensor, convolution kernel weights, convolution biases
n, u, v, c, o	Batch item, output row/column, input channel, output channel; indices start at zero
H, W, C_in, C_out	Image height/width and channel counts
Z, A, h, s	Convolution pre-activation map, ReLU activation map, flattened column vector, class-logit column vector
W, c	Dense matrix with one class per row and one input feature per column; dense bias column vector
F_h, F_w, K	Filter height/width, and the learned kernel tensor. C_out counts filters.
y, p, L, θ	One-hot true label, predicted class scores, scalar loss, all trainable weights/biases
$\frac{\partial L}{\partial θ}$ , η	Local sensitivity of loss to a parameter, and learning rate

One vector convention throughout this lecture

Features and class scores are column vectors. The dense matrix has shape classes × features: s = Wh + c. In the miniature below, h has 4 entries, W is 2×4, and s, c, p, y each have 2 entries. A subscript such as W_kj means class k, feature j. Let g = $\frac{\partial L}{\partial s}$ = p−y for softmax with cross-entropy. Juxtaposition is matrix multiplication; ⊙ is elementwise multiplication. Bold letters in matrix equations denote vectors or matrices; indexed entries are scalars.

s = W h + c \frac{\partial L}{\partial W} = g h^{T} \frac{\partial L}{\partial h} = W^{T} g

Row-vector notation: the equivalent calculation is sᵀ = hᵀWᵀ + cᵀ. For a row-batch feature matrix H it becomes S = HWᵀ + cᵀ, with the bias broadcast over rows. These are transposes of the column-vector calculation above.

F_h and F_w denote filter height and width; C_in and C_out denote input and output channel counts. K names the learned kernel, and D denotes dilation.

Read a derivative in plain English: if $\frac{\partial L}{\partial w}$ is +0.4, a sufficiently small increase of 0.01 in w changes loss by approximately +0.004, holding the other parameters fixed. It is a local slope, not the amount a weight must change.

MATH 02 / EVERY MULTIPLICATION

Convolution reuses one set of weights across many patches.

Z_{n, u, v, o} = b_{o} + \sum_{a = 0}^{F_{h} - 1} \sum_{d = 0}^{F_{w} - 1} \sum_{c = 0}^{C_{in} - 1} K_{a, d, c, o} X_{n, u S + a - P, v S + d - P, c}

All indices start at zero. Here a and d index kernel rows/columns, S is stride and P is zero padding. This sum assumes dilation D = 1; with general dilation replace a and d in the input coordinates by Da and Dd. Out-of-bounds padded pixels equal zero. One output channel uses its own kernel and bias, shared across all spatial locations.

CNN convention: the forward operation is cross-correlation: the stored kernel slides over the image without flipping. Many libraries call this “convolution.” Mathematical convolution flips the kernel first; either forward convention can learn filters, but its backward indices must match. The input gradient below adds contributions to the exact input pixels read in this equation.

Editable convolution: input, weights, products and output

Edit image values in [0,1] or kernel weights in [−3,3]. Select an output cell to see the exact patch that produced it. The highlighted image patch and nine products stay linked.

Filter preset Bias 0 Apply ReLU

Input X: 5 × 5

Kernel K: 3 × 3

Output: 3 × 3

All nine products for this selected patch

The initial patch is [[0,0,1],[0,0,1],[0,0,1]]. Each row contributes 1, giving 3. The rightmost patch is uniform, so its response is 0. Reversing brightness makes the first response −3; turning ReLU on changes that activation to zero.

Multiple channels: if a filter’s red, green and blue patch sums are 2, −1 and 3, and its bias is 0.5, the output is 4.5. The channels contribute to one output value. They do not automatically become three separate decisions.

MATH 03 / SHAPES, PARAMETERS AND COMPUTATION

Dimensions explain what the code allocates and learns.

H_{out} = ⌊ \frac{H + 2 P - D (F_{h} - 1) - 1}{S} ⌋ + 1

D is dilation. Use the same formula for width; F_h is the kernel height. For dilation D = 1, the output-size expression before taking the floor is $\frac{N + 2 P - F}{S} + 1$ . The floor in the general formula discards trailing partial windows; a layer that requires an exact stride fit instead rejects a non-integral result. Check the framework’s padding policy. For N=6, F=3, S=2, P=0, the full windows start at 0 and 2, giving two outputs and an unused trailing row/column. Never silently round up. Our examples use D = 1. For a 16-pixel side, kernel 3, stride 1 and padding 1: $⌊ \frac{16 + 2 - 3}{1} ⌋ + 1 = 16$ . With zero padding it becomes 14. “Same” padding at stride 1 preserves size; at larger stride, frameworks can require asymmetric padding.

parameters = (F_{h} F_{w} C_{in} + 1) C_{out}

Actual symbol-CNN layer	Output per image	Learned parameters
3 × 3, 1 → 8 channels, same	16 × 16 × 8	(9×1+1)×8 = 80
2 × 2 pooling, stride 2	8 × 8 × 8	0
3 × 3, 8 → 12 channels, valid	6 × 6 × 12	(9×8+1)×12 = 876
Pooling and flatten	108	0
108 → 3 dense outputs	3	(108+1)×3 = 327
Total		1,283

Parameters versus compute: an ordinary convolution needs approximately H_outW_outC_outF_hF_wC_in multiply-accumulates per image. Our two convolution layers use 18,432 and 31,104, and the dense layer uses 324. That totals 49,860 before bias, activation and pooling overhead. This arithmetic is not a measured runtime.

Receptive field: start with field r = 1 and jump j = 1. For kernel extent F and dilation D = 1, each layer gives r′ = r+(F−1)j and j′ = jS. For conv3, pool2, conv3, pool2, the fields become 3, 4, 8, 10 and the final jump is 4. A final pooled cell can depend on a 10 × 10 input region, including padded positions near edges.

The weights alone take 1,283×4 = 5,132 bytes in float32. Training also needs gradients, optimizer moments and activations. Input resolution changes activation storage even when convolution parameter count remains unchanged.

MATH 04 / THE BACKWARD RULE OF EACH GATE

ReLU gates gradients. Max pooling routes them.

A = \max (0, Z) \frac{\partial L}{\partial Z} = \frac{\partial L}{\partial A} ⊙ 1 [Z > 0]

⊙ means elementwise multiplication. The bracket is an indicator: one where the condition is true, zero elsewhere. At Z = 0 ReLU is not differentiable; the worked engine chooses derivative zero.

Forward input Z	ReLU output A	Incoming $\frac{\partial L}{\partial A}$	Outgoing $\frac{\partial L}{\partial Z}$
[−2, 0, 3]	[0, 0, 3]	[0.5, 0.2, −1]	[0, 0, −1]

Max-pool forward

\max ([\begin{matrix} 1 & 3 \\ 2 & 0.5 \end{matrix}]) = 3

Max-pool backward

g = 0.4 \Rightarrow [\begin{matrix} 0 & 0.4 \\ 0 & 0 \end{matrix}]

The unique winning cell receives the incoming derivative. Other cells get zero for this window. Overlapping pooling windows can send multiple contributions to the same input, which must add. Ties are nondifferentiable points and need an implementation convention. Flattening simply reshapes the forward values and reverses that reshape for the gradient.

Why nonlinearities matter: two affine layers without an activation collapse: W₂(W₁x+b₁)+b₂ = (W₂W₁)x + (W₂b₁+b₂). ReLU prevents that collapse and changes which paths carry gradients.

MATH 05 / SCORES, LOSS AND THEIR DERIVATIVES

Cross-entropy and softmax simplify to p − y.

s_{k} = c_{k} + \sum_{j} h_{j} W_{k j}

p_{k} = \frac{e^{s_{k} - m}}{\sum_{j} e^{s_{j} - m}} m = \max_{j} s_{j}

Subtracting the same constant changes neither the mathematical probabilities nor their ordering. It prevents overflow when logits are large. Logits [2,1,0] and [1002,1001,1000] both give approximately [0.665241, 0.244728, 0.090031].

L = - \sum_{k} y_{k} \log p_{k} = \log (\sum_{j} e^{s_{j}}) - s_{true}

For one-hot labels, exactly one y entry is 1. With [2,1,0] and true class 1, L = −log(0.244728) = 1.407606. The implementation uses the shifted log-sum-exp form, keeping loss finite even when exp of an unshifted logit would overflow.

\frac{\partial p_{k}}{\partial s_{j}} = p_{k} (δ_{k j} - p_{j})

δ_kj is 1 when k = j and 0 otherwise. Combine this Jacobian with $\frac{\partial L}{\partial p_{k}} = - \frac{y_{k}}{p_{k}}$ . The terms cancel to give the key result below, provided the label vector sums to one.

\frac{\partial L}{\partial s_{j}} = p_{j} - y_{j}

Change the logits and the correct label

s₀ s₁ s₂ True class

p₀ p₁ p₂

The complete 3 × 3 softmax Jacobian

For the default example, $\frac{\partial L}{\partial s}$ ≈ [0.665241, −0.755272, 0.090031]. Lowering loss calls for raising the correct-class logit relative to the others. Their gradients sum to zero, matching softmax’s invariance to a common logit shift.

MATH 06 / CHAIN RULE THROUGH THE WHOLE CNN

A shared weight receives a sum of all the paths that use it.

Let g_k = $\frac{\partial L}{\partial s_{k}}$ = p_k−y_k. For the dense layer s_k = c_k + Σ_jh_jW_kj, differentiating gives:

\frac{\partial L}{\partial W_{k j}} = h_{j} g_{k} \frac{\partial L}{\partial h_{j}} = \sum_{k} W_{k j} g_{k}

The bias gradient is g_k. Reshape the feature gradient backward through flattening, route it through pooling, then multiply by the ReLU mask. Call the resulting convolution pre-activation gradient Δ = $\frac{\partial L}{\partial Z}$ . With zero-based indices and D = 1, sum below over n=0,…,B−1, u=0,…,H_out−1 and v=0,…,W_out−1. For dilation D, the input offsets become Da and Dd.

\frac{\partial L}{\partial K_{a, d, c, o}} = \sum_{n, u, v} Δ_{n, u, v, o} X_{n, u S + a - P, v S + d - P, c}

One kernel entry is reused at many locations, so each use contributes to the same parameter gradient. The convolution bias gradient is the sum of Δ over batch and spatial positions for that output channel. For the input gradient, scatter each Δ×K contribution back to the input coordinate it read and add overlapping contributions.

A four-use numerical example: suppose a shared weight reads input pixels [1,2,0,3] at four output positions, whose Δ values are [0.2,−0.1,0.4,0.3]. Its gradient is 1×0.2 + 2×(−0.1) + 0×0.4 + 3×0.3 = 0.9. Updating once with 0.9 is different from updating the shared weight after every use during a forward pass.

For mean batch loss, include a factor $\frac{1}{B}$ once. If Δ already comes from a mean-reduced loss, it already contains that scaling. Dividing again would shrink every update incorrectly.

Background: Stanford CS231n convolution notes and chain-rule notes. The arithmetic and interactive engines in this lecture are local teaching implementations.

MATH 07 / A COMPLETE NUMERICAL TRAINING STEP

Forward, backward and update the same shared weights.

Matrix orientation: the workbench uses the same column-vector convention as the equations above: s = Wh + c, W is 2×4, and h is 4×1. Shared kernel K is 2×2. Follow forward values to the right and gradients back to the left.

Feed-forward computation inside a training loop

The network computes left to right. Backpropagation sends derivatives through the same graph in reverse, using the cached forward values. The optimizer then changes the weights. A new forward pass uses those new weights to calculate new scores and a new loss. The architecture remains a feed-forward CNN; the training iteration is the loop.

θ_{t} \overset{forward}{\to} L (θ_{t}) \overset{backward}{\to} \nabla_{θ} L (θ_{t}) \overset{update}{\to} θ_{t + 1} \overset{forward again}{\to} L (θ_{t + 1})

Showing weights one by one is an explanation of a proposed simultaneous update. Each uses a gradient evaluated at θ_t. Only the commit step creates θ_t+1. During inference the network runs only the forward calculation; it does not learn from the displayed prediction.

A complete CNN gradient, small enough to calculate

This exact fifteen-parameter network uses a fixed 3×3 image, one shared 2×2 filter, ReLU, flatten and a two-output dense classifier. There is no pooling in this tiny derivation, so every spatial contribution stays visible. It is a single-image, plain-SGD teaching engine. The separate 1,283-parameter symbol lab uses Adam.

Revisit the forward and backward animation ↑

True class Kernel K[0,0] Learning rate η

Changing a control restarts the derivation. Displayed numbers are rounded to six decimals; calculations keep JavaScript double precision. Image entries are deliberately small integer intensities so you can check the arithmetic by hand. They are not the normalized images used by the other CNN lab.

Z_{r c} = b + \sum_{u = 0}^{1} \sum_{v = 0}^{1} K_{u v} X_{r + u, c + v}

\frac{\partial L}{\partial K_{u v}} = \sum_{r = 0}^{1} \sum_{c = 0}^{1} \frac{\partial L}{\partial Z_{r c}} X_{r + u, c + v}

Finite-difference check: $\frac{L (θ + ε) - L (θ - ε)}{2 ε}$ . All fifteen derivatives are checked independently. At an exact ReLU kink, the ordinary derivative does not exist; this engine chooses zero for backpropagation there.

Exact backpropagation code · Python / JavaScript

Run the same fifteen-parameter example in Python with NumPy, including every gradient, one simultaneous SGD update and a numerical gradient check. Choose JavaScript to inspect the numerical engine used by the animation. Both use the lecture’s column-vector convention.

Backpropagation code language Download source

Python setup: install NumPy in your own environment with python -m pip install numpy (Mac: python3 -m pip install numpy). Run commands from the workshop folder.

Inspectable engine: Python forward, backward and SGD functions. No pretrained model, hidden trajectory or simulated learning curve supplies these numbers.

MATH 08 / AN UPDATE IS AN ALGORITHM

SGD follows the gradient. Adam also tracks its history.

θ_{t + 1} = θ_{t} - η g_{t}

At weight 0.5, gradient +0.2 and learning rate 0.1, SGD gives 0.48. A negative gradient would increase that weight. A large step can cross a nonlinear boundary or increase the loss. Backpropagation computes a direction; it does not choose the step size.

m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t} v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}

{\hat{m}}_{t} = \frac{m_{t}}{1 - β_{1}^{t}} {\hat{v}}_{t} = \frac{v_{t}}{1 - β_{2}^{t}}

θ_{t} = θ_{t - 1} - η \frac{{\hat{m}}_{t}}{\sqrt{{\hat{v}}_{t}} + ε}

These operations act elementwise. With zero initial moments, g₁ = 0.2, β₁ = 0.9 and β₂ = 0.999, the first moments are m₁ = 0.02 and v₁ = 0.00004. Bias correction gives m̂₁ = 0.2 and v̂₁ = 0.04, so a conceptual Adam step with η = 0.01 is about 0.01.

The running symbol CNN calls tf.train.adam(0.01). The bundled runtime reports β₁ = 0.9, β₂ = 0.999 and ε = 10⁻⁷. The displayed form matches the bundled optimizer: it bias-corrects both moments, then divides by the square root of the corrected second moment plus epsilon. The worked SGD engine intentionally exposes a simpler update rather than claiming to reproduce the full optimizer.

L_{batch} = \frac{1}{B} \sum_{n = 1}^{B} L_{n}

With 384 training images and batch size 32, each epoch has 12 optimizer updates. Twelve epochs have 144. Validation computes loss/scores without updating weights. The generated-data test score describes this generator, not every possible drawing.

MATH 09 / A CALCULATION CAN BE CHECKED

Finite differences test whether the gradient code is right.

\frac{\partial L}{\partial θ_{i}} \approx \frac{L (θ + ε e_{i}) - L (θ - ε e_{i})}{2 ε}

e_i changes only parameter i. Compute the loss twice, once slightly above and once below its current value. Compare that central-difference slope with the analytic backprop result. This epsilon is a perturbation for testing, separate from Adam’s denominator epsilon.

The local tests check every convolution weight, dense weight and bias in the worked engine. The softmax Jacobian and p−y derivative also have numerical checks. Choose points away from ReLU zeros and pooling ties: at those kinks a unique derivative does not exist, so a finite difference may disagree with a chosen one-sided convention.

Prediction question: why not use a very tiny perturbation? Floating-point subtraction can erase the loss difference. A large perturbation measures a wider nonlinear region. Gradient checks are debugging tools; training with a central difference for every weight would require many extra forward passes.

Inspect the executable calculations: Python forward / softmax / backward / update functions, and their tests in the workshop’s tests/ folder.

Worked derivations: follow every local derivative, shared path and training state

Use these six extended-study pieces to audit the reasoning behind the live examples. Each piece states the assumptions, substitutes numbers and identifies the implementation to inspect. The two-example regression and dense-layer examples are separate teaching calculations; the convolution input-gradient values come from the exact miniature CNN on this page.

DERIVATION 1 / FOLLOW ONE EDGE, THEN ADD THE PATHS

A backward message is an incoming gradient multiplied by a local derivative.

Suppose a scalar intermediate value v depends on x and the final loss depends on v. The forward pass saves the values needed to evaluate the local derivative. The backward pass asks how a small change in x would change that same loss. If x influences several downstream values v_j, every path contributes.

\frac{\partial L}{\partial x} = \sum_{j = 1}^{M} \frac{\partial L}{\partial v_{j}} \frac{\partial v_{j}}{\partial x}

One shared-weight example. Let r₁=wx₁ and r₂=wx₂, with one shared scalar w. Take x₁=1, x₂=3, target values y₁=1 and y₂=0, and initial w=0.2. Use a sum of two squared-error terms; there is no batch-mean factor in this separate example.

L = \frac{1}{2} {(r_{1} - y_{1})}^{2} + \frac{1}{2} {(r_{2} - y_{2})}^{2}

Path	Forward value	Incoming gradient	Local derivative	Contribution to shared w
w → r₁ → L	r₁=0.2	r₁−y₁=−0.8	x₁=1	−0.8×1=−0.8
w → r₂ → L	r₂=0.6	r₂−y₂=0.6	x₂=3	0.6×3=1.8

\frac{\partial L}{\partial w} = −0.8 + 1.8 = 1; w_{new} = 0.2 - 0.1 \times 1 = 0.1

The loss starts at 0.5. A fresh forward pass at w=0.1 gives predictions 0.1 and 0.3, and loss 0.45. The two paths prefer opposite directions; their sum defines the update. This is the same accumulation principle used when a convolution kernel is shared across image positions.

Read the units: a gradient is loss change per unit change in its variable. Multiplying the incoming loss-per-output slope by an output-per-weight slope produces a loss-per-weight slope. The learning rate then sets the size of the parameter change.

Checkpoint: why save the forward values?

The derivative of wx with respect to w needs the x used by that forward pass; ReLU needs the saved sign of its preactivation. Updating a weight before all its gradient contributions are calculated mixes parameter snapshots and no longer differentiates the original loss.

Inspect the accumulation in Python backward(). The two-output squared-error numbers above are an independent teaching example.

DERIVATION 2 / SHAPES MAKE THE BATCH FORMULA CHECKABLE

Average the loss once, then carry that scaling through every gradient.

Use D input features, C classes and B examples. Arrange feature columns in H∈ℝ^D×B, weight rows in W∈ℝ^C×D, and bias c∈ℝ^C. Scores S, probabilities P and one-hot labels Y have shape C×B. Softmax operates separately within each example’s column. Let 1_B be a column of B ones.

S = W H + c {1_{B}}^{T}, G = \frac{P - Y}{B}

\frac{\partial L}{\partial W} = G H^{T}, \frac{\partial L}{\partial c} = G 1_{B}, \frac{\partial L}{\partial H} = W^{T} G

Quantity	Shape calculation	Resulting shape
WH	(C×D)(D×B)	C×B
GHᵀ	(C×B)(B×D)	C×D: same as W
WᵀG	(D×C)(C×B)	D×B: same as H

Substitute a two-feature, two-class, two-example batch. This smaller dense layer makes every entry visible. Use zero bias and labels class 0 for the first example, class 1 for the second.

H = [\begin{matrix} 1 & 3 \\ 2 & 4 \end{matrix}], W = [\begin{matrix} 0.1 & 0.2 \\ -0.1 & 0.05 \end{matrix}], S = [\begin{matrix} 0.5 & 1.1 \\ 0 & -0.1 \end{matrix}]

Example	Class probabilities	True class	Mean-loss score-gradient column
h₁=[1,2]ᵀ	[0.622459, 0.377541]ᵀ	0	[−0.188770, 0.188770]ᵀ
h₂=[3,4]ᵀ	[0.768525, 0.231475]ᵀ	1	[0.384262, −0.384262]ᵀ

\frac{\partial L}{\partial W} \approx [\begin{matrix} 0.964017 & 1.159509 \\ -0.964017 & -1.159509 \end{matrix}], \frac{\partial L}{\partial c} \approx [\begin{matrix} 0.195492 \\ -0.195492 \end{matrix}]

For example, the upper-left weight gradient is (−0.188770)×1 + 0.384262×3 ≈ 0.964017. The batch mean loss is approximately 0.968680. G already contains the factor one half, so GHᵀ must not be divided by B a second time.

Translate to row-batch code. If examples are rows, H_row=Hᵀ has shape B×D. Then S_row=H_rowWᵀ+cᵀ. The parameters still have the lecture’s class-by-feature storage; the batch axis has moved. In the Python examples, PyTorch’s dense layer accepts a row batch and stores its weight as classes×features.

Checkpoint: when should the factor one over B appear?

It appears once when the objective averages B example losses. Starting with G=(P−Y) scaled by that factor already includes it in every downstream derivative. Starting with unscaled per-example score gradients requires averaging the accumulated parameter gradient later. Both routes describe the same mean objective.

The scalar miniature is B=1. Inspect CrossEntropyLoss and the training loop for mean-reduced batch training, and NumPy backward for the one-example outer product.

DERIVATION 3 / REVERSE EVERY READ OF THE INPUT

The input gradient adds overlapping patches back into image coordinates.

For the live miniature, use one input channel, one shared 2×2 kernel, stride 1, no padding and no dilation. Its 3×3 image produces a 2×2 preactivation map. Write Δ_r,c for the loss gradient arriving at that preactivation. The forward read is X_r+u,c+v; its backward contribution goes to exactly that same coordinate.

\frac{\partial L}{\partial X_{i, j}} = \sum_{u = 0}^{1} \sum_{v = 0}^{1} K_{u, v} Δ_{i - u, j - v}

Here out-of-range Δ entries are defined as zero. The equivalent implementation loops over output position (r,c) and kernel offset (u,v), then adds Δ_r,cK_u,v to the input-gradient cell at (r+u,c+v).

Δ \approx [\begin{matrix} 0.270798 & -0.270798 \\ -0.135399 & 0.406198 \end{matrix}], K = [\begin{matrix} 0.2 & -0.1 \\ 0.1 & 0.3 \end{matrix}]

The central image pixel belongs to all four receptive fields. Trace each read:

Output position	Kernel entry that read X₁,₁	Contribution
(0,0)	K₁,₁=0.3	0.270798×0.3 ≈ 0.081240
(0,1)	K₁,₀=0.1	−0.270798×0.1 ≈ −0.027080
(1,0)	K₀,₁=−0.1	−0.135399×(−0.1) ≈ 0.013540
(1,1)	K₀,₀=0.2	0.406198×0.2 ≈ 0.081240

\frac{\partial L}{\partial X_{1, 1}} \approx 0.148939, \frac{\partial L}{\partial X} \approx [\begin{matrix} 0.05416 & -0.08124 & 0.02708 \\ 0 & 0.148939 & -0.121859 \\ -0.01354 & 0 & 0.121859 \end{matrix}]

Why “rotate the kernel” sometimes appears in derivations. The forward CNN operation here is unflipped cross-correlation. Under these stride-one assumptions, its input gradient can be expressed as full mathematical convolution of Δ with K, or as a padded cross-correlation of Δ with a 180-degree rotated K. The direct scatter loop above already implements the correct coordinate reversal; it needs no additional flip.

With a zero-padded input border, form the gradient of the padded input and remove the padded border afterward. Larger strides insert gaps between output-gradient positions in a transposed-operation view. Multiple output channels contribute sums to the same input channel. The coordinate equation above is intentionally limited to the exact single-channel miniature.

Checkpoint: are input pixels updated by this optimizer?

No. This derivative describes the loss sensitivity to each fixed input pixel. The teaching optimizer updates K, its bias, dense W and dense c. The input-gradient heatmap is an explanation of sensitivity. Finite differences on all nine input entries agree with this map to approximately 1.33×10⁻¹¹ at the default smooth point; displayed entries are rounded to six decimals.

Inspect the input-gradient accumulation loop in Python backward().

DERIVATION 4 / KEEP EVERY OUTPUT PATH IN THE SUM

Derive p − y by multiplying the loss gradient by the whole softmax Jacobian.

Let s contain C real class scores and p=softmax(s). Let y be a fixed, normalized label vector with entries summing to one; the live example uses a one-hot label. Let δ_kj be one when k=j and zero otherwise. Each score affects every probability, so the chain rule sums across output classes.

\begin{array}{l} \frac{\partial L}{\partial s_{j}} = \sum_{k = 0}^{C - 1} \frac{- y_{k}}{p_{k}} p_{k} (δ_{k, j} - p_{j}) \\ = - y_{j} + p_{j} \sum_{k = 0}^{C - 1} y_{k} \\ = p_{j} - y_{j} \end{array}

For scores [2,1,0] and true class 1, p≈[0.665241,0.244728,0.090031]. Only the true-class probability has a nonzero cross-entropy derivative with respect to p, but it depends on all three scores.

\frac{\partial L}{\partial p_{1}} = \frac{- 1}{p_{1}} \approx −4.086161

Score being changed	Derivative of p₁ with respect to that score	Multiply by −4.086161	Result
s₀	−0.162803	(−4.086161)(−0.162803)	0.665241
s₁	0.184836	(−4.086161)(0.184836)	−0.755272
s₂	−0.022033	(−4.086161)(−0.022033)	0.090031

The positive gradients on the other scores and the negative gradient on the true-class score explain the direction of a small descent step. The gradient entries sum to zero because adding the same constant to every score leaves softmax unchanged. Its Jacobian has the same shift-invariant direction.

Numerical implementation. The engine calculates the loss using shifted log-sum-exp directly from scores. This keeps the loss finite when a probability underflows numerically to zero. The probability-space cancellation above is a mathematical derivation; computing a huge reciprocal of a tiny displayed probability is unnecessary in the implementation.

Checkpoint: does this work with soft labels?

Yes, for fixed nonnegative soft labels whose entries sum to one, the same derivation gives p−y. With labels summing to q, the algebra instead gives qp−y. The supplied UI and miniature code accept a class index and construct one-hot labels; they do not expose a soft-label mode.

Compare Python forward() and backward() with the framework loss in the Python training example.

DERIVATION 5 / ROUTING, OVERLAP AND NONDIFFERENTIABLE POINTS

A gate decides which paths receive a gradient; overlap decides where paths add.

Consider a 2×2 max pool with stride 1 on the following 3×3 activation map. Every pooling window has the same unique winner, the central 9. This overlapping example differs from the live symbol CNN’s non-overlapping stride-two pooling.

A = [\begin{matrix} 1 & 2 & 3 \\ 4 & 9 & 5 \\ 6 & 7 & 8 \end{matrix}], P = [\begin{matrix} 9 & 9 \\ 9 & 9 \end{matrix}], \frac{\partial L}{\partial P} = [\begin{matrix} 0.1 & 0.2 \\ 0.3 & 0.4 \end{matrix}]

\frac{\partial L}{\partial A} = [\begin{matrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{matrix}]

The center receives 0.1+0.2+0.3+0.4=1. Copying only the last contribution would lose three paths. For a window with a unique maximum, every non-winning cell receives zero from that window.

At a tie, the ordinary derivative is not unique. In the block [[4,4],[1,0]], two cells attain the maximum. Sending an incoming gradient of 1 to one selected maximum is a valid convention; splitting it as 0.5 and 0.5 is another valid subgradient. A specific framework operation chooses its own routing convention. A finite-difference check at this exact tie need not equal a selected convention.

ReLU uses its saved preactivation. For Z>0 its local derivative is 1; for Z<0 it is 0. At exactly zero, this miniature chooses 0. The sign of the incoming gradient does not decide whether the gate opens: the saved Z does.

In a concrete variant of the miniature, setting its convolution bias to −1.5 makes all four preactivations negative: [[−0.4,−1.1],[−1.3,−0.9]]. Then h is all zero, the convolution-weight and dense-weight gradients are zero, but the dense-bias gradient remains approximately [0.524979,−0.524979] for true class B. The final biases can still change because their gradient is p−y and does not multiply by h.

Checkpoint: what does an all-zero feature map imply?

For this one example, it blocks gradients into the preceding convolution and gives a zero outer-product gradient for dense W. It does not imply that every parameter gradient is zero. Another input may open different gates, and dense output biases still receive gradients. This is a local statement about the saved forward pass.

Inspect the ReLU mask in NumPy backward() and the matching architecture’s Python stride-two max-pooling layers. The overlapping pool and tie blocks above are separate teaching examples.

DERIVATION 6 / CONNECT THE EQUATIONS TO EXECUTED CODE

One optimizer step changes parameters; an explanation step only reveals a calculation.

The animation’s 28 stages explain one training iteration. Its fifteen parameter previews all use the same cached gradient snapshot and its commit stage performs one simultaneous SGD update. A later forward pass needs the new weights; a later backward pass needs the activations from that new forward pass.

Training action	Values read	State changed	Where to inspect
Forward	Inputs and current parameters	Cached intermediate activations, scores and loss	forward
Backward	Cached forward values and unchanged parameters	Gradient buffers; parameters remain fixed	backward
SGD commit	Saved parameters, gradients and learning rate	All fifteen parameter values	sgdStep
Adam batch step	Batch gradients, first/second moments, step count	Parameters and optimizer moments	optimizer.step()
Validation or test	Current model and separate examples	Metrics; no optimizer update	evaluate

Count the actual updates. The CNN visits 384 training images in batches of 32. It performs 12 optimizer steps per epoch and 144 steps across 12 epochs. Validation runs after each epoch and the held-out test runs afterward. Neither adds an optimizer step. The fixed synthetic split order is shuffled once when the dataset is built; the current Python and browser training loops do not reshuffle it between epochs.

Track Adam’s extra state. Its gradient moments belong to a particular optimizer run. Replacing model weights while retaining old moments would define a different training experiment. The miniature intentionally uses plain SGD without moments so the one-step arithmetic is transparent.

For 1,283 float32 parameters, weights alone need 5,132 bytes. One gradient array adds another 5,132; Adam’s two same-sized moment arrays add 10,264. Those four arrays total 20,528 bytes. This is a storage calculation, excluding activations, temporary tensors, counters, metadata and framework overhead; it is not a measured browser-memory footprint.

Python training line	Connection to the formulas	What to verify during review
`optimizer.zero_grad()`	Start the next batch’s gradient buffers clean.	PyTorch normally accumulates gradients across backward calls.
`criterion(model(images), labels)`	Forward scores and mean cross-entropy.	Input is raw logits; labels are integer class indices.
`loss.backward()`	Reverse differentiation through the cached graph.	Gradients have the shapes of their parameters.
`optimizer.step()`	Apply the optimizer rule once.	The rate, epsilon, moments and step count belong to this run.
`with torch.no_grad():`	Evaluate without building a backward graph.	The evaluation function does not call optimizer.step().

Compare frameworks by meaning and shape. Browser images use NHWC and Python images use NCHW. Their flatten operations therefore order the same spatial/channel features differently. Matching the architecture and seeded generated images does not imply identical initializer streams, dense columns or learned weights. Each framework’s dense weights are consistent with its own flatten order. The exact NumPy miniature, by contrast, follows the same single-channel flatten order and parameter constants as the browser engine.

Checkpoint: what should be reproducible before reviewing model accuracy?

Confirm the input normalization, split membership/order, tensor layouts, parameter count, label convention, loss reduction, optimizer settings and update count. Then compare measured results under named backend/device conditions. The supplied Python file runs a genuine training experiment on generated symbols; it does not establish hand-camera, football or arbitrary drawing accuracy.

Review the complete Python CNN, the browser training and disposal loop, and the numerical and generator parity checks.

Implementation review: coordinate systems, detector stages and timing contracts

These sections describe the code running in this workshop. The numerical examples are invented checks, not new measurements from a camera or a football clip.

SYSTEM REVIEW 01 / THE COORDINATE METRIC

A scale-normalized feature still depends on its coordinates.

For RPS, app.mjs first uses the recognizer’s world landmarks and falls back to image landmarks when that output is absent. openness() computes three-dimensional Euclidean distances directly from the supplied x, y and z values. It does not reconstruct camera intrinsics or correct image aspect ratio. The air-art pinch rule is separate: it uses normalized image x/y distances.

r = \frac{‖ p_{3} - p_{0} ‖}{\sum_{j = 0}^{2} ‖ p_{j + 1} - p_{j} ‖}

A common translation, rotation, or uniform positive scale cancels in this ratio. Scaling x and y by different factors generally does not. For the planar finger chain (0,0), (0,1), (1,1), (2,1), endpoint distance is √5 and path length is 3, giving 0.745356. Halving x alone gives (0,0), (0,1), (0.5,1), (1,1): endpoint √2 and path 2, giving 0.707107. Both results were checked with the actual openness() function.

Normalized image x/y values represent fractions of different image dimensions. A non-square image can therefore change the Euclidean metric used by the fallback. World coordinates reduce this particular issue, but remain model estimates and do not remove perspective, occlusion, or tracking errors. A zero-length chain returns zero in the implementation.

Review question: which coordinate output produced an exported measurement, and would the same scaler and centroids transfer if that output changed?

Python functions: openness() and pinch_ratio(). Browser integration: world-landmark fallback · JavaScript.

SYSTEM REVIEW 02 / WHICH DETECTIONS ARE STILL AVAILABLE?

The threshold slider filters an already processed detector result.

Decoded padded frameSSD box/class tensorsMaximum class score per candidateWrapper NMS and score filteringSaved detection recordsApp threshold and associationSmoothing / interpolation / artwork

The app calls detector.detect(cnnInput, 100, 0.2). In the bundled wrapper, the third argument is forwarded to nonMaxSuppression(boxes, scores, maxBoxes, minScore, minScore). Thus this particular wrapper uses 0.2 for both the IoU suppression threshold and the score floor; 100 limits the retained boxes. The arguments have different meanings even though they have the same numeric value here.

NMS keeps high-scoring candidates and removes overlapping alternatives whose IoU exceeds its threshold. The wrapper first chooses a maximum-scoring class for each candidate and runs suppression on those scores. The workshop’s saved raw array contains these returned box/class/score objects, after that processing. It is useful evidence, but it is not the full dense anchor tensor or all class logits.

The confidence slider (0.20–0.85) then accepts saved sports ball records with score at least the selected value. Changing it reruns filtering, association and smoothing over saved records; it does not run the CNN again. In an invented isolated-box example, a retained score of 0.34 is rejected at 0.35 and accepted at 0.30. A box suppressed upstream, below the initial score floor, or beyond the wrapper’s retained set cannot be recovered by this slider.

The original-view green boxes show saved sports-ball records near the playback time, including ones rejected by the app’s current threshold. The yellow selection and replacement use the accepted trajectory. A visible green box with no replacement can therefore be consistent with the current threshold.

Review question: did a failure arise in the detector’s candidate processing, the app threshold, identity association, or display interpolation? Freeze model/version and sampling when comparing settings.

Python functions: choose_ball() and trajectory(). Browser sources: COCO-SSD wrapper · JavaScript and analysis / threshold handlers · JavaScript.

SYSTEM REVIEW 03 / TWO DIFFERENT MEANINGS OF MOVEMENT

The fallback and the early classifier inspect different time spans.

The eight-feature classifier uses only [cutoff−450 ms, cutoff], with at least four samples, at least 100 ms of span, and a final sample no more than 150 ms old. Its rate measures the first-to-last change in that window. The finger-only fallback instead uses all recorded samples at or before cutoff and reports a finger when its observed maximum minus minimum is at least 0.12. It needs two observations and does not apply the classifier’s freshness/span rules.

For cutoff 1000 ms, suppose index straightness is 0.20 at 100 ms, then 0.40 at 600, 700, 800 and 900 ms; the other fingers stay at 0.30. The classifier’s recent window yields latest index 0.40 and rate zero. The fallback still reports Index moved, because its attempt-wide range is 0.20. This was checked with the actual feature and fallback functions.

The fallback means movement was observed earlier in this attempt. It supplies no RPS category or calibrated confidence, and cannot turn an invalid early prediction into a scored guess. Separately, the animation callback that freezes a round must execute before reveal; otherwise the app records no early prediction even if earlier measurements exist.

Review question: should your product report attempt-wide movement or only recent movement? Define that window before changing the threshold or interpreting the message.

Inspect Python features() and finger_movement() and the Python deadline rule.

APPLIED MATH 1 / LANDMARKS TO A TOOL

A dimensionless ratio becomes a brush decision.

Let p_i = (x_i, y_i) denote landmark i in normalized image coordinates. Point 4 is the thumb tip, 8 the index tip, and 5/17 the index/little-finger bases. The following coordinates are invented teaching values, not a camera recording.

d (a, b) = \sqrt{{(x_{a} - x_{b})}^{2} + {(y_{a} - y_{b})}^{2}}

r = \frac{d (4, 8)}{\max (d (5, 17), 0.001)}, paint when r < 0.35

Quantity	Substitute the numbers	Result
Tip gap	p₄=(0.67,0.36), p₈=(0.70,0.40)	√(0.03²+0.04²)=0.05
Palm width	p₅=(0.55,0.60), p₁₇=(0.75,0.60)	0.20
Pinch ratio	$\frac{0.05}{0.20}$	0.25: paint

If both distances double, the ratio remains 0.25. This explains the approximate scale normalization. The implementation measures 2D distances in normalized coordinates; it does not correct perspective or recover physical centimeters.

For the 800×600 drawing board, the mirrored cursor is x=(1−x₈)×800, y=y₈×600. Our index tip becomes (240,240). Multiplying by the displayed CSS width instead would mix coordinate systems.

A time inequality controls erasing

With no pinch, Open_Palm score ≥0.6 starts a dwell at time t₀. Erasing starts when t−t₀≥350 ms. If recognition starts at 1250 ms, then 1490 ms gives 240 ms and a lifted brush; 1600 ms gives 350 ms and active erasing. A pinch immediately restores the pen and clears the dwell. A low score or different gesture also clears it. These are programmed state transitions, not extra CNN training.

Python functions: pinch_ratio(), cursor_position() and camera_brush().

Hand control functions · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/hand_functions.py; download the complete file for imports, shared helpers and runnable examples.

Hand control functions language Download complete source

Loading Python functions…

Loading Python functions…

APPLIED MATH 2 / A CAUSAL FEATURE VECTOR

Finger geometry and elapsed time produce eight features.

For one finger, let q₀,q₁,q₂,q₃ be MCP, PIP, DIP and tip coordinates. RPS prefers world landmarks and falls back to image landmarks. Its distance function includes x, y and available z.

s = \frac{‖ q_{3} - q_{0} ‖}{\sum_{j = 0}^{2} ‖ q_{j + 1} - q_{j} ‖}

The triangle inequality makes the ratio at most one. A chain with total segment length 6 and endpoint distance 4.8 gives s=0.8. Units cancel when coordinates use a consistent unit. This is chain straightness, not an anatomical joint angle. A zero-length chain returns zero in the implementation.

Four chains begin at landmark indices 5,9,13,17. Let v(t) be their four straightness values. A toy valid window starts at 550 ms with [0.40,0.50,0.30,0.40] and ends at 850 ms with [0.70,0.80,0.35,0.45]. Intermediate detections at 650 and 750 ms satisfy the four-sample requirement.

Δt = \frac{850 - 550}{1000} = 0.30 s

rates = \frac{last - first}{Δt} = [1.00, 1.00, 0.1667, 0.1667] per second

x=[0.70,0.80,0.35,0.45, 1.00,1.00,0.1667,0.1667]

The cutoff determines which observations exist for prediction

At default pace, READY ends at 600 ms, MOVE lasts 700 ms, scheduled reveal is 1300 ms, and a 300 ms lead puts cutoff at 1000 ms. The eligible window is [550,1000] ms. Our last sample at 850 ms is exactly 150 ms old and remains valid. A final sample at 849 ms would be 151 ms old and fail. Require at least 100 ms between the first and last selected samples. Post-cutoff samples never enter these features.

A callback committing at 1008 ms achieves 1300−1008=292 ms of lead, despite requesting 300 ms. A callback first arriving at or after 1300 ms invalidates the early guess. HOLD continues until 2000 ms, and final-label confirmation follows.

Python functions: openness(), features() and finger_movement().

Early feature functions · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/hand_functions.py; download the complete file for imports, shared helpers and runnable examples.

Early feature functions language Download complete source

Loading Python functions…

Loading Python functions…

APPLIED MATH 3 / FITTING A SMALL MODEL

Training fixes the coordinate system before comparing classes.

For feature i, compute its mean and population standard deviation over the N training rounds. Keep those statistics fixed for test rounds. Let N_c count training rounds in class c.

μ_{i} = \frac{1}{N} \sum_{n = 1}^{N} x_{n i}, σ_{i}^{2} = \frac{1}{N} \sum_{n = 1}^{N} {(x_{n i} - μ_{i})}^{2}

z_{i} = \frac{x_{i} - μ_{i}}{\max (σ_{i}, 0.05)}

m_{c} = \frac{1}{N_{c}} \sum standardized training vectors in class c

D꜀=Σᵢ₌₁⁸(zᵢ−m꜀,ᵢ)²; choose the class with smallest D꜀

For one illustrative feature with training values [0.2,0.5,0.8], μ=0.5 and $σ = \sqrt{\frac{0.09 + 0 + 0.09}{3}} = 0.244949$ . A new value 0.7 becomes z≈0.816497. For a constant training feature [1,1,1], σ=0, so the floor 0.05 makes a new value 1.01 become z=0.2. The floor prevents division by zero and limits amplification of nearly constant features.

Here is a separate eight-dimensional distance illustration. The vectors are chosen for arithmetic, not learned participant measurements. Use query z=[.4,.6,.1,−.1,.6,.4,−.1,.1].

Class center m꜀	Squared distance
Rock: [−1,−1,−1,−1,−1,−1,−1,−1]	1.96+2.56+1.21+.81+2.56+1.96+.81+1.21=13.08
Paper: [.5,.5,1,1,.5,.5,1,1]	.01+.01+.81+1.21+.01+.01+1.21+.81=4.08
Scissors: [.5,.5,0,0,.5,.5,0,0]	Eight terms of .01=0.08

Choose scissors, then commit rock to beat the guessed move. A distance of 0.08 has no calibrated probability interpretation. Ties follow class order: rock, paper, scissors. This classifier always chooses a nearest center when valid features exist; identical early fists can therefore produce a guess without meaningful separating evidence.

Python functions: train(), predict() and counter().

Classifier functions · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/hand_functions.py; download the complete file for imports, shared helpers and runnable examples.

Classifier functions language Download complete source

Loading Python functions…

Loading Python functions…

APPLIED MATH 4 / DETECTOR COORDINATES

Padding requires an inverse coordinate transform.

Let a resized frame have width W and height H. Ball Remix preserves its aspect ratio, then centers it in a black square. Detector output boxes already use that padded canvas’s pixel coordinates.

S=ceil(1.3×max(W,H))

d_{x} = \frac{S - W}{2}; d_{y} = \frac{S - H}{2}

[x,y,w,h]frame=[xpad−dₓ, ypad−dᵧ, w,h]

For a 960×540 frame, S=1248, dₓ=144 and dᵧ=354. An invented padded detection [544,554,100,80] maps to [400,200,100,80]. Its center is (450,240). Width and height do not change when removing translation-only padding. Do not apply an extra model-input resize factor to boxes the wrapper already mapped to canvas coordinates.

center = (x + \frac{w}{2}, y + \frac{h}{2}), r = 0.55 \times \max (w, h) \times size

At default size 0.9, this box produces radius 0.55×100×0.9=49.5 px. This radius draws an overlay; it is not a segmentation boundary learned by the CNN.

The CNN-only association rule first chooses the largest accepted sports-ball box, then the nearest center. From (450,240), candidates (500,240) and (452,260) have distances 50 and √404≈20.10, so the second wins. A nearby wrong ball can win this rule too. The optional color-prototype step can supply a measured position when the CNN misses; it does not establish object identity. A sample with neither source resets association. There is no velocity gate or learned appearance-identity embedding.

Python functions: padding_geometry(), remove_padding() and choose_ball().

Ball coordinate functions · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/video_functions.py; download the complete file for imports, shared helpers and runnable examples.

Ball coordinate functions language Download complete source

Loading Python functions…

Loading Python functions…

APPLIED MATH 5 / BETWEEN VIDEO SAMPLES

Smoothing and interpolation answer different questions.

Let qₜ be a detected center and sₜ its smoothed center. The optional exponential moving average remembers the previous result. The first valid point initializes directly from its measurement.

s_{t} = α s_{t - 1} + (1 - α) q_{t}

With α=0.8, previous x=100 and new measured x=160, smoothed x is 112. If the next measurement is 220, smoothing gives 133.6. The model has new observations, but the display lags by 48 and 86.4 pixels. Larger α can reduce jitter while adding delay.

For a stationary step from 100 to 200, the remaining error follows eₙ=αⁿe₀. After five updates at α=.8, error is .8⁵×100=32.768 px. This closed form assumes an unchanged target and no missing samples.

Interpolation instead estimates a position between two existing valid samples during playback. With left point (100,200) at 1.0 s and right point (160,230) at 1.1 s, playback at 1.04 s has fraction $u = \frac{0.04}{0.10} = 0.4$ .

p(t)=(1−u)pL+upR=pL+u(pR−pL)
p(1.04)=(100,200)+.4×(60,30)=(124,212)

The app samples at 10 Hz, so ordinary neighboring samples are 0.1 s apart. It interpolates only valid neighboring points with gap ≤0.3 s. A row with no position breaks the path, even if valid points exist on both sides. The final point can hold for less than 0.101 s. Interpolated positions inherit the left measurement’s source and score: appearance similarity is not CNN confidence. Interpolation makes no additional CNN call and is not evidence of a newly observed ball.

Python functions: trajectory() and point_at().

Temporal tracking functions · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/video_functions.py; download the complete file for imports, shared helpers and runnable examples.

Temporal tracking functions language Download complete source

Loading Python functions…

Loading Python functions…

APPLIED MATH 6 / EVALUATION DENOMINATORS

A small reported error can hide missed balls.

For axis-aligned boxes, intersection width is max(0,min(rightA,rightB)−max(leftA,leftB)); compute height similarly. Multiply them for intersection area. Union adds the two box areas and subtracts their intersection once.

IoU = \frac{I}{A_{1} + A_{2} - I}

Boxes [0,0,100,100] and [50,0,100,100] overlap by 50×100=5000 pixels². Union is 10000+10000−5000=15000. Therefore $IoU = \frac{1}{3}$ . A class confidence score cannot substitute for this localization calculation. IoU appears in the lecture’s teaching tool; Ball Remix does not currently implement box-label IoU or detector AP.

The optional Ball Remix comparison accepts human center labels. Let D contain labeled, visible-ball samples where an accepted tracked point exists. Let gᵢ be the human center and pᵢ the tracked center.

E = \frac{1}{| D |} \sum_{i \in D}^{} \frac{‖ p_{i} - g_{i} ‖}{\sqrt{W^{2} + H^{2}}}

On an 800×600 image, the diagonal is 1000 px. A predicted center (403,304) versus human center (400,300) has distance 5 px and normalized error 0.005=0.5%.

Now label six sample times: four visible and two absent. Suppose three visible samples have errors 5,10,15 px, one visible ball is missed, and one absent sample has a detection. Report mean detected-visible error 1%, one visible miss, one absent-frame false detection. Accepted-point coverage is $\frac{4}{6} = 66.7 %$ , but those four include the false detection. The low center error does not erase the missed sample.

If no visible labeled sample has a detection, mean error is unavailable, not zero. A tracked wrong ball can count as an available point while giving a large center error; review identity mistakes separately. These numbers are arithmetic examples, not a measured accuracy claim.

Python functions: annotation_metrics() and intersection_over_union().

Evaluation functions · Python / JavaScript

Read the Python functions below. The displayed excerpt belongs to python/video_functions.py; download the complete file for imports, shared helpers and runnable examples.

Evaluation functions language Download complete source

Loading Python functions…

Loading Python functions…

Technical reference: training, experiments, system design and source map

A / The complete learning loop

Define the task. Choose inputs, labels, classes and a cost of error.
Collect or generate data. Record source, consent/permissions where relevant, labels and conditions.
Split by independent unit. Use separate images for the synthetic lab, complete rounds/players for RPS, and clips/sessions for video evaluation.
Fit. Training data determines weights, preprocessing statistics or centroids.
Validate. Use validation data to choose settings and inspect learning curves.
Freeze and test. Report errors on data not used to make those choices.
Inspect failures. Identify confusion, missing detections, identity jumps, latency and interaction mistakes.
Improve one thing. Compare under the same protocol and keep new final-test data if earlier test results guided revisions.

A low training loss with weak validation performance suggests a generalization problem. More epochs may worsen it. Poor training and validation performance can also reflect insufficient capacity, optimization trouble, bad labels or mismatched input preprocessing. Inspect actual examples before choosing a remedy.

Data augmentation changes training examples, such as small shifts or noise. Apply only transformations that preserve the intended label. Rotating a plus by 45 degrees creates an X in this task, so that transformation changes its class.

B / Architecture and optimization choices

Convolution learns local feature extractors. ReLU introduces nonlinearity. Pooling reduces spatial resolution. Flattening reshapes. A dense layer maps features to class logits. Softmax normalizes class scores. Cross-entropy penalizes low score for the true class. Backpropagation supplies gradients, and the optimizer updates parameters.

The toy network is intentionally small and task-specific. Real detectors need box regression, classification, candidate matching, suppression and much more diverse data. Training the symbol CNN does not retrain MediaPipe or COCO-SSD. Fine-tuning a detector requires compatible labeled boxes, a training/export workflow and a separate evaluation.

The lab code uses a fixed, pre-shuffled dataset order with shuffle:false. Seed 314 controls image generation and weight initialization. The network starts fresh when retrained. A run is reproducible enough for learning comparisons without promising identical GPU results on every device.

Softmax outputs sum to one. This does not establish calibration on human drawings, unseen cameras or novel classes. A classifier trained on three classes still assigns scores to an image of something else. Add an abstention policy only after defining and evaluating it.

C / Confusion, precision, recall and coverage

Our confusion matrices put true classes in rows and predicted classes in columns. Diagonal entries are correct classifications. Overall accuracy is diagonal sum divided by the number of scored examples. In the fractions below, TP means true positives, FP means false positives, and FN means false negatives.

Precision = \frac{TP}{TP + FP}

Recall = \frac{TP}{TP + FN}

F_{1} = \frac{2 \times precision \times recall}{precision + recall}

Define the positive class and matching rule first. For object detection, a common evaluation matches a predicted box to a human box using an IoU threshold and class agreement. One target cannot count as multiple true positives. Average precision summarizes a precision–recall curve under a defined matching protocol. This workshop does not compute detector AP.

Example: 10 true positive detections, 5 false positives and 2 misses give precision $\frac{10}{15}$ and recall $\frac{10}{12}$ . Report undefined ratios when a denominator is zero rather than inventing a perfect score. For early RPS, also report attempted rounds with no usable prediction.

D / Why the RPS model can fail honestly

It cannot read intention. If all three future moves show the same fist before cutoff, the observed inputs may carry little or no separating information. A confident nearest-centroid choice is still possible because the implementation always selects a nearest class when features exist. It has no calibrated confidence threshold.

Four finger chains begin at landmarks 5, 9, 13 and 17. For each, sum MCP-to-PIP, PIP-to-DIP and DIP-to-tip segment distances. Divide MCP-to-tip distance by that sum. The final eight-feature vector combines position-like straightness and time-normalized change.

Current-camera recognition separately maps Closed_Fist, Open_Palm and Victory to RPS at category score ≥0.5. The simulation path uses explicit finger thresholds: open ≥0.8, curled <0.65. Built-in traces demonstrate a workflow rather than a human-motion benchmark.

Source/timing changes reset incompatible data. Personal camera training discards artificial examples before collecting the real training set. Test rows do not update centroids. At least three training rounds per class permit fitting; that minimum is not enough to claim broad predictive performance.

E / A detector and a tracker solve different problems

Detection asks where an object is in one image. Association asks which detection belongs to an object observed earlier. A filter can smooth observations. A motion model can predict a future state. Optical flow estimates image motion. These methods are related but not interchangeable.

Ball Remix currently associates by nearest center after starting with the largest box. Two nearby balls, occlusion or a sudden camera move can break identity. A smoothing lag can also affect the next nearest-center choice because the implementation stores the smoothed previous position.

Possible extensions include a maximum-distance gate, size consistency, estimated velocity, an appearance descriptor or a multi-object assignment method. State the change, preserve comparison labels and inspect identity errors. A Kalman filter would add an explicit motion/uncertainty model, but this starter does not implement one.

F / Browser execution, resources and reproducibility

TensorFlow.js manages tensors and may execute on WebGL or CPU. The CNN lab yields between training batches, supports cancellation and disposes intermediate tensors. Ball Remix prefers WebGL and falls back to CPU if backend setup fails. The hand tasks use local MediaPipe WASM with a CPU delegate.

Canvas sampling, video decode and UI work also consume time. Repeated CPU-to-GPU uploads or tensor copies can dominate a tiny network. Measure the full path instead of treating model inference time as the whole experience.

Stopping a camera must stop every media track. Hiding a video is not stopping capture. Epoch tokens prevent late async work from attaching to a newer session. Cancellation must wait for in-flight work before reusing shared model/input resources.

For repeatable evidence, save model/version, asset hashes, source clip, sample times, threshold, smoothing, backend and human labels. Runtime logs and test results remain software-test evidence rather than proof of accuracy on every device.

G / Advanced experiments by role

Question	Controlled comparison	Evidence
Does pooling help?	Keep generator/splits fixed; change architecture	Parameters, training time, test errors and human-drawing failures
Does confidence filtering help?	Same clip and labels, several thresholds	Misses, false detections and center error
Does smoothing help?	Same raw detections, alpha 0 versus a chosen value	Jitter, lag and identity changes
Does the interface help?	Same tasks, palm dwell settings or preview visibility	Accidental actions, task completion and participant observations
Does the predictor transfer?	Train on one player; test another without fitting	Per-player errors, rejected rounds and baselines

Each role can own real work without writing the whole application. Labeling policy, failure review, interface design, artwork, experiment records and source attribution are substantial contributions.

H / Source map for this exact workshop

Source	What to inspect
Python hand functions	Pinch, cursor, eraser, finger features, classifier and deadline rules
Python video functions	Padding, ball selection, interpolation and evaluation
Python runtime examples	Drawing alpha and resource ownership patterns
vision-lab.mjs	Convolution, ReLU, pooling, pinch ratio and palm dwell
cnn-lab-core.mjs	Seeded images, split generation, CNN architecture and scoring
cnn-lab.mjs	Training callbacks, inference, drawing, feature maps and disposal
explore.mjs / party.mjs	Camera integration, rendering, tool state and game phases
core.mjs / app.mjs	RPS eight-feature model and early cutoff enforcement
camera.mjs / model-loader.mjs	Capture errors, late-stream cleanup and model import recovery
tracking.mjs	Ball association, smoothing, interpolation and error metrics
lecture-math.mjs	Tensor calculator, EMA and IoU teaching calculations

I / Primary references and limits

TensorFlow.js models and layers and training guide for the runnable API.
Gesture Recognizer task guide and MediaPipe Hands paper for the pretrained hand pipeline.
COCO-SSD source, SSD paper and MobileNetV1 paper for the ball detector.
MobileNetV2 paper for the alternate model’s architecture.

The small CNN’s generated-data results, a successful camera interaction, software tests and recorded ball-detection coverage support different claims. Keep each attached to the conditions and measurements that produced it. The code is inspectable and the examples are runnable, while real-world reliability still needs testing across users, clips and devices.

Make the next project yours.Open the take-home project →Compare methods after the demo