VIDEO TRACKING / OPTIONAL EXTENSION
A tiny sports movie with real computer vision.
Explore how a detector, an appearance tracker and rendering code cooperate to turn a ball into a moving effect. Start with the supplied clear single-ball clip; finish with your own design and measured evidence.
1 · See the whole pipeline
Video frame → CNN features → SSD boxes/classes → sports-ball selection → temporal association → replacement artwork.
MobileNetV1 is the default convolutional feature extractor; MobileNetV2 is available for a model comparison. SSD predicts object boxes and class scores from image features. The bundled COCO-SSD model has pretrained weights; Analyze performs inference, not training. Our JavaScript then chooses a ball, links detections through time, and renders the effect. Those tracking and graphics steps are not CNN layers.
Before inference, frames are centered in a padded square and boxes are mapped back to video coordinates. This avoids squashing a round ball when the CNN resizes its input. The demo samples ten frames per second, with an additional final-frame sample; the clear starter clip is about seven seconds. During playback, interpolation fills positions between consecutive valid samples. A separately labeled appearance tracker can measure matching colored regions after a CNN anchor. Missing positions break the path; there is no hidden hand-entered trajectory. “Sports ball” is a broad object class, not proof that a specific ball is a football.
2 · Make your first remix
- Analyze the supplied clip. Inspect the boxes in the original view.
- Play the capture-ball effect. Pause and scrub to a miss or an identity switch.
- Compare confidence 0.35 and 0.65. What disappears? Try smoothing and observe the delay during a kick.
- Choose another skin or upload your own PNG. Draw your own sprite if you want a distinct visual identity.
Try a second local video. Upload uses a browser object URL; your video is not sent anywhere. Analysis covers the full duration, including a final decoded frame. A distant match broadcast is a harder case than a close-up: a ball may be too small or blurred for this detector.
3 · Choose an improvement you own
| Track | Your contribution | Evidence |
|---|---|---|
| Visual creator | Make a custom capture effect, glow trail or animated sprite. Change app.mjs, not only the dropdown. | Before/after frames and a short explanation of scale, rotation and compositing. |
| Tracking engineer | Improve chooseBall in tracking.mjs with a distance gate or size consistency. Handle two balls crossing. | Baseline vs changed tracker on the same manually labeled frames. |
| Experiment designer | Build a small two-clip benchmark; compare confidence, smoothing and motion conditions. | Misses, false detections, center error and three annotated failure cases. |
A portfolio version should include one actual code/visual contribution and an honest evaluation. Merely running the provided starter does not make the pretrained model your own training work.
4 · Measure a baseline, then change one thing
- Pause at a sampled time. Turn on Click to label ball center, then click the intended ball in the original view. Use Ball absent when it is out of frame.
- Label at least 20 distinct sampled times per clip, including visible balls and times when the ball is absent. For harder clips, include fast motion, two balls and occlusion. The orange marker is your human label.
- Export results JSON for the baseline. Change one tracker setting or algorithm; export again using the same labels.
- Use one clip to tune your design. Freeze the choice, then evaluate on a second clip you did not tune on.
Report detected visible-ball samples, missed visible balls, detections on absent-ball frames, and mean center error divided by the image diagonal. A wrong ball can have a high CNN score. “All samples tracked” is detection coverage, not 100% accuracy. Discuss identity switches separately.
Click labels are approximate. Document how you labeled partial occlusions, and do not claim broad accuracy from two short clips.
5 · Build a portfolio deliverable
- A runnable local project with Windows/Mac instructions and a clear project description.
- A before/after image or short screen-recorded demo, plus your source and asset credits.
- A report: problem → model → tracking method → your change → evaluation → failures → next step.
- A results table on both clips, exported labels/results, and three failure examples.
Suggested scope: 2–4 hours for a customized starter and small evaluation; a more substantial tracker or model experiment takes longer. Describe your work as “built a browser video-remix pipeline using a pretrained CNN detector.” Say you trained a model only if you actually did.
The sample clip and adapted footage are the source video license; keep its attribution and share-alike notice. The red/white capture-ball graphic is drawn in canvas; no official Pokémon artwork is bundled. You can replace it with your own artwork.
6 · Stretch: train for a harder ball
If your new clip defeats the starter, collect ball bounding boxes from additional clips, split by complete video, and fine-tune an object detector using an official training workflow. Keep the held-out video untouched. Compare the pretrained baseline with the fine-tuned detector at the same evaluation times.
This browser starter does not train CNN weights. A custom detector may have different outputs and need a new inference adapter; changing a model URL alone does not guarantee compatibility. Use random seed 314 for any training/data sampling you add.