AI · Computer vision experiments

Vision Lab

Integrated public vision models with evaluation and visualization across 18 technology pages. The lab presents recorded results and errors in object relations, motion comparison, simulated control, tracking, pose estimation, motion capture, and anomaly detection, while distinguishing executed stages from stages awaiting model release, access, or required files.

ROLEInference integration · evaluation design · failure analysis · control implementation · results viewer
PERIOD2026.09–11
BUILT WITHPython · PyTorch · MuJoCo · WebGL · Vercel
Vision LabTraceAnything 3D trajectories projected onto a DAVIS train frame
A frame from the 80-frame train run. Colored trajectories are model estimates; the website explores recorded outputs.
Technology pages
18

Recorded results · stage progress · model preparation status

Relation experiment frames
481

One generated clip · baseline and revised runs

Physics simulation evaluations
400

Internal repeats of one reference motion and four-joint model

01 / EXPERIMENT MAP

Execution scope by technology

Thirteen technology pages extend the five existing experiments. Eighteen is the page count, not a claim that 18 complete model pipelines ran successfully. Result pages compare inputs with raw predictions; preparation pages document completed stages and required model release, access, or files. Existing cases retain the detailed relation and motion-control designs.

ExperimentWhat it investigatesRun scope
RelateAnythingRelation scores and errors before and after object association481 generated-video frames
MotionCheck V1Motion differences using 2D joints and temporal alignmentReference and performed clips, including failures
MotionCheck V2Motion-stage decisions and torque control of simulated arms400 internal simulation evaluations
TraceAnythingTrajectories over footage and in a 3D point cloud80 train and 90 dance frames, plus input variants
DeepLabCut2D human and mouse keypoints and human 3D pose estimates90 dance frames with 17 joints; 120 mouse frames with 27 points
TAPNext++2D trajectories of surface points in videoInference on 90 dance and 80 train frames
Sapiens2308 keypoints and body/clothing segmentation90 dance frames, pose and segmentation run separately
HTD-RefinePVA-Net joint position, velocity, and acceleration estimatesPVA-Net on 90 frames · final mesh refinement pending
ReViVBody, hands, camera, gaze, and relative-depth estimates from egocentric videoOfficial 2-second sample · 256 model, 16 input frames → 60 motion frames
4DAnyoneNovel-view generation from a human videoPublic weights prepared · SMPLX_NEUTRAL.npz required
SAM 3.1Mask and ID tracking of text-prompted targetsCode and environment prepared · model access approval pending
Fast SAM 3D BodyAccelerated per-image human mesh estimationModel API imports verified · checkpoint access approval pending
CARI4D3D human-object motion and interaction reconstructionCode and environment prepared · SMPL-H model files required
Dense4DDense 4D surface correspondence of body, clothing, and hairOfficial demo links · code and checkpoint release pending · no local inference
S4VYObject segmentation and identity association across space and timeExisting segmentation and point-tracking comparison on 90 dance frames, plus point-prompt preparation · S4VY code release approval pending
PatchCoreVisual deviation from normal oranges, anomaly heatmaps, and missed cases248 evaluated photos · detection and tracking integrated on 247 real conveyor frames
SOMASubject-specific skin and muscle deformation from skeletal poseOfficial resources reviewed · SKIM v2 data and checkpoint access not approved · no local inference
MoCapAnything V2Joint position and rotation estimation with a reference skeleton, plus BVH exportInference on 34 rendered jaguar frames and a 103-frame polar bear wild sample

02 / MY CONTRIBUTION

Implementation and validation scope

I implemented object association and temporal processing, joint comparison and stage decisions, and MuJoCo torque control. For TraceAnything inference, I adapted execution scheduling and attention compatibility for Windows and a 24 GB GPU, then checked exported coordinates against raw predictions.

Evaluation separates missing observations from relation errors, pose differences from timing, and control error from task completion. For DeepLabCut, I compared default and bounding-box selection settings and distinguished prediction confidence from accuracy. For additional models, I recorded executed stages, exported data, and remaining requirements, connecting results and preparation status through 18 technology pages on one Vercel site.

03 / 2D POINT TRACKING

TAPNext++

Using the official public checkpoint, I ran 2D point tracking on 90 dance frames and 80 train frames. Trajectories over the source footage and per-frame predictions show where points move and where they drift from the surface.

These outputs are point trajectories in image pixel coordinates. They do not estimate named body joints or world-space 3D motion; occlusion, fast rotation, and accuracy against ground truth require separate evaluation.

04 / POSE AND SEGMENTATION

Sapiens2

I ran Sapiens2-0.4B pose and segmentation separately on the same 90 dance frames. The viewer compares 308 keypoints and 29 body/clothing segmentation classes, including background, with the source footage. Model weights stay local; videos, raw predictions, and execution settings are exported.

Pose selects the largest detected person box in each frame; segmentation labels each frame independently. Neither guarantees persistent identity or temporal mask consistency. Joint scores are raw heatmap values that can exceed 1, not probabilities or accuracy. I verified alignment of all 90 exported frames and their raw data.

05 / PARTIAL PIPELINE

HTD-Refine

I executed the PVA-Net stage of HTD-Refine to estimate joint positions, velocities, and accelerations across 90 frames of the official exercise sample. The page exposes this stage’s actual outputs and inputs and identifies how far the full pipeline has run.

Final human mesh refinement has not run. It requires the separate SMPLX_NEUTRAL.npz model file; PVA-Net outputs are not presented as completed 3D motion refinement or mesh reconstruction.

06 / EGOCENTRIC MOTION

ReViV

I ran the 256×256 reviv_500b model on an official two-second RGB sample to estimate body motion, both hands, camera motion, gaze, and relative depth. From the original 32 frames at 16 fps, I selected 16 frames at 8 fps as input. Body, hand, camera, and gaze outputs contain 60 frames at 30 fps; relative depth contains 16 frames at 8 fps. The viewer repeats source and depth frames to align their timelines.

The body and camera 3D view retains the predicted coordinate system. Ground-truth camera intrinsics K from the official sample are used only to project the predicted hands onto the image, not as model inputs. I preserved the substantial hand-position mismatch, especially in the first frame, without manually correcting coordinates.

This experiment verifies execution and output formats on one official sample; joint and gaze accuracy were not evaluated against ground truth. The grayscale relative-depth output has not been validated as distance in meters. The 512×512 metric_depth model, which requires Cosmos 1.0 access agreement, was not run.

07 / KEYPOINT ESTIMATION

DeepLabCut

SuperAnimalHumanBody with RTMPose-x estimated 17 joints across 90 dance frames; SuperAnimalTopViewMouse with HRNet-w32 estimated 27 keypoints across 120 mouse frames. The viewer compares source and prediction videos at matching frames and provides execution settings, confidence values, and prediction data.

The default run switched people in the dance footage. I added a comparison that selects the largest detected bounding box in each frame. This rule does not track identity and does not guarantee that the same person stays selected or that joint locations are correct.

FMPose3D 0.0.10, through the official integration, estimated 17 H36M 3D joints for the same 90 dance frames. It uses a separate YOLOv3 and HRNet-w48 pipeline; the viewer compares default and largest-box predictions with rotating 3D views. Coordinates are pelvis-relative, and joint-placement errors remain near frame 60 even with largest-box selection.

08 / 3D TRACKING

TraceAnything

TraceAnything processed 80 DAVIS train frames and 90 dance frames. The dance viewer compares all 90 frames, 30 frames sampled every third frame, and a fixed person crop of those 30 frames, with projected overlays and a rotatable 3D view.

DAVIS masks select displayed tracks, define the fixed crop, and weight camera fitting. They do not correct predicted 3D coordinates. Points still drift from body surfaces during fast rotation and occlusion; tracking quality remains under review.

09 / ANOMALY DETECTION

PatchCore

Using an independent PyTorch implementation of PatchCore, I built a feature memory bank from 110 normal orange photos and fixed the threshold using 30 separate normal photos. The viewer exposes source images, anomaly heatmaps, scores, and missed cases for 248 evaluation images. Only 12 of 198 anomalous images were detected, giving 6.06% recall, with no false positives among 50 normal images. This exploratory run preserves its weak detection performance.

On 247 frames of real Pexels conveyor footage, I connected RF-DETR Small detection, ByteTrack tracking, PatchCore anomaly scores, and decisions across successive frames. The viewer compares recorded source footage, overlays, and tracking logs. Because the clip differs from the photo-training conditions and lacks quality ground truth, it is documented as a pipeline integration check.

10 / MOTION CAPTURE

MoCapAnything V2

Using the authors’ released clean reimplementation and checkpoint, I inferred 39 joints from a rendered Jaguar-Run sample containing 34 frames at 30 fps, and 34 joints from the authors’ background-removed polar bear wild sample containing 103 frames at 15 fps. A monocular clip and a single reference frame with skeletal information from the reference asset provide the inputs; the resulting joint rotations are exported as BVH.

The viewer synchronizes input footage with an interactive 3D skeleton and switches between directly predicted joint positions and FK positions reconstructed by reloading the exported BVH. Raw coordinates, videos, BVH, and execution conditions are available. This validates execution on two public examples rather than reproducing the paper benchmark. The polar bear clip has no ground-truth joints, so accuracy is not reported.

ENGINEERING DECISIONS

Engineering decisions

01

Separate inference from result exploration

GPU inference runs locally, with per-frame outputs and execution conditions saved. The browser compares inputs and outputs at the same timestamp without rerunning the model.

02

Distinguish predictions, processing, and display

Mask selection and clearer visualization are not presented as accuracy improvements. Raw outputs and processing conditions remain available for inspection.

SCOPE & LIMITS

Limitations and scope

  • MoCapAnything V2 was run on a public rendered jaguar example and the authors’ background-removed polar bear wild sample on November 3, 2026, using the repository’s clean reimplementation. This does not claim equivalence to the paper’s evaluation results. The polar bear’s original capture conditions and camera metadata were not verified. Coordinates are normalized and root-relative, without validated world-space movement or metric scale. A reference skeleton is required, and generalization to new people, animals, or rigs has not been evaluated.
  • On October 29, 2026, I reviewed SOMA’s public MIT code and SKIM v2 resources. Inference needs the matching subject’s skeletal pose, anatomical meshes, rig, and learned weights; it does not directly process arbitrary monocular footage. Data and checkpoints are distributed through a portal requiring institutional account approval under noncommercial academic terms. Without an approved account, local inference has not run; the code’s MIT license is not treated as permission to redistribute data or weights.
  • The PatchCore photo experiment missed 186 of 198 anomalous images. The conveyor clip has no quality ground truth and differs from the training-photo domain, so its anomaly flags do not establish defects or industrial inspection reliability. Track IDs are not a count of physical fruit, and heatmap localization accuracy has not been evaluated.
  • On October 29, 2026, the official S4VY repository stated that code release approval was pending. I implemented synchronized comparison of the same 90 dance frames: source footage, existing Sapiens2 semantic segmentation, and TAPNext++ 2D point tracks, with export of selected point coordinates for preparation. These are not S4VY inference or person-identity tracking results. The official Playground explores the author’s reconstructed meshes and does not accept user-video uploads.
  • On October 20, 2026, the official Dense4D page still stated Code Coming soon. This entry covers paper review and links to the author’s precomputed demos; it is not a successful local installation or inference run.
  • The contribution is integration, evaluation, and visualization using credited public models. I did not develop or retrain the original models; Vercel serves recorded results rather than GPU inference on new uploads.
  • The 18 technology pages include recorded results, partially executed pipelines, and items awaiting model release, access, or files. SAM 3.1 and Fast SAM 3D Body require official checkpoint access approval; later stages of 4DAnyone and HTD-Refine require SMPLX_NEUTRAL.npz; CARI4D requires SMPL-H model files. Code installation or successful imports do not count as completed inference.
  • Unresolved failures include the on relation, abstention and false rejection on real motion footage, 3D tracking drift, and DeepLabCut target switches and keypoint errors. These are development experiments on limited inputs, without demonstrated generalization to real robots or new people and scenes.
  • Confidence in the DeepLabCut viewer comes from 2D models; it is neither ground-truth accuracy nor FMPose3D 3D accuracy. FMPose3D outputs pelvis-relative coordinates. World-space translation, metric scale, ground-truth 3D accuracy, and long-term identity tracking have not been validated.
  • TraceAnything is a separate model from TrackEverything, whose code release was not found in the September 29, 2026 review. These runs do not validate 999-plus-frame tracking or 3D accuracy against ground truth. TAPNext++ performs 2D point tracking; Sapiens2 estimates pose and segmentation per frame.
Let’s build somethinghyunaeee@gmail.com ↗

Portfolio PDF

A casebook with roles, engineering decisions, evidence, and source links.

Choose “Save as PDF” in the print dialog.