AI · Relation prediction experiments
Ran an open-vocabulary relation model on an RTX 4090. On a generated clip, I diagnosed missing relation scores caused by changing detector classes, then compared persistent object association and temporal postprocessing separately.
287/481 → 481/481 · the same generated clip
End to end · before temporal postprocessing · 74 eligible draft labels
Threshold 0.5 · recall remains zero after association
01 / DIAGNOSIS
I generated a roughly 20-second clip of picking up a bottle and placing it on a table. YOLO11s and RelateAnything ran on every frame to inspect person–holding–bottle and bottle–on–table relations.
The baseline selected detections classified as bottle in each frame. During the lift, the same object was labeled vase, cell phone, or cup, so valid localizations were dropped from the bottle relation input.
02 / IMPLEMENTATION
A confident bottle detection initializes the object identity. Later observations are associated across detector classes using position, motion, and size. Only boxes actually observed in the current frame enter the relation model; a missing detection remains unobserved.
RelateAnything receives a single image, regions, and predicate strings per frame. Object association, EMA, and hysteresis are separate temporal components. The viewer distinguishes model scores from temporal system state.
Original timestamp and frame index
Currently observed regions
Ordered object pairs × predicate scores
Unsmoothed output / temporal state
03 / EVIDENCE
Draft labels were created by reviewing the clip before inference at 0.25-second intervals. Ambiguous contact transitions are marked uncertain and excluded. Evaluation matches exact frame indices; timings requested in the generation prompt are not treated as ground truth.
A missing detection is different from observing a false relation. Conditional and end-to-end metrics are reported separately; missed positive relations count as false negatives end to end. The 0.5 threshold has not been optimized on a separate validation set.
The baseline supplied bottle regions in 287 of 481 frames and reached holding F1 0.120. Class-agnostic association recovered observed boxes in all 481 frames and reached F1 0.825 before temporal postprocessing: precision 70.2%, recall 100%, with 14 false positives remaining.
Temporal postprocessing reduced holding false positives from 14 to 13, yielding F1 0.835. The on relation still missed all 56 positives after association. The result separates recovered object availability from accurate relation decisions and keeps the unresolved failure visible.
On an RTX 4090 with FP32 and batch size 1, the pipeline including association and postprocessing averaged 38.68 ms/frame, with p95 46.14 ms. The inverse mean is about 25.85 FPS; it is not the web playback rate or deployed-server performance.
| Measure | Automatic baseline | With object association |
|---|---|---|
| Bottle regions · 481 frames | 287 / 481 | 481 / 481 |
| holding · TP / FP / FN | 3 / 14 / 30 | 33 / 14 / 0 |
| holding F1 · before postprocessing | 0.120 | 0.825 |
| on recall · 56 positives | 0% | 0% |
04 / RECORDED DEMO
The public demo aligns source footage with saved run outputs. It distinguishes the detector class observed for the object, availability of a relation score, and changes introduced by temporal postprocessing.
Vercel hosts the results viewer. The public page does not run GPU inference or calculate relations for newly uploaded videos. Displayed inference timings were measured locally on an RTX 4090.
ENGINEERING DECISIONS
The failure path discarded frames where the bottle had already been localized. After a confident initial observation, association preserves identity while recording detector label changes. The tracking approach does not use color or manual regions.
Per-frame scores with the released calibration are saved separately from EMA and hysteresis outputs. Temporal system state is not presented as a probability predicted by the model.
Records retain model and detector versions, vocabulary, input-video hash, score formula, and timing scope. The results view can be rebuilt without rerunning the model.
SCOPE & LIMITS
South Korea · Korean / English
Hyunae Park