STRIVE · 08/28/2026
Dyn generalization · val_24 · super_clevr_1099
Everything fed into evaluation and the full per-sample output it produced.
Dataset row: exact video + user prompt + assistant target
{
"messages": [
{
"role": "user",
"content": "<video>\nAnalyze the provided video and return ONLY a valid sentinel sequence in this format:\n[SCENE-START] [OBJS-START]\n[OBJ] id \"type\" birth death [VIS] t vis ... [IN-FRAME] t is_in_frame ... [PX] t px py ...\n[OBJ] ...\n[OBJS-END]\n[INTERACTIONS-START]\n[INTERACT] id \"type\" begins_at ends_at [IN] in0 in1 ... [OUT] out0 out1 ...\n[INTERACT] ...\n[INTERACTIONS-END] [SCENE-END]\n\nConstraints:\n- Return only the sentinel text (no JSON, XML, markdown, comments, or extra prose).\n- Keep tags exactly as written:\n `[SCENE-START] [OBJS-START] [OBJ] [VIS] [IN-FRAME] [PX] [OBJS-END] [INTERACTIONS-START] [INTERACT] [IN] [OUT] [INTERACTIONS-END] [SCENE-END]`\n- Use quoted strings for `type` (double quotes).\n- `VIS` uses repeated groups of 2 values in order:\n `t vis`.\n `vis` is a boolean value.\n- `IN-FRAME` uses repeated groups of 2 values in order:\n `t is_in_frame`.\n `is_in_frame` is a boolean value.\n- `PX` uses repeated groups of 3 values in order: `t px py`.\n- `px` and `py` are normalized pixel coordinates in the range `[0, 1]`,\n scaled by `1000` and then converted to integers by dropping the decimal part.\n- For occluded or out-of-frame frames under this setting, `PX` may use `t null null`.\n- `IN` and `OUT` contain whitespace-separated integer ids.\n- Use lowercase booleans (`true`/`false`) for `vis` and `is_in_frame`."
},
{
"role": "assistant",
"content": "[SCENE-START] [OBJS-START] [OBJ] 0 \"blue airliner\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 10 214 1 15 213 2 18 214 5 18 217 6 18 220 7 23 230 8 23 234 9 29 240 10 23 245 11 31 253 12 47 259 13 62 267 14 77 273 15 92 280 16 109 289 17 123 296 18 168 303 19 23 285 20 34 292 21 51 303 22 67 313 23 84 325 [OBJ] 1 \"green dirtbike\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 240 665 6 242 665 10 244 664 13 247 664 15 248 663 17 251 663 19 254 661 20 215 675 21 256 660 23 219 674 [OBJ] 2 \"yellow utility\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 838 693 1 837 669 2 808 686 3 762 646 4 805 659 5 742 641 6 733 640 7 721 636 8 744 671 9 701 632 10 691 630 11 680 628 12 670 625 13 661 624 14 651 622 15 641 619 16 630 617 17 660 649 18 612 613 19 602 610 20 593 608 21 584 607 22 573 604 23 564 602 [OBJ] 3 \"green fighter\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 988 341 2 987 338 3 985 338 4 980 338 5 973 338 6 967 338 7 961 336 8 955 336 9 948 336 10 941 336 11 935 336 12 928 336 13 986 345 14 987 347 15 980 347 16 973 347 17 966 347 18 960 347 19 953 347 20 946 347 21 938 346 22 931 346 23 985 341 [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 23 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 2 65535 [OUT] 2 65535 [INTERACTIONS-END] [SCENE-END]"
}
],
"videos": [
"/mmfs1/gscratch/krishna/suryadv/strive_dynsuperclevr/data/lf/videos/val_24/super_clevr_1099/frames.max_frames_24.mp4"
]
}Inference record: exact prompt + prediction + label + parse debug
{
"prompt": "user\n\nAnalyze the provided video and return ONLY a valid sentinel sequence in this format:\n[SCENE-START] [OBJS-START]\n[OBJ] id \"type\" birth death [VIS] t vis ... [IN-FRAME] t is_in_frame ... [PX] t px py ...\n[OBJ] ...\n[OBJS-END]\n[INTERACTIONS-START]\n[INTERACT] id \"type\" begins_at ends_at [IN] in0 in1 ... [OUT] out0 out1 ...\n[INTERACT] ...\n[INTERACTIONS-END] [SCENE-END]\n\nConstraints:\n- Return only the sentinel text (no JSON, XML, markdown, comments, or extra prose).\n- Keep tags exactly as written:\n `[SCENE-START] [OBJS-START] [OBJ] [VIS] [IN-FRAME] [PX] [OBJS-END] [INTERACTIONS-START] [INTERACT] [IN] [OUT] [INTERACTIONS-END] [SCENE-END]`\n- Use quoted strings for `type` (double quotes).\n- `VIS` uses repeated groups of 2 values in order:\n `t vis`.\n `vis` is a boolean value.\n- `IN-FRAME` uses repeated groups of 2 values in order:\n `t is_in_frame`.\n `is_in_frame` is a boolean value.\n- `PX` uses repeated groups of 3 values in order: `t px py`.\n- `px` and `py` are normalized pixel coordinates in the range `[0, 1]`,\n scaled by `1000` and then converted to integers by dropping the decimal part.\n- For occluded or out-of-frame frames under this setting, `PX` may use `t null null`.\n- `IN` and `OUT` contain whitespace-separated integer ids.\n- Use lowercase booleans (`true`/`false`) for `vis` and `is_in_frame`.\nassistant\n",
"predict": "[SCENE-START] [OBJS-START] [OBJ] 0 \"light blue plastic conditioner tube\" 0 12 [VIS] 0 true 10 false [IN-FRAME] 0 true 10 false [PX] 0 1 208 1 21 220 2 25 228 3 29 240 4 13 248 5 1 255 6 1 267 7 5 283 8 5 294 9 1 306 10 null null [OBJ] 1 \"grey and green running shoe\" 0 12 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 1 451 1 null null [OBJ] 2 \"green and white plastic portable speaker\" 0 12 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 1 498 1 null null [OBJ] 3 \"grey and green running shoe\" 0 12 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 248 705 1 255 701 3 244 705 5 251 701 6 240 705 8 248 701 10 236 701 12 244 697 13 232 701 14 240 697 16 251 689 17 240 693 19 248 689 20 236 693 21 244 689 23 255 681 [OBJ] 4 \"yellow wooden toy construction vehicle set\" 0 12 [VIS] 0 true 3 false [IN-FRAME] 0 true [PX] 0 283 419 3 null null [OBJ] 5 \"green and orange plastic magnifying glass\" 0 12 [VIS] 0 true 1 false [IN-FRAME] 0 true [PX] 0 322 435 1 null null [OBJ] 6 \"yellow and orange plastic toy airplane with pilot\" 0 12 [VIS] 0 true 4 false [IN-FRAME] 0 true [PX] 0 748 439 4 null null [OBJ] 7 \"green and yellow running shoe\" 0 12 [VIS] 0 true 1 false [IN-FRAME] 0 true [PX] 0 755 435 1 null null [OBJ] 8 \"green Lego Android robot\" 0 12 [VIS] 0 true 3 false [IN-FRAME] 0 true [PX] 0 767 435 3 null null [OBJ] 9 \"yellow and blue plastic Spongebob toy set\" 0 12 [VIS] 0 true 3 false [IN-FRAME] 0 true [PX] 0 837 431 3 null null [OBJ] 10 \"dark grey and orange athletic shoe\" 0 12 [VIS] 0 true 3 false [IN-FRAME] 0 true [PX] 0 853 435 3 null null [OBJ] 11 \"yellow plastic toy school bus\" 0 12 [VIS] 0 true 4 false [IN-FRAME] 0 true [PX] 0 865 431 4 null null [OBJ] 12 \"green metallic lime squeezer\" 0 12 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 978 345 1 958 345 2 974 345 3 951 345 4 966 345 5 947 345 6 962 345 7 943 345 8 958 345 9 939 345 10 958 345 11 939 345 12 958 345 [OBJ] 13 \"brown and white plastic bald eagle figurine\" 0 12 [VIS] 0 true 6 false [IN-FRAME] 0 true [PX] 0 986 435 6 null null [OBJ] 14 \"red and white cardboard ink cartridge box\" 0 12 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 998 447 1 null null [OBJ] 15 \"black and grey athletic shoe with pink accents\" 1 12 [VIS] 0 false 1 true 9 false [IN-FRAME] 0 false 1 true 9 false [PX] 0 null null 1 998 318 5 986 318 6 998 322 9 null null [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 11 [IN] 0 15 [OUT] 0 15 [INTERACT] 1 \"contact\" 0 11 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 2 \"contact\" 0 12 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 10 \"contact\" 0 12 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 11 \"contact\" 0 12 [IN] 3 65535 [OUT] 3 65535 [INTERACT] 12 \"contact\" 0 12 [IN] 4 65535 [OUT] 4 65535 [INTERACT] 13 \"contact\" 0 12 [IN] 5 65535 [OUT] 5 65535 [INTERACT] 14 \"contact\" 0 12 [IN] 6 65535 [OUT] 6 65535 [INTERACT] 15 \"contact\" 0 12 [IN] 7 65535 [OUT] 7 65535 [INTERACT] 16 \"contact\" 0 12 [IN] 8 65535 [OUT] 8 65535 [INTERACT] 17 \"contact\" 0 12 [IN] 9 65535 [OUT] 9 65535 [INTERACT] 3 \"contact\" 0 12 [IN] 10 65535 [OUT] 10 65535 [INTERACT] 4 \"contact\" 0 12 [IN] 11 65535 [OUT] 11 65535 [INTERACT] 5 \"contact\" 0 12 [IN] 12 65535 [OUT] 12 65535 [INTERACT] 6 \"contact\" 0 12 [IN] 13 65535 [OUT] 13 65535 [INTERACT] 7 \"contact\" 0 12 [IN] 14 65535 [OUT] 14 65535 [INTERACT] 8 \"contact\" 0 12 [IN] 15 65535 [OUT] 15 65535 [INTERACT] 9 \"collision\" 10 11 [IN] 2 14 [OUT] 2 14 [INTERACTIONS-END] [SCENE-END]",
"label": "[SCENE-START] [OBJS-START] [OBJ] 0 \"blue airliner\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 10 214 1 15 213 2 18 214 5 18 217 6 18 220 7 23 230 8 23 234 9 29 240 10 23 245 11 31 253 12 47 259 13 62 267 14 77 273 15 92 280 16 109 289 17 123 296 18 168 303 19 23 285 20 34 292 21 51 303 22 67 313 23 84 325 [OBJ] 1 \"green dirtbike\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 240 665 6 242 665 10 244 664 13 247 664 15 248 663 17 251 663 19 254 661 20 215 675 21 256 660 23 219 674 [OBJ] 2 \"yellow utility\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 838 693 1 837 669 2 808 686 3 762 646 4 805 659 5 742 641 6 733 640 7 721 636 8 744 671 9 701 632 10 691 630 11 680 628 12 670 625 13 661 624 14 651 622 15 641 619 16 630 617 17 660 649 18 612 613 19 602 610 20 593 608 21 584 607 22 573 604 23 564 602 [OBJ] 3 \"green fighter\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 988 341 2 987 338 3 985 338 4 980 338 5 973 338 6 967 338 7 961 336 8 955 336 9 948 336 10 941 336 11 935 336 12 928 336 13 986 345 14 987 347 15 980 347 16 973 347 17 966 347 18 960 347 19 953 347 20 946 347 21 938 346 22 931 346 23 985 341 [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 23 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 2 65535 [OUT] 2 65535 [INTERACTIONS-END] [SCENE-END]\n"
}Parsed prediction
{
"objects": [
{
"id": 0,
"type": "light blue plastic conditioner tube",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 10,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 10,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 1,
"y": 208
},
{
"t": 1,
"x": 21,
"y": 220
},
{
"t": 2,
"x": 25,
"y": 228
},
{
"t": 3,
"x": 29,
"y": 240
},
{
"t": 4,
"x": 13,
"y": 248
},
{
"t": 5,
"x": 1,
"y": 255
},
{
"t": 6,
"x": 1,
"y": 267
},
{
"t": 7,
"x": 5,
"y": 283
},
{
"t": 8,
"x": 5,
"y": 294
},
{
"t": 9,
"x": 1,
"y": 306
},
{
"t": 10,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 1,
"type": "grey and green running shoe",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 1,
"y": 451
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 2,
"type": "green and white plastic portable speaker",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 1,
"y": 498
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 3,
"type": "grey and green running shoe",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 248,
"y": 705
},
{
"t": 1,
"x": 255,
"y": 701
},
{
"t": 3,
"x": 244,
"y": 705
},
{
"t": 5,
"x": 251,
"y": 701
},
{
"t": 6,
"x": 240,
"y": 705
},
{
"t": 8,
"x": 248,
"y": 701
},
{
"t": 10,
"x": 236,
"y": 701
},
{
"t": 12,
"x": 244,
"y": 697
},
{
"t": 13,
"x": 232,
"y": 701
},
{
"t": 14,
"x": 240,
"y": 697
},
{
"t": 16,
"x": 251,
"y": 689
},
{
"t": 17,
"x": 240,
"y": 693
},
{
"t": 19,
"x": 248,
"y": 689
},
{
"t": 20,
"x": 236,
"y": 693
},
{
"t": 21,
"x": 244,
"y": 689
},
{
"t": 23,
"x": 255,
"y": 681
}
],
"birth": 0,
"death": 12
},
{
"id": 4,
"type": "yellow wooden toy construction vehicle set",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 3,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 283,
"y": 419
},
{
"t": 3,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 5,
"type": "green and orange plastic magnifying glass",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 322,
"y": 435
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 6,
"type": "yellow and orange plastic toy airplane with pilot",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 4,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 748,
"y": 439
},
{
"t": 4,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 7,
"type": "green and yellow running shoe",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 755,
"y": 435
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 8,
"type": "green Lego Android robot",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 3,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 767,
"y": 435
},
{
"t": 3,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 9,
"type": "yellow and blue plastic Spongebob toy set",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 3,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 837,
"y": 431
},
{
"t": 3,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 10,
"type": "dark grey and orange athletic shoe",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 3,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 853,
"y": 435
},
{
"t": 3,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 11,
"type": "yellow plastic toy school bus",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 4,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 865,
"y": 431
},
{
"t": 4,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 12,
"type": "green metallic lime squeezer",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 978,
"y": 345
},
{
"t": 1,
"x": 958,
"y": 345
},
{
"t": 2,
"x": 974,
"y": 345
},
{
"t": 3,
"x": 951,
"y": 345
},
{
"t": 4,
"x": 966,
"y": 345
},
{
"t": 5,
"x": 947,
"y": 345
},
{
"t": 6,
"x": 962,
"y": 345
},
{
"t": 7,
"x": 943,
"y": 345
},
{
"t": 8,
"x": 958,
"y": 345
},
{
"t": 9,
"x": 939,
"y": 345
},
{
"t": 10,
"x": 958,
"y": 345
},
{
"t": 11,
"x": 939,
"y": 345
},
{
"t": 12,
"x": 958,
"y": 345
}
],
"birth": 0,
"death": 12
},
{
"id": 13,
"type": "brown and white plastic bald eagle figurine",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 6,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 986,
"y": 435
},
{
"t": 6,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 14,
"type": "red and white cardboard ink cartridge box",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 447
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 12
},
{
"id": 15,
"type": "black and grey athletic shoe with pink accents",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
},
{
"t": 9,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
},
{
"t": 9,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 1,
"x": 998,
"y": 318
},
{
"t": 5,
"x": 986,
"y": 318
},
{
"t": 6,
"x": 998,
"y": 322
},
{
"t": 9,
"x": null,
"y": null
}
],
"birth": 1,
"death": 12
}
],
"interactions": [
{
"id": 0,
"inputs": [
0,
15
],
"outputs": [
0,
15
],
"begins_at": 0,
"ends_at": 11,
"type": "contact"
},
{
"id": 1,
"inputs": [
0,
65535
],
"outputs": [
0,
65535
],
"begins_at": 0,
"ends_at": 11,
"type": "contact"
},
{
"id": 2,
"inputs": [
1,
65535
],
"outputs": [
1,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 10,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 11,
"inputs": [
3,
65535
],
"outputs": [
3,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 12,
"inputs": [
4,
65535
],
"outputs": [
4,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 13,
"inputs": [
5,
65535
],
"outputs": [
5,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 14,
"inputs": [
6,
65535
],
"outputs": [
6,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 15,
"inputs": [
7,
65535
],
"outputs": [
7,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 16,
"inputs": [
8,
65535
],
"outputs": [
8,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 17,
"inputs": [
9,
65535
],
"outputs": [
9,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 3,
"inputs": [
10,
65535
],
"outputs": [
10,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 4,
"inputs": [
11,
65535
],
"outputs": [
11,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 5,
"inputs": [
12,
65535
],
"outputs": [
12,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 6,
"inputs": [
13,
65535
],
"outputs": [
13,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 7,
"inputs": [
14,
65535
],
"outputs": [
14,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 8,
"inputs": [
15,
65535
],
"outputs": [
15,
65535
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 9,
"inputs": [
2,
14
],
"outputs": [
2,
14
],
"begins_at": 10,
"ends_at": 11,
"type": "collision"
}
]
}Parsed ground truth
{
"objects": [
{
"id": 0,
"type": "blue airliner",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 10,
"y": 214
},
{
"t": 1,
"x": 15,
"y": 213
},
{
"t": 2,
"x": 18,
"y": 214
},
{
"t": 5,
"x": 18,
"y": 217
},
{
"t": 6,
"x": 18,
"y": 220
},
{
"t": 7,
"x": 23,
"y": 230
},
{
"t": 8,
"x": 23,
"y": 234
},
{
"t": 9,
"x": 29,
"y": 240
},
{
"t": 10,
"x": 23,
"y": 245
},
{
"t": 11,
"x": 31,
"y": 253
},
{
"t": 12,
"x": 47,
"y": 259
},
{
"t": 13,
"x": 62,
"y": 267
},
{
"t": 14,
"x": 77,
"y": 273
},
{
"t": 15,
"x": 92,
"y": 280
},
{
"t": 16,
"x": 109,
"y": 289
},
{
"t": 17,
"x": 123,
"y": 296
},
{
"t": 18,
"x": 168,
"y": 303
},
{
"t": 19,
"x": 23,
"y": 285
},
{
"t": 20,
"x": 34,
"y": 292
},
{
"t": 21,
"x": 51,
"y": 303
},
{
"t": 22,
"x": 67,
"y": 313
},
{
"t": 23,
"x": 84,
"y": 325
}
],
"birth": 0,
"death": 24
},
{
"id": 1,
"type": "green dirtbike",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 240,
"y": 665
},
{
"t": 6,
"x": 242,
"y": 665
},
{
"t": 10,
"x": 244,
"y": 664
},
{
"t": 13,
"x": 247,
"y": 664
},
{
"t": 15,
"x": 248,
"y": 663
},
{
"t": 17,
"x": 251,
"y": 663
},
{
"t": 19,
"x": 254,
"y": 661
},
{
"t": 20,
"x": 215,
"y": 675
},
{
"t": 21,
"x": 256,
"y": 660
},
{
"t": 23,
"x": 219,
"y": 674
}
],
"birth": 0,
"death": 24
},
{
"id": 2,
"type": "yellow utility",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 838,
"y": 693
},
{
"t": 1,
"x": 837,
"y": 669
},
{
"t": 2,
"x": 808,
"y": 686
},
{
"t": 3,
"x": 762,
"y": 646
},
{
"t": 4,
"x": 805,
"y": 659
},
{
"t": 5,
"x": 742,
"y": 641
},
{
"t": 6,
"x": 733,
"y": 640
},
{
"t": 7,
"x": 721,
"y": 636
},
{
"t": 8,
"x": 744,
"y": 671
},
{
"t": 9,
"x": 701,
"y": 632
},
{
"t": 10,
"x": 691,
"y": 630
},
{
"t": 11,
"x": 680,
"y": 628
},
{
"t": 12,
"x": 670,
"y": 625
},
{
"t": 13,
"x": 661,
"y": 624
},
{
"t": 14,
"x": 651,
"y": 622
},
{
"t": 15,
"x": 641,
"y": 619
},
{
"t": 16,
"x": 630,
"y": 617
},
{
"t": 17,
"x": 660,
"y": 649
},
{
"t": 18,
"x": 612,
"y": 613
},
{
"t": 19,
"x": 602,
"y": 610
},
{
"t": 20,
"x": 593,
"y": 608
},
{
"t": 21,
"x": 584,
"y": 607
},
{
"t": 22,
"x": 573,
"y": 604
},
{
"t": 23,
"x": 564,
"y": 602
}
],
"birth": 0,
"death": 24
},
{
"id": 3,
"type": "green fighter",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 988,
"y": 341
},
{
"t": 2,
"x": 987,
"y": 338
},
{
"t": 3,
"x": 985,
"y": 338
},
{
"t": 4,
"x": 980,
"y": 338
},
{
"t": 5,
"x": 973,
"y": 338
},
{
"t": 6,
"x": 967,
"y": 338
},
{
"t": 7,
"x": 961,
"y": 336
},
{
"t": 8,
"x": 955,
"y": 336
},
{
"t": 9,
"x": 948,
"y": 336
},
{
"t": 10,
"x": 941,
"y": 336
},
{
"t": 11,
"x": 935,
"y": 336
},
{
"t": 12,
"x": 928,
"y": 336
},
{
"t": 13,
"x": 986,
"y": 345
},
{
"t": 14,
"x": 987,
"y": 347
},
{
"t": 15,
"x": 980,
"y": 347
},
{
"t": 16,
"x": 973,
"y": 347
},
{
"t": 17,
"x": 966,
"y": 347
},
{
"t": 18,
"x": 960,
"y": 347
},
{
"t": 19,
"x": 953,
"y": 347
},
{
"t": 20,
"x": 946,
"y": 347
},
{
"t": 21,
"x": 938,
"y": 346
},
{
"t": 22,
"x": 931,
"y": 346
},
{
"t": 23,
"x": 985,
"y": 341
}
],
"birth": 0,
"death": 24
}
],
"interactions": [
{
"id": 0,
"inputs": [
1,
65535
],
"outputs": [
1,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 1,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
}
]
}Complete per-sample evaluator report
{
"metrics": {
"avg_type_similarity": 0.4368785444192847,
"avg_size_diff": null,
"avg_3d_iou": null,
"avg_image_position_px_diff": 440.33804092396934,
"avg_image_position_segmentation_acc": 0.44791666666666663,
"avg_image_position_segmentation_acc_small": null,
"avg_image_position_segmentation_acc_medium": 0.5972222222222222,
"avg_image_position_segmentation_acc_large": 0.0,
"avg_image_position_segmentation_acc_first_frame": 0.25,
"avg_image_position_segmentation_visible_acc": 0.44791666666666663,
"avg_quaternion_geodesic_deg": null,
"visibility_acc": 0.5729166666666667,
"visibility_true_acc": 0.5729166666666667,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 0.5729166666666667,
"vis_acc_from_px_true_acc": 0.5729166666666667,
"vis_acc_from_px_false_acc": null,
"num_matches": 4,
"num_unmatched_pred": 12,
"num_unmatched_target": 0
},
"supports": {
"avg_type_similarity": 4,
"avg_size_diff": 0,
"avg_3d_iou": 0,
"avg_image_position_px_diff": 4,
"avg_image_position_segmentation_acc": 4,
"avg_image_position_segmentation_acc_small": 0,
"avg_image_position_segmentation_acc_medium": 3,
"avg_image_position_segmentation_acc_large": 1,
"avg_image_position_segmentation_acc_first_frame": 4,
"avg_image_position_segmentation_visible_acc": 4,
"avg_quaternion_geodesic_deg": 0,
"visibility_acc": 4,
"visibility_true_acc": 4,
"visibility_false_acc": 0,
"is_in_frame_acc": 4,
"is_in_frame_true_acc": 4,
"is_in_frame_false_acc": 0,
"vis_acc_from_px": 4,
"vis_acc_from_px_true_acc": 4,
"vis_acc_from_px_false_acc": 0
},
"matching": {
"matches": [
{
"target_index": 0,
"pred_index": 6,
"target_id": 0,
"pred_id": 6,
"cost": 1.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
}
]
},
{
"target_index": 1,
"pred_index": 3,
"target_id": 1,
"pred_id": 3,
"cost": 0.7083333333333334,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.5,
"weighted_cost": 0.5,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.20833333333333334,
"weighted_cost": 0.20833333333333334,
"used_missing_value": false
}
]
},
{
"target_index": 2,
"pred_index": 4,
"target_id": 2,
"pred_id": 4,
"cost": 1.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
}
]
},
{
"target_index": 3,
"pred_index": 12,
"target_id": 3,
"pred_id": 12,
"cost": 1.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
}
],
"unmatched_target_ids": [],
"unmatched_pred_ids": [
0,
1,
2,
5,
7,
8,
9,
10,
11,
13,
14,
15
],
"target_type_counts": {
"blue airliner": 1,
"green dirtbike": 1,
"yellow utility": 1,
"green fighter": 1
},
"pred_type_counts": {
"light blue plastic conditioner tube": 1,
"grey and green running shoe": 2,
"green and white plastic portable speaker": 1,
"yellow wooden toy construction vehicle set": 1,
"green and orange plastic magnifying glass": 1,
"yellow and orange plastic toy airplane with pilot": 1,
"green and yellow running shoe": 1,
"green Lego Android robot": 1,
"yellow and blue plastic Spongebob toy set": 1,
"dark grey and orange athletic shoe": 1,
"yellow plastic toy school bus": 1,
"green metallic lime squeezer": 1,
"brown and white plastic bald eagle figurine": 1,
"red and white cardboard ink cartridge box": 1,
"black and grey athletic shoe with pink accents": 1
},
"assignment_backend": "scipy_dense",
"cost_functions": [
"mutual_type",
"same_seg"
],
"cost_weights": {
"mutual_type": 1.0,
"same_seg": 1.0
},
"num_frames": 24
},
"scene_metrics": {
"thresholds": [
0.5,
1.0,
2.0,
3.0
],
"counts_by_threshold": {
"0.5": {
"true_positives": 4,
"false_positives": 0,
"false_negatives": 0
},
"1.0": {
"true_positives": 1,
"false_positives": 3,
"false_negatives": 0
},
"2.0": {
"true_positives": 0,
"false_positives": 4,
"false_negatives": 0
},
"3.0": {
"true_positives": 0,
"false_positives": 4,
"false_negatives": 0
}
},
"jaccard_by_threshold": {
"0.5": 1.0,
"1.0": 0.25,
"2.0": 0.0,
"3.0": 0.0
},
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 2
},
"0.5": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 2
},
"0.75": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 2
},
"1.0": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 2
}
},
"visibility_precision_by_threshold": {
"0.25": 0.125,
"0.5": 0.125,
"0.75": 0.125,
"1.0": 0.125
},
"visibility_recall_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
},
"visibility_f1_by_threshold": {
"0.25": 0.2,
"0.5": 0.2,
"0.75": 0.2,
"1.0": 0.2
},
"visibility_by_motion_group": {
"static": {
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 1,
"false_positives": 1,
"false_negatives": 1
},
"0.5": {
"true_positives": 1,
"false_positives": 1,
"false_negatives": 1
},
"0.75": {
"true_positives": 1,
"false_positives": 1,
"false_negatives": 1
},
"1.0": {
"true_positives": 1,
"false_positives": 1,
"false_negatives": 1
}
},
"visibility_precision_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
},
"visibility_recall_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
},
"visibility_f1_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
}
},
"unknown": {
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 1
},
"0.5": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 1
},
"0.75": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 1
},
"1.0": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 1
}
},
"visibility_precision_by_threshold": {
"0.25": 0.07142857142857142,
"0.5": 0.07142857142857142,
"0.75": 0.07142857142857142,
"1.0": 0.07142857142857142
},
"visibility_recall_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
},
"visibility_f1_by_threshold": {
"0.25": 0.125,
"0.5": 0.125,
"0.75": 0.125,
"1.0": 0.125
}
}
},
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 2
},
"0.5": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 2
},
"0.75": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 2
},
"1.0": {
"true_positives": 1,
"false_positives": 15,
"false_negatives": 3
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.125,
"0.5": 0.125,
"0.75": 0.125,
"1.0": 0.0625
},
"segmentation_recall_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.25
},
"segmentation_f1_by_threshold": {
"0.25": 0.2,
"0.5": 0.2,
"0.75": 0.2,
"1.0": 0.1
},
"segmentation_frame_average_precision": 0.109375,
"segmentation_frame_average_recall": 0.4375,
"segmentation_frame_average_f1": 0.17500000000000002,
"segmentation_first_frame_counts": {
"true_positives": 1,
"false_positives": 15,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.0625,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 0.11764705882352941,
"segmentation_by_motion_group": {
"static": {
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 1,
"false_positives": 1,
"false_negatives": 1
},
"0.5": {
"true_positives": 1,
"false_positives": 1,
"false_negatives": 1
},
"0.75": {
"true_positives": 1,
"false_positives": 1,
"false_negatives": 1
},
"1.0": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 2
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.0
},
"segmentation_recall_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.0
},
"segmentation_f1_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.0
},
"segmentation_frame_average_precision": 0.375,
"segmentation_frame_average_recall": 0.375,
"segmentation_frame_average_f1": 0.375,
"segmentation_first_frame_counts": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.0,
"segmentation_first_frame_recall": 0.0,
"segmentation_first_frame_f1": 0.0
},
"unknown": {
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 1
},
"0.5": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 1
},
"0.75": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 1
},
"1.0": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 1
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.07142857142857142,
"0.5": 0.07142857142857142,
"0.75": 0.07142857142857142,
"1.0": 0.07142857142857142
},
"segmentation_recall_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
},
"segmentation_f1_by_threshold": {
"0.25": 0.125,
"0.5": 0.125,
"0.75": 0.125,
"1.0": 0.125
},
"segmentation_frame_average_precision": 0.07142857142857142,
"segmentation_frame_average_recall": 0.5,
"segmentation_frame_average_f1": 0.125,
"segmentation_first_frame_counts": {
"true_positives": 1,
"false_positives": 13,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.07142857142857142,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 0.13333333333333333
}
}
},
"per_match_metrics": [
{
"target_id": 0,
"pred_id": 6,
"cost": 1.0,
"type_similarity": 0.5142861439781006,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 961.7605322315596,
"image_position_segmentation_acc": 0.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": 0.0,
"image_position_segmentation_acc_first_frame": 0.0,
"image_position_segmentation_visible_acc": 0.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 0.16666666666666666,
"visibility_true_acc": 0.16666666666666666,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 0.16666666666666666,
"vis_acc_from_px_true_acc": 0.16666666666666666,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 1,
"pred_id": 3,
"cost": 0.7083333333333334,
"type_similarity": 0.39966522604487387,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 35.54570895018401,
"image_position_segmentation_acc": 0.7916666666666666,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 0.7916666666666666,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 0.0,
"image_position_segmentation_visible_acc": 0.7916666666666666,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 2,
"pred_id": 4,
"cost": 1.0,
"type_similarity": 0.4666376444087237,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 741.1951711122152,
"image_position_segmentation_acc": 0.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 0.0,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 0.0,
"image_position_segmentation_visible_acc": 0.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 0.125,
"visibility_true_acc": 0.125,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 0.125,
"vis_acc_from_px_true_acc": 0.125,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 3,
"pred_id": 12,
"cost": 1.0,
"type_similarity": 0.36692516324544067,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 22.85075140191864,
"image_position_segmentation_acc": 1.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 1.0,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 1.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
}
],
"interaction_metrics": {
"avg_interaction_temporal_span_giou": 0.5416666666666666,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 2,
"num_unmatched_pred_interactions": 16,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 18,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 18,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.1111111111111111,
"0.5": 0.1111111111111111,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.19999999999999998,
"0.5": 0.19999999999999998,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
},
"0.75": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
},
"1.0": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.1111111111111111,
"0.5": 0.1111111111111111,
"0.75": 0.1111111111111111,
"1.0": 0.1111111111111111
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.19999999999999998,
"0.5": 0.19999999999999998,
"0.75": 0.19999999999999998,
"1.0": 0.19999999999999998
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
},
"0.75": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
},
"1.0": {
"true_positives": 2,
"false_positives": 16,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.1111111111111111,
"0.5": 0.1111111111111111,
"0.75": 0.1111111111111111,
"1.0": 0.1111111111111111
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.19999999999999998,
"0.5": 0.19999999999999998,
"0.75": 0.19999999999999998,
"1.0": 0.19999999999999998
}
},
"interaction_supports": {
"avg_interaction_temporal_span_giou": 2,
"avg_interaction_input_object_jaccard": 2,
"avg_interaction_output_object_jaccard": 2,
"avg_interaction_type_exact_match_acc": 2
},
"interaction_matching": {
"matches": [
{
"target_index": 0,
"pred_index": 11,
"target_id": 0,
"pred_id": 11,
"cost": 0.22916666666666674,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.22916666666666674,
"weighted_cost": 0.22916666666666674,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 1,
"pred_index": 12,
"target_id": 1,
"pred_id": 12,
"cost": 0.22916666666666674,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.22916666666666674,
"weighted_cost": 0.22916666666666674,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
}
],
"unmatched_target_ids": [],
"unmatched_pred_ids": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
13,
14,
15,
16,
17
],
"target_type_counts": {
"contact": 2
},
"pred_type_counts": {
"contact": 17,
"collision": 1
},
"assignment_backend": "scipy_dense",
"cost_functions": [
"type_exact",
"temporal_span",
"input_object_set_mutual_jaccard",
"output_object_set_mutual_jaccard"
],
"cost_weights": {
"type_exact": 1.0,
"temporal_span": 1.0,
"input_object_set_mutual_jaccard": 0.5,
"output_object_set_mutual_jaccard": 0.5
}
},
"per_interaction_match_metrics": [
{
"target_index": 0,
"pred_index": 11,
"target_id": 0,
"pred_id": 11,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.22916666666666674,
"temporal_span_iou": 0.5416666666666666,
"temporal_span_giou": 0.5416666666666666,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 0.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 1,
"pred_index": 12,
"target_id": 1,
"pred_id": 12,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.22916666666666674,
"temporal_span_iou": 0.5416666666666666,
"temporal_span_giou": 0.5416666666666666,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 0.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
}
],
"interaction_metrics_by_participants": {
"object_ground": {
"avg_interaction_temporal_span_giou": 0.5416666666666666,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 2,
"num_unmatched_pred_interactions": 14,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 16,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 16,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.125,
"0.5": 0.125,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.2222222222222222,
"0.5": 0.2222222222222222,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
},
"0.75": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
},
"1.0": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.125,
"0.5": 0.125,
"0.75": 0.125,
"1.0": 0.125
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.2222222222222222,
"0.5": 0.2222222222222222,
"0.75": 0.2222222222222222,
"1.0": 0.2222222222222222
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
},
"0.75": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
},
"1.0": {
"true_positives": 2,
"false_positives": 14,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.125,
"0.5": 0.125,
"0.75": 0.125,
"1.0": 0.125
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.2222222222222222,
"0.5": 0.2222222222222222,
"0.75": 0.2222222222222222,
"1.0": 0.2222222222222222
}
},
"object_object": {
"avg_interaction_temporal_span_giou": null,
"avg_interaction_input_object_jaccard": null,
"avg_interaction_output_object_jaccard": null,
"avg_interaction_type_exact_match_acc": null,
"num_interaction_matches": 0,
"num_unmatched_pred_interactions": 2,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 2,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
}
},
"interaction_supports_by_participants": {
"object_ground": {
"avg_interaction_temporal_span_giou": 2,
"avg_interaction_input_object_jaccard": 2,
"avg_interaction_output_object_jaccard": 2,
"avg_interaction_type_exact_match_acc": 2
},
"object_object": {
"avg_interaction_temporal_span_giou": 0,
"avg_interaction_input_object_jaccard": 0,
"avg_interaction_output_object_jaccard": 0,
"avg_interaction_type_exact_match_acc": 0
}
},
"interaction_metrics_by_type": {
"collision": {
"avg_interaction_temporal_span_giou": null,
"avg_interaction_input_object_jaccard": null,
"avg_interaction_output_object_jaccard": null,
"avg_interaction_type_exact_match_acc": null,
"num_interaction_matches": 0,
"num_unmatched_pred_interactions": 1,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
},
"contact": {
"avg_interaction_temporal_span_giou": 0.5416666666666666,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 2,
"num_unmatched_pred_interactions": 15,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 17,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 17,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"0.75": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"1.0": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.11764705882352941,
"1.0": 0.11764705882352941
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.21052631578947367,
"1.0": 0.21052631578947367
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"0.5": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"0.75": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"1.0": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.11764705882352941,
"1.0": 0.11764705882352941
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.21052631578947367,
"1.0": 0.21052631578947367
}
}
},
"interaction_supports_by_type": {
"collision": {
"avg_interaction_temporal_span_giou": 0,
"avg_interaction_input_object_jaccard": 0,
"avg_interaction_output_object_jaccard": 0,
"avg_interaction_type_exact_match_acc": 0
},
"contact": {
"avg_interaction_temporal_span_giou": 2,
"avg_interaction_input_object_jaccard": 2,
"avg_interaction_output_object_jaccard": 2,
"avg_interaction_type_exact_match_acc": 2
}
}
}Exact evaluation configuration
{
"predictions_jsonl": "../inference_output.jsonl",
"dataset_json": "../test_data/movi_dump_sentinel_sharegpt.json",
"scene_root": "../test_data/movi_dump",
"output_json": "../evaluation_report.json",
"max_samples": null,
"num_workers": 1,
"show_progress": true,
"log_level": "INFO",
"visibility_filter_mode": "any_frame",
"pos2d_mode": "scaled_normalized",
"pos2d_scale": 1000.0,
"image_width": null,
"image_height": null,
"type_embed_model_name": "all-MiniLM-L6-v2",
"scene_similarity_thresholds": [
0.5,
1.0,
2.0,
3.0
],
"scene_segmentation_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"matching": {
"costs": [
{
"name": "mutual_type",
"weight": 1.0
},
{
"name": "same_seg",
"weight": 1.0,
"missing_value": 1.0,
"kwargs": {
"only_use_first_frame": false
}
}
]
},
"interaction_matching": {
"costs": [
{
"name": "type_exact",
"weight": 1.0
},
{
"name": "temporal_span",
"weight": 1.0
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5
}
]
},
"return_per_sample": true
}