STRIVE · 08/28/2026
Dyn generalization · test_24 · super_clevr_1195
Everything fed into evaluation and the full per-sample output it produced.
Dataset row: exact video + user prompt + assistant target
{
"messages": [
{
"role": "user",
"content": "<video>\nAnalyze the provided video and return ONLY a valid sentinel sequence in this format:\n[SCENE-START] [OBJS-START]\n[OBJ] id \"type\" birth death [VIS] t vis ... [IN-FRAME] t is_in_frame ... [PX] t px py ...\n[OBJ] ...\n[OBJS-END]\n[INTERACTIONS-START]\n[INTERACT] id \"type\" begins_at ends_at [IN] in0 in1 ... [OUT] out0 out1 ...\n[INTERACT] ...\n[INTERACTIONS-END] [SCENE-END]\n\nConstraints:\n- Return only the sentinel text (no JSON, XML, markdown, comments, or extra prose).\n- Keep tags exactly as written:\n `[SCENE-START] [OBJS-START] [OBJ] [VIS] [IN-FRAME] [PX] [OBJS-END] [INTERACTIONS-START] [INTERACT] [IN] [OUT] [INTERACTIONS-END] [SCENE-END]`\n- Use quoted strings for `type` (double quotes).\n- `VIS` uses repeated groups of 2 values in order:\n `t vis`.\n `vis` is a boolean value.\n- `IN-FRAME` uses repeated groups of 2 values in order:\n `t is_in_frame`.\n `is_in_frame` is a boolean value.\n- `PX` uses repeated groups of 3 values in order: `t px py`.\n- `px` and `py` are normalized pixel coordinates in the range `[0, 1]`,\n scaled by `1000` and then converted to integers by dropping the decimal part.\n- For occluded or out-of-frame frames under this setting, `PX` may use `t null null`.\n- `IN` and `OUT` contain whitespace-separated integer ids.\n- Use lowercase booleans (`true`/`false`) for `vis` and `is_in_frame`."
},
{
"role": "assistant",
"content": "[SCENE-START] [OBJS-START] [OBJ] 0 \"gray school\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 248 988 1 249 985 2 261 983 3 262 981 4 264 979 5 269 977 6 273 975 7 277 973 8 282 971 9 288 969 10 294 967 11 301 965 12 299 963 13 308 960 15 316 957 17 330 953 18 327 952 19 330 950 20 331 949 21 334 947 22 337 946 23 341 944 [OBJ] 1 \"brown airliner\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 290 46 2 291 48 3 276 48 4 278 51 5 279 54 6 278 56 7 273 55 8 272 60 9 273 70 10 287 96 11 285 102 12 287 114 13 291 130 14 294 144 15 295 157 16 293 168 17 296 184 18 299 201 19 306 222 20 305 235 21 313 257 22 316 275 23 319 294 [OBJ] 2 \"brown tandem\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 365 716 1 367 719 2 371 723 4 382 757 5 369 714 6 380 732 7 393 752 8 384 738 9 387 741 10 401 763 11 405 766 12 398 752 13 411 774 14 415 777 15 415 795 16 422 785 17 425 789 18 429 794 19 433 798 20 425 782 21 427 785 22 444 811 23 449 816 [OBJ] 3 \"blue utility\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 492 871 1 493 884 2 493 869 3 491 823 4 494 861 5 494 864 6 496 871 7 495 856 8 493 810 9 496 851 10 496 843 11 498 855 12 497 839 13 498 831 14 498 836 15 498 844 16 499 842 17 500 839 18 501 835 19 501 832 20 501 829 21 502 826 22 502 822 23 503 820 [OBJ] 4 \"cyan double\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 495 690 11 497 688 12 498 685 15 499 681 16 501 679 18 501 677 20 503 674 22 505 671 23 505 669 [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 23 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 2 \"contact\" 0 23 [IN] 3 65535 [OUT] 3 65535 [INTERACT] 3 \"contact\" 0 23 [IN] 4 65535 [OUT] 4 65535 [INTERACTIONS-END] [SCENE-END]"
}
],
"videos": [
"/mmfs1/gscratch/krishna/suryadv/strive_dynsuperclevr/data/lf/videos/test_24/super_clevr_1195/frames.max_frames_24.mp4"
]
}Inference record: exact prompt + prediction + label + parse debug
{
"prompt": "user\n\nAnalyze the provided video and return ONLY a valid sentinel sequence in this format:\n[SCENE-START] [OBJS-START]\n[OBJ] id \"type\" birth death [VIS] t vis ... [IN-FRAME] t is_in_frame ... [PX] t px py ...\n[OBJ] ...\n[OBJS-END]\n[INTERACTIONS-START]\n[INTERACT] id \"type\" begins_at ends_at [IN] in0 in1 ... [OUT] out0 out1 ...\n[INTERACT] ...\n[INTERACTIONS-END] [SCENE-END]\n\nConstraints:\n- Return only the sentinel text (no JSON, XML, markdown, comments, or extra prose).\n- Keep tags exactly as written:\n `[SCENE-START] [OBJS-START] [OBJ] [VIS] [IN-FRAME] [PX] [OBJS-END] [INTERACTIONS-START] [INTERACT] [IN] [OUT] [INTERACTIONS-END] [SCENE-END]`\n- Use quoted strings for `type` (double quotes).\n- `VIS` uses repeated groups of 2 values in order:\n `t vis`.\n `vis` is a boolean value.\n- `IN-FRAME` uses repeated groups of 2 values in order:\n `t is_in_frame`.\n `is_in_frame` is a boolean value.\n- `PX` uses repeated groups of 3 values in order: `t px py`.\n- `px` and `py` are normalized pixel coordinates in the range `[0, 1]`,\n scaled by `1000` and then converted to integers by dropping the decimal part.\n- For occluded or out-of-frame frames under this setting, `PX` may use `t null null`.\n- `IN` and `OUT` contain whitespace-separated integer ids.\n- Use lowercase booleans (`true`/`false`) for `vis` and `is_in_frame`.\nassistant\n",
"predict": "[SCENE-START] [OBJS-START] [OBJ] 0 \"silver portable drive\" 0 24 [VIS] 0 true 3 false [IN-FRAME] 0 true 3 false [PX] 0 1 361 3 null null [OBJ] 1 \"white and orange cardboard serum box\" 0 24 [VIS] 0 true 11 false [IN-FRAME] 0 true 11 false [PX] 0 228 994 2 251 990 3 267 986 4 279 986 5 291 982 6 302 982 7 314 978 8 326 974 9 337 974 10 349 970 11 null null [OBJ] 2 \"brown and white cardboard latte mix box\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 232 1 1 null null [OBJ] 3 \"red plastic board game with sleeping figure\" 0 24 [VIS] 0 true 3 false [IN-FRAME] 0 true 3 false [PX] 0 271 1 1 263 1 3 null null [OBJ] 4 \"yellow and grey plastic dinosaur defense base\" 0 24 [VIS] 0 true 2 false [IN-FRAME] 0 true 2 false [PX] 0 279 9 1 263 1 2 null null [OBJ] 5 \"white and yellow plastic hyaluronic acid bottle\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 291 1 1 null null [OBJ] 6 \"red and natural wooden dollhouse furniture set\" 0 24 [VIS] 0 true 2 false [IN-FRAME] 0 true 2 false [PX] 0 310 1 1 294 0 2 null null [OBJ] 7 \"brown and orange leather wingtip shoe\" 0 24 [VIS] 0 true 11 false [IN-FRAME] 0 true 11 false [PX] 0 318 9 1 306 5 2 294 1 3 283 1 5 271 5 7 259 9 8 244 1 10 232 1 11 null null [OBJ] 8 \"silver, red, and yellow running shoe\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 322 9 1 null null [OBJ] 9 \"white cardboard remanufactured ink cartridge box\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 326 1 1 null null [OBJ] 10 \"yellow cardboard DVD set\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 357 1 1 null null [OBJ] 11 \"dark grey plastic supplement bottle with orange cap\" 0 24 [VIS] 0 true 5 false [IN-FRAME] 0 true 5 false [PX] 0 361 9 1 349 1 5 null null [OBJ] 12 \"brown and dark grey plastic dinosaur robot toy\" 0 24 [VIS] 0 true 5 false [IN-FRAME] 0 true 5 false [PX] 0 369 9 1 357 1 5 null null [OBJ] 13 \"brown and dark grey leather loafer\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 373 1 1 null null [OBJ] 14 \"wooden garden swing with blue striped cushions\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 380 1 1 null null [OBJ] 15 \"wooden garden swing with blue striped cushions\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 431 1 1 null null [OBJ] 16 \"blue and red running shoe\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 494 880 2 498 869 4 501 861 6 501 849 8 505 837 10 505 826 11 501 818 13 501 806 15 501 794 17 501 783 19 501 771 21 501 759 23 501 751 [OBJ] 17 \"teal plastic Easter basket\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 509 705 2 505 716 4 509 705 6 505 716 12 501 724 13 498 732 14 501 720 15 494 736 16 498 724 18 494 732 20 490 740 22 486 748 [OBJ] 18 \"dark green plastic triceratops figurine\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 595 568 21 595 580 [OBJ] 19 \"red and yellow wooden toy airplane with pilot\" 0 24 [VIS] 0 true 5 false [IN-FRAME] 0 true 5 false [PX] 0 658 1 1 646 1 3 630 1 5 null null [OBJ] 20 \"orange and blue cardboard video game dual pack\" 0 24 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 998 560 1 null null [OBJ] 21 \"yellow cardboard ink cartridge box\" 2 24 [VIS] 0 false 2 true 11 false [IN-FRAME] 0 false 2 true 11 false [PX] 0 null null 2 13 1 3 17 25 4 13 17 5 13 44 6 9 80 7 5 119 8 5 158 9 1 166 10 1 197 11 null null [OBJ] 22 \"white and yellow plastic protein powder jar\" 4 24 [VIS] 0 false 4 true 11 false [IN-FRAME] 0 false 4 true 11 false [PX] 0 null null 4 1 994 5 5 986 7 13 978 8 21 974 10 29 994 11 null null [OBJ] 23 \"white cardboard elderberry syrup box\" 5 24 [VIS] 0 false 5 true 10 false [IN-FRAME] 0 false 5 true 10 false [PX] 0 null null 5 5 998 7 13 994 9 25 998 10 null null [OBJ] 24 \"white cardboard box of fondant\" 6 24 [VIS] 0 false 6 true 10 false [IN-FRAME] 0 false 6 true 10 false [PX] 0 null null 6 1 998 8 9 994 10 null null [OBJ] 25 \"white cardboard box of fondant\" 10 24 [VIS] 0 false 10 true [IN-FRAME] 0 false 10 true [PX] 0 null null 10 416 998 12 423 994 13 431 990 15 439 986 17 447 982 19 451 974 21 455 966 23 458 958 [OBJ] 26 \"blue and white cardboard snack cake box\" 11 24 [VIS] 0 false 11 true 14 false [IN-FRAME] 0 false 11 true 14 false [PX] 0 null null 11 486 1 13 474 1 14 null null [OBJ] 27 \"blue and white cardboard snack cake box\" 12 24 [VIS] 0 false 12 true 14 false [IN-FRAME] 0 false 12 true 14 false [PX] 0 null null 12 490 1 14 null null [OBJ] 28 \"white cardboard box of fondant\" 15 24 [VIS] 0 false 15 true [IN-FRAME] 0 false 15 true [PX] 0 null null 15 498 1 17 486 1 20 474 1 22 462 1 [OBJS-END] [INTERACTIONS-START] [INTERACT] 22 \"contact\" 0 3 [IN] 24 25 [OUT] 24 25 [INTERACT] 0 \"contact\" 0 23 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 29 \"contact\" 0 23 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 34 \"contact\" 0 23 [IN] 3 65535 [OUT] 3 65535 [INTERACT] 35 \"contact\" 0 23 [IN] 4 65535 [OUT] 4 65535 [INTERACT] 36 \"contact\" 0 23 [IN] 5 65535 [OUT] 5 65535 [INTERACT] 37 \"contact\" 0 23 [IN] 6 65535 [OUT] 6 65535 [INTERACT] 38 \"contact\" 0 23 [IN] 7 65535 [OUT] 7 65535 [INTERACT] 39 \"contact\" 0 23 [IN] 8 65535 [OUT] 8 65535 [INTERACT] 40 \"contact\" 0 23 [IN] 9 65535 [OUT] 9 65535 [INTERACT] 2 \"contact\" 0 23 [IN] 10 65535 [OUT] 10 65535 [INTERACT] 3 \"contact\" 0 23 [IN] 11 65535 [OUT] 11 65535 [INTERACT] 4 \"contact\" 0 23 [IN] 12 65535 [OUT] 12 65535 [INTERACT] 5 \"contact\" 0 23 [IN] 13 65535 [OUT] 13 65535 [INTERACT] 6 \"contact\" 0 23 [IN] 14 65535 [OUT] 14 65535 [INTERACT] 7 \"contact\" 0 23 [IN] 15 65535 [OUT] 15 65535 [INTERACT] 8 \"contact\" 0 23 [IN] 16 65535 [OUT] 16 65535 [INTERACT] 9 \"contact\" 0 23 [IN] 17 65535 [OUT] 17 65535 [INTERACT] 10 \"contact\" 0 23 [IN] 18 65535 [OUT] 18 65535 [INTERACT] 11 \"contact\" 0 23 [IN] 19 65535 [OUT] 19 65535 [INTERACT] 12 \"contact\" 0 23 [IN] 20 65535 [OUT] 20 65535 [INTERACT] 13 \"contact\" 0 23 [IN] 21 65535 [OUT] 21 65535 [INTERACT] 14 \"contact\" 0 23 [IN] 22 65535 [OUT] 22 65535 [INTERACT] 15 \"contact\" 0 23 [IN] 23 65535 [OUT] 23 65535 [INTERACT] 23 \"contact\" 0 23 [IN] 24 65535 [OUT] 24 65535 [INTERACT] 24 \"contact\" 0 23 [IN] 25 65535 [OUT] 25 65535 [INTERACT] 25 \"contact\" 0 23 [IN] 26 65535 [OUT] 26 65535 [INTERACT] 26 \"contact\" 0 23 [IN] 27 65535 [OUT] 27 65535 [INTERACT] 27 \"contact\" 0 23 [IN] 28 65535 [OUT] 28 65535 [INTERACT] 16 \"collision\" 4 4 [IN] 24 25 [OUT] 24 25 [INTERACT] 17 \"contact\" 5 6 [IN] 24 25 [OUT] 24 25 [INTERACT] 18 \"collision\" 6 6 [IN] 24 25 [OUT] 24 25 [INTERACT] 19 \"contact\" 7 11 [IN] 24 25 [OUT] 24 25 [INTERACT] 20 \"collision\" 10 10 [IN] 24 25 [OUT] 24 25 [INTERACT] 28 \"collision\" 11 11 [IN] 30 65535 [OUT] 30 65535 [INTERACT] 21 \"contact\" 11 23 [IN] 24 25 [OUT] 24 25 [INTERACT] 30 \"collision\" 15 15 [IN] 30 65535 [OUT] 30 65535 [INTERACT] 31 \"collision\" 17 17 [IN] 30 65535 [OUT] 30 65535 [INTERACT] 32 \"collision\" 19 19 [IN] 30 65535 [OUT] 30 65535 [INTERACT] 33 \"contact\" 20 23 [IN] 30 65535 [OUT] 30 65535 [INTERACTIONS-END] [SCENE-END]",
"label": "[SCENE-START] [OBJS-START] [OBJ] 0 \"gray school\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 248 988 1 249 985 2 261 983 3 262 981 4 264 979 5 269 977 6 273 975 7 277 973 8 282 971 9 288 969 10 294 967 11 301 965 12 299 963 13 308 960 15 316 957 17 330 953 18 327 952 19 330 950 20 331 949 21 334 947 22 337 946 23 341 944 [OBJ] 1 \"brown airliner\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 290 46 2 291 48 3 276 48 4 278 51 5 279 54 6 278 56 7 273 55 8 272 60 9 273 70 10 287 96 11 285 102 12 287 114 13 291 130 14 294 144 15 295 157 16 293 168 17 296 184 18 299 201 19 306 222 20 305 235 21 313 257 22 316 275 23 319 294 [OBJ] 2 \"brown tandem\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 365 716 1 367 719 2 371 723 4 382 757 5 369 714 6 380 732 7 393 752 8 384 738 9 387 741 10 401 763 11 405 766 12 398 752 13 411 774 14 415 777 15 415 795 16 422 785 17 425 789 18 429 794 19 433 798 20 425 782 21 427 785 22 444 811 23 449 816 [OBJ] 3 \"blue utility\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 492 871 1 493 884 2 493 869 3 491 823 4 494 861 5 494 864 6 496 871 7 495 856 8 493 810 9 496 851 10 496 843 11 498 855 12 497 839 13 498 831 14 498 836 15 498 844 16 499 842 17 500 839 18 501 835 19 501 832 20 501 829 21 502 826 22 502 822 23 503 820 [OBJ] 4 \"cyan double\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 495 690 11 497 688 12 498 685 15 499 681 16 501 679 18 501 677 20 503 674 22 505 671 23 505 669 [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 23 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 2 \"contact\" 0 23 [IN] 3 65535 [OUT] 3 65535 [INTERACT] 3 \"contact\" 0 23 [IN] 4 65535 [OUT] 4 65535 [INTERACTIONS-END] [SCENE-END]\n"
}Parsed prediction
{
"objects": [
{
"id": 0,
"type": "silver portable drive",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 3,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 3,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 1,
"y": 361
},
{
"t": 3,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 1,
"type": "white and orange cardboard serum box",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 11,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 11,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 228,
"y": 994
},
{
"t": 2,
"x": 251,
"y": 990
},
{
"t": 3,
"x": 267,
"y": 986
},
{
"t": 4,
"x": 279,
"y": 986
},
{
"t": 5,
"x": 291,
"y": 982
},
{
"t": 6,
"x": 302,
"y": 982
},
{
"t": 7,
"x": 314,
"y": 978
},
{
"t": 8,
"x": 326,
"y": 974
},
{
"t": 9,
"x": 337,
"y": 974
},
{
"t": 10,
"x": 349,
"y": 970
},
{
"t": 11,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 2,
"type": "brown and white cardboard latte mix box",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 232,
"y": 1
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 3,
"type": "red plastic board game with sleeping figure",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 3,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 3,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 271,
"y": 1
},
{
"t": 1,
"x": 263,
"y": 1
},
{
"t": 3,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 4,
"type": "yellow and grey plastic dinosaur defense base",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 2,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 2,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 279,
"y": 9
},
{
"t": 1,
"x": 263,
"y": 1
},
{
"t": 2,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 5,
"type": "white and yellow plastic hyaluronic acid bottle",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 291,
"y": 1
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 6,
"type": "red and natural wooden dollhouse furniture set",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 2,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 2,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 310,
"y": 1
},
{
"t": 1,
"x": 294,
"y": 0
},
{
"t": 2,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 7,
"type": "brown and orange leather wingtip shoe",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 11,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 11,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 318,
"y": 9
},
{
"t": 1,
"x": 306,
"y": 5
},
{
"t": 2,
"x": 294,
"y": 1
},
{
"t": 3,
"x": 283,
"y": 1
},
{
"t": 5,
"x": 271,
"y": 5
},
{
"t": 7,
"x": 259,
"y": 9
},
{
"t": 8,
"x": 244,
"y": 1
},
{
"t": 10,
"x": 232,
"y": 1
},
{
"t": 11,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 8,
"type": "silver, red, and yellow running shoe",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 322,
"y": 9
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 9,
"type": "white cardboard remanufactured ink cartridge box",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 326,
"y": 1
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 10,
"type": "yellow cardboard DVD set",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 357,
"y": 1
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 11,
"type": "dark grey plastic supplement bottle with orange cap",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 5,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 5,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 361,
"y": 9
},
{
"t": 1,
"x": 349,
"y": 1
},
{
"t": 5,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 12,
"type": "brown and dark grey plastic dinosaur robot toy",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 5,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 5,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 369,
"y": 9
},
{
"t": 1,
"x": 357,
"y": 1
},
{
"t": 5,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 13,
"type": "brown and dark grey leather loafer",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 373,
"y": 1
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 14,
"type": "wooden garden swing with blue striped cushions",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 380,
"y": 1
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 15,
"type": "wooden garden swing with blue striped cushions",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 431,
"y": 1
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 16,
"type": "blue and red running shoe",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 494,
"y": 880
},
{
"t": 2,
"x": 498,
"y": 869
},
{
"t": 4,
"x": 501,
"y": 861
},
{
"t": 6,
"x": 501,
"y": 849
},
{
"t": 8,
"x": 505,
"y": 837
},
{
"t": 10,
"x": 505,
"y": 826
},
{
"t": 11,
"x": 501,
"y": 818
},
{
"t": 13,
"x": 501,
"y": 806
},
{
"t": 15,
"x": 501,
"y": 794
},
{
"t": 17,
"x": 501,
"y": 783
},
{
"t": 19,
"x": 501,
"y": 771
},
{
"t": 21,
"x": 501,
"y": 759
},
{
"t": 23,
"x": 501,
"y": 751
}
],
"birth": 0,
"death": 24
},
{
"id": 17,
"type": "teal plastic Easter basket",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 509,
"y": 705
},
{
"t": 2,
"x": 505,
"y": 716
},
{
"t": 4,
"x": 509,
"y": 705
},
{
"t": 6,
"x": 505,
"y": 716
},
{
"t": 12,
"x": 501,
"y": 724
},
{
"t": 13,
"x": 498,
"y": 732
},
{
"t": 14,
"x": 501,
"y": 720
},
{
"t": 15,
"x": 494,
"y": 736
},
{
"t": 16,
"x": 498,
"y": 724
},
{
"t": 18,
"x": 494,
"y": 732
},
{
"t": 20,
"x": 490,
"y": 740
},
{
"t": 22,
"x": 486,
"y": 748
}
],
"birth": 0,
"death": 24
},
{
"id": 18,
"type": "dark green plastic triceratops figurine",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 595,
"y": 568
},
{
"t": 21,
"x": 595,
"y": 580
}
],
"birth": 0,
"death": 24
},
{
"id": 19,
"type": "red and yellow wooden toy airplane with pilot",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 5,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 5,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 658,
"y": 1
},
{
"t": 1,
"x": 646,
"y": 1
},
{
"t": 3,
"x": 630,
"y": 1
},
{
"t": 5,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 20,
"type": "orange and blue cardboard video game dual pack",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 560
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 24
},
{
"id": 21,
"type": "yellow cardboard ink cartridge box",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 2,
"v": true
},
{
"t": 11,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 2,
"v": true
},
{
"t": 11,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 2,
"x": 13,
"y": 1
},
{
"t": 3,
"x": 17,
"y": 25
},
{
"t": 4,
"x": 13,
"y": 17
},
{
"t": 5,
"x": 13,
"y": 44
},
{
"t": 6,
"x": 9,
"y": 80
},
{
"t": 7,
"x": 5,
"y": 119
},
{
"t": 8,
"x": 5,
"y": 158
},
{
"t": 9,
"x": 1,
"y": 166
},
{
"t": 10,
"x": 1,
"y": 197
},
{
"t": 11,
"x": null,
"y": null
}
],
"birth": 2,
"death": 24
},
{
"id": 22,
"type": "white and yellow plastic protein powder jar",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 4,
"v": true
},
{
"t": 11,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 4,
"v": true
},
{
"t": 11,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 4,
"x": 1,
"y": 994
},
{
"t": 5,
"x": 5,
"y": 986
},
{
"t": 7,
"x": 13,
"y": 978
},
{
"t": 8,
"x": 21,
"y": 974
},
{
"t": 10,
"x": 29,
"y": 994
},
{
"t": 11,
"x": null,
"y": null
}
],
"birth": 4,
"death": 24
},
{
"id": 23,
"type": "white cardboard elderberry syrup box",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 5,
"v": true
},
{
"t": 10,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 5,
"v": true
},
{
"t": 10,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 5,
"x": 5,
"y": 998
},
{
"t": 7,
"x": 13,
"y": 994
},
{
"t": 9,
"x": 25,
"y": 998
},
{
"t": 10,
"x": null,
"y": null
}
],
"birth": 5,
"death": 24
},
{
"id": 24,
"type": "white cardboard box of fondant",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 6,
"v": true
},
{
"t": 10,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 6,
"v": true
},
{
"t": 10,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 6,
"x": 1,
"y": 998
},
{
"t": 8,
"x": 9,
"y": 994
},
{
"t": 10,
"x": null,
"y": null
}
],
"birth": 6,
"death": 24
},
{
"id": 25,
"type": "white cardboard box of fondant",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 10,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 10,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 10,
"x": 416,
"y": 998
},
{
"t": 12,
"x": 423,
"y": 994
},
{
"t": 13,
"x": 431,
"y": 990
},
{
"t": 15,
"x": 439,
"y": 986
},
{
"t": 17,
"x": 447,
"y": 982
},
{
"t": 19,
"x": 451,
"y": 974
},
{
"t": 21,
"x": 455,
"y": 966
},
{
"t": 23,
"x": 458,
"y": 958
}
],
"birth": 10,
"death": 24
},
{
"id": 26,
"type": "blue and white cardboard snack cake box",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 11,
"v": true
},
{
"t": 14,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 11,
"v": true
},
{
"t": 14,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 11,
"x": 486,
"y": 1
},
{
"t": 13,
"x": 474,
"y": 1
},
{
"t": 14,
"x": null,
"y": null
}
],
"birth": 11,
"death": 24
},
{
"id": 27,
"type": "blue and white cardboard snack cake box",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 12,
"v": true
},
{
"t": 14,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 12,
"v": true
},
{
"t": 14,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 12,
"x": 490,
"y": 1
},
{
"t": 14,
"x": null,
"y": null
}
],
"birth": 12,
"death": 24
},
{
"id": 28,
"type": "white cardboard box of fondant",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 15,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 15,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 15,
"x": 498,
"y": 1
},
{
"t": 17,
"x": 486,
"y": 1
},
{
"t": 20,
"x": 474,
"y": 1
},
{
"t": 22,
"x": 462,
"y": 1
}
],
"birth": 15,
"death": 24
}
],
"interactions": [
{
"id": 22,
"inputs": [
24,
25
],
"outputs": [
24,
25
],
"begins_at": 0,
"ends_at": 3,
"type": "contact"
},
{
"id": 0,
"inputs": [
0,
65535
],
"outputs": [
0,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 1,
"inputs": [
1,
65535
],
"outputs": [
1,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 29,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 34,
"inputs": [
3,
65535
],
"outputs": [
3,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 35,
"inputs": [
4,
65535
],
"outputs": [
4,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 36,
"inputs": [
5,
65535
],
"outputs": [
5,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 37,
"inputs": [
6,
65535
],
"outputs": [
6,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 38,
"inputs": [
7,
65535
],
"outputs": [
7,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 39,
"inputs": [
8,
65535
],
"outputs": [
8,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 40,
"inputs": [
9,
65535
],
"outputs": [
9,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 2,
"inputs": [
10,
65535
],
"outputs": [
10,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 3,
"inputs": [
11,
65535
],
"outputs": [
11,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 4,
"inputs": [
12,
65535
],
"outputs": [
12,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 5,
"inputs": [
13,
65535
],
"outputs": [
13,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 6,
"inputs": [
14,
65535
],
"outputs": [
14,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 7,
"inputs": [
15,
65535
],
"outputs": [
15,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 8,
"inputs": [
16,
65535
],
"outputs": [
16,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 9,
"inputs": [
17,
65535
],
"outputs": [
17,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 10,
"inputs": [
18,
65535
],
"outputs": [
18,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 11,
"inputs": [
19,
65535
],
"outputs": [
19,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 12,
"inputs": [
20,
65535
],
"outputs": [
20,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 13,
"inputs": [
21,
65535
],
"outputs": [
21,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 14,
"inputs": [
22,
65535
],
"outputs": [
22,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 15,
"inputs": [
23,
65535
],
"outputs": [
23,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 23,
"inputs": [
24,
65535
],
"outputs": [
24,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 24,
"inputs": [
25,
65535
],
"outputs": [
25,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 25,
"inputs": [
26,
65535
],
"outputs": [
26,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 26,
"inputs": [
27,
65535
],
"outputs": [
27,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 27,
"inputs": [
28,
65535
],
"outputs": [
28,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 16,
"inputs": [
24,
25
],
"outputs": [
24,
25
],
"begins_at": 4,
"ends_at": 4,
"type": "collision"
},
{
"id": 17,
"inputs": [
24,
25
],
"outputs": [
24,
25
],
"begins_at": 5,
"ends_at": 6,
"type": "contact"
},
{
"id": 18,
"inputs": [
24,
25
],
"outputs": [
24,
25
],
"begins_at": 6,
"ends_at": 6,
"type": "collision"
},
{
"id": 19,
"inputs": [
24,
25
],
"outputs": [
24,
25
],
"begins_at": 7,
"ends_at": 11,
"type": "contact"
},
{
"id": 20,
"inputs": [
24,
25
],
"outputs": [
24,
25
],
"begins_at": 10,
"ends_at": 10,
"type": "collision"
},
{
"id": 28,
"inputs": [
30,
65535
],
"outputs": [
30,
65535
],
"begins_at": 11,
"ends_at": 11,
"type": "collision"
},
{
"id": 21,
"inputs": [
24,
25
],
"outputs": [
24,
25
],
"begins_at": 11,
"ends_at": 23,
"type": "contact"
},
{
"id": 30,
"inputs": [
30,
65535
],
"outputs": [
30,
65535
],
"begins_at": 15,
"ends_at": 15,
"type": "collision"
},
{
"id": 31,
"inputs": [
30,
65535
],
"outputs": [
30,
65535
],
"begins_at": 17,
"ends_at": 17,
"type": "collision"
},
{
"id": 32,
"inputs": [
30,
65535
],
"outputs": [
30,
65535
],
"begins_at": 19,
"ends_at": 19,
"type": "collision"
},
{
"id": 33,
"inputs": [
30,
65535
],
"outputs": [
30,
65535
],
"begins_at": 20,
"ends_at": 23,
"type": "contact"
}
]
}Parsed ground truth
{
"objects": [
{
"id": 0,
"type": "gray school",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 248,
"y": 988
},
{
"t": 1,
"x": 249,
"y": 985
},
{
"t": 2,
"x": 261,
"y": 983
},
{
"t": 3,
"x": 262,
"y": 981
},
{
"t": 4,
"x": 264,
"y": 979
},
{
"t": 5,
"x": 269,
"y": 977
},
{
"t": 6,
"x": 273,
"y": 975
},
{
"t": 7,
"x": 277,
"y": 973
},
{
"t": 8,
"x": 282,
"y": 971
},
{
"t": 9,
"x": 288,
"y": 969
},
{
"t": 10,
"x": 294,
"y": 967
},
{
"t": 11,
"x": 301,
"y": 965
},
{
"t": 12,
"x": 299,
"y": 963
},
{
"t": 13,
"x": 308,
"y": 960
},
{
"t": 15,
"x": 316,
"y": 957
},
{
"t": 17,
"x": 330,
"y": 953
},
{
"t": 18,
"x": 327,
"y": 952
},
{
"t": 19,
"x": 330,
"y": 950
},
{
"t": 20,
"x": 331,
"y": 949
},
{
"t": 21,
"x": 334,
"y": 947
},
{
"t": 22,
"x": 337,
"y": 946
},
{
"t": 23,
"x": 341,
"y": 944
}
],
"birth": 0,
"death": 24
},
{
"id": 1,
"type": "brown airliner",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 290,
"y": 46
},
{
"t": 2,
"x": 291,
"y": 48
},
{
"t": 3,
"x": 276,
"y": 48
},
{
"t": 4,
"x": 278,
"y": 51
},
{
"t": 5,
"x": 279,
"y": 54
},
{
"t": 6,
"x": 278,
"y": 56
},
{
"t": 7,
"x": 273,
"y": 55
},
{
"t": 8,
"x": 272,
"y": 60
},
{
"t": 9,
"x": 273,
"y": 70
},
{
"t": 10,
"x": 287,
"y": 96
},
{
"t": 11,
"x": 285,
"y": 102
},
{
"t": 12,
"x": 287,
"y": 114
},
{
"t": 13,
"x": 291,
"y": 130
},
{
"t": 14,
"x": 294,
"y": 144
},
{
"t": 15,
"x": 295,
"y": 157
},
{
"t": 16,
"x": 293,
"y": 168
},
{
"t": 17,
"x": 296,
"y": 184
},
{
"t": 18,
"x": 299,
"y": 201
},
{
"t": 19,
"x": 306,
"y": 222
},
{
"t": 20,
"x": 305,
"y": 235
},
{
"t": 21,
"x": 313,
"y": 257
},
{
"t": 22,
"x": 316,
"y": 275
},
{
"t": 23,
"x": 319,
"y": 294
}
],
"birth": 0,
"death": 24
},
{
"id": 2,
"type": "brown tandem",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 365,
"y": 716
},
{
"t": 1,
"x": 367,
"y": 719
},
{
"t": 2,
"x": 371,
"y": 723
},
{
"t": 4,
"x": 382,
"y": 757
},
{
"t": 5,
"x": 369,
"y": 714
},
{
"t": 6,
"x": 380,
"y": 732
},
{
"t": 7,
"x": 393,
"y": 752
},
{
"t": 8,
"x": 384,
"y": 738
},
{
"t": 9,
"x": 387,
"y": 741
},
{
"t": 10,
"x": 401,
"y": 763
},
{
"t": 11,
"x": 405,
"y": 766
},
{
"t": 12,
"x": 398,
"y": 752
},
{
"t": 13,
"x": 411,
"y": 774
},
{
"t": 14,
"x": 415,
"y": 777
},
{
"t": 15,
"x": 415,
"y": 795
},
{
"t": 16,
"x": 422,
"y": 785
},
{
"t": 17,
"x": 425,
"y": 789
},
{
"t": 18,
"x": 429,
"y": 794
},
{
"t": 19,
"x": 433,
"y": 798
},
{
"t": 20,
"x": 425,
"y": 782
},
{
"t": 21,
"x": 427,
"y": 785
},
{
"t": 22,
"x": 444,
"y": 811
},
{
"t": 23,
"x": 449,
"y": 816
}
],
"birth": 0,
"death": 24
},
{
"id": 3,
"type": "blue utility",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 492,
"y": 871
},
{
"t": 1,
"x": 493,
"y": 884
},
{
"t": 2,
"x": 493,
"y": 869
},
{
"t": 3,
"x": 491,
"y": 823
},
{
"t": 4,
"x": 494,
"y": 861
},
{
"t": 5,
"x": 494,
"y": 864
},
{
"t": 6,
"x": 496,
"y": 871
},
{
"t": 7,
"x": 495,
"y": 856
},
{
"t": 8,
"x": 493,
"y": 810
},
{
"t": 9,
"x": 496,
"y": 851
},
{
"t": 10,
"x": 496,
"y": 843
},
{
"t": 11,
"x": 498,
"y": 855
},
{
"t": 12,
"x": 497,
"y": 839
},
{
"t": 13,
"x": 498,
"y": 831
},
{
"t": 14,
"x": 498,
"y": 836
},
{
"t": 15,
"x": 498,
"y": 844
},
{
"t": 16,
"x": 499,
"y": 842
},
{
"t": 17,
"x": 500,
"y": 839
},
{
"t": 18,
"x": 501,
"y": 835
},
{
"t": 19,
"x": 501,
"y": 832
},
{
"t": 20,
"x": 501,
"y": 829
},
{
"t": 21,
"x": 502,
"y": 826
},
{
"t": 22,
"x": 502,
"y": 822
},
{
"t": 23,
"x": 503,
"y": 820
}
],
"birth": 0,
"death": 24
},
{
"id": 4,
"type": "cyan double",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 495,
"y": 690
},
{
"t": 11,
"x": 497,
"y": 688
},
{
"t": 12,
"x": 498,
"y": 685
},
{
"t": 15,
"x": 499,
"y": 681
},
{
"t": 16,
"x": 501,
"y": 679
},
{
"t": 18,
"x": 501,
"y": 677
},
{
"t": 20,
"x": 503,
"y": 674
},
{
"t": 22,
"x": 505,
"y": 671
},
{
"t": 23,
"x": 505,
"y": 669
}
],
"birth": 0,
"death": 24
}
],
"interactions": [
{
"id": 0,
"inputs": [
0,
65535
],
"outputs": [
0,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 1,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 2,
"inputs": [
3,
65535
],
"outputs": [
3,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 3,
"inputs": [
4,
65535
],
"outputs": [
4,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
}
]
}Complete per-sample evaluator report
{
"metrics": {
"avg_type_similarity": 0.35556446778668993,
"avg_size_diff": null,
"avg_3d_iou": null,
"avg_image_position_px_diff": 438.31226853866184,
"avg_image_position_segmentation_acc": 0.33333333333333337,
"avg_image_position_segmentation_acc_small": null,
"avg_image_position_segmentation_acc_medium": 0.3541666666666667,
"avg_image_position_segmentation_acc_large": 0.3194444444444445,
"avg_image_position_segmentation_acc_first_frame": 0.4,
"avg_image_position_segmentation_visible_acc": 0.33333333333333337,
"avg_quaternion_geodesic_deg": null,
"visibility_acc": 0.5583333333333333,
"visibility_true_acc": 0.5583333333333333,
"visibility_false_acc": null,
"is_in_frame_acc": 0.5583333333333333,
"is_in_frame_true_acc": 0.5583333333333333,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 0.5583333333333333,
"vis_acc_from_px_true_acc": 0.5583333333333333,
"vis_acc_from_px_false_acc": null,
"num_matches": 5,
"num_unmatched_pred": 24,
"num_unmatched_target": 0
},
"supports": {
"avg_type_similarity": 5,
"avg_size_diff": 0,
"avg_3d_iou": 0,
"avg_image_position_px_diff": 5,
"avg_image_position_segmentation_acc": 5,
"avg_image_position_segmentation_acc_small": 0,
"avg_image_position_segmentation_acc_medium": 2,
"avg_image_position_segmentation_acc_large": 3,
"avg_image_position_segmentation_acc_first_frame": 5,
"avg_image_position_segmentation_visible_acc": 5,
"avg_quaternion_geodesic_deg": 0,
"visibility_acc": 5,
"visibility_true_acc": 5,
"visibility_false_acc": 0,
"is_in_frame_acc": 5,
"is_in_frame_true_acc": 5,
"is_in_frame_false_acc": 0,
"vis_acc_from_px": 5,
"vis_acc_from_px_true_acc": 5,
"vis_acc_from_px_false_acc": 0
},
"matching": {
"matches": [
{
"target_index": 0,
"pred_index": 3,
"target_id": 0,
"pred_id": 3,
"cost": 1.5,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.5,
"weighted_cost": 0.5,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
}
]
},
{
"target_index": 1,
"pred_index": 19,
"target_id": 1,
"pred_id": 19,
"cost": 1.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
}
]
},
{
"target_index": 2,
"pred_index": 7,
"target_id": 2,
"pred_id": 7,
"cost": 1.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
}
]
},
{
"target_index": 3,
"pred_index": 16,
"target_id": 3,
"pred_id": 16,
"cost": 0.2916666666666667,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.2916666666666667,
"weighted_cost": 0.2916666666666667,
"used_missing_value": false
}
]
},
{
"target_index": 4,
"pred_index": 17,
"target_id": 4,
"pred_id": 17,
"cost": 1.0416666666666667,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.041666666666666664,
"weighted_cost": 0.041666666666666664,
"used_missing_value": false
}
]
}
],
"unmatched_target_ids": [],
"unmatched_pred_ids": [
0,
1,
2,
4,
5,
6,
8,
9,
10,
11,
12,
13,
14,
15,
18,
20,
21,
22,
23,
24,
25,
26,
27,
28
],
"target_type_counts": {
"gray school": 1,
"brown airliner": 1,
"brown tandem": 1,
"blue utility": 1,
"cyan double": 1
},
"pred_type_counts": {
"silver portable drive": 1,
"white and orange cardboard serum box": 1,
"brown and white cardboard latte mix box": 1,
"red plastic board game with sleeping figure": 1,
"yellow and grey plastic dinosaur defense base": 1,
"white and yellow plastic hyaluronic acid bottle": 1,
"red and natural wooden dollhouse furniture set": 1,
"brown and orange leather wingtip shoe": 1,
"silver, red, and yellow running shoe": 1,
"white cardboard remanufactured ink cartridge box": 1,
"yellow cardboard DVD set": 1,
"dark grey plastic supplement bottle with orange cap": 1,
"brown and dark grey plastic dinosaur robot toy": 1,
"brown and dark grey leather loafer": 1,
"wooden garden swing with blue striped cushions": 2,
"blue and red running shoe": 1,
"teal plastic Easter basket": 1,
"dark green plastic triceratops figurine": 1,
"red and yellow wooden toy airplane with pilot": 1,
"orange and blue cardboard video game dual pack": 1,
"yellow cardboard ink cartridge box": 1,
"white and yellow plastic protein powder jar": 1,
"white cardboard elderberry syrup box": 1,
"white cardboard box of fondant": 3,
"blue and white cardboard snack cake box": 2
},
"assignment_backend": "scipy_dense",
"cost_functions": [
"mutual_type",
"same_seg"
],
"cost_weights": {
"mutual_type": 1.0,
"same_seg": 1.0
},
"num_frames": 24
},
"scene_metrics": {
"thresholds": [
0.5,
1.0,
2.0,
3.0
],
"counts_by_threshold": {
"0.5": {
"true_positives": 4,
"false_positives": 1,
"false_negatives": 0
},
"1.0": {
"true_positives": 1,
"false_positives": 4,
"false_negatives": 0
},
"2.0": {
"true_positives": 0,
"false_positives": 5,
"false_negatives": 0
},
"3.0": {
"true_positives": 0,
"false_positives": 5,
"false_negatives": 0
}
},
"jaccard_by_threshold": {
"0.5": 0.8,
"1.0": 0.2,
"2.0": 0.0,
"3.0": 0.0
},
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 26,
"false_negatives": 2
},
"0.5": {
"true_positives": 2,
"false_positives": 27,
"false_negatives": 3
},
"0.75": {
"true_positives": 2,
"false_positives": 27,
"false_negatives": 3
},
"1.0": {
"true_positives": 2,
"false_positives": 27,
"false_negatives": 3
}
},
"visibility_precision_by_threshold": {
"0.25": 0.10344827586206896,
"0.5": 0.06896551724137931,
"0.75": 0.06896551724137931,
"1.0": 0.06896551724137931
},
"visibility_recall_by_threshold": {
"0.25": 0.6,
"0.5": 0.4,
"0.75": 0.4,
"1.0": 0.4
},
"visibility_f1_by_threshold": {
"0.25": 0.17647058823529413,
"0.5": 0.1176470588235294,
"0.75": 0.1176470588235294,
"1.0": 0.1176470588235294
},
"visibility_by_motion_group": {
"static": {
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 1,
"false_negatives": 1
},
"0.5": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 2
},
"0.75": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 2
},
"1.0": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 2
}
},
"visibility_precision_by_threshold": {
"0.25": 0.75,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
},
"visibility_recall_by_threshold": {
"0.25": 0.75,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
},
"visibility_f1_by_threshold": {
"0.25": 0.75,
"0.5": 0.5,
"0.75": 0.5,
"1.0": 0.5
}
},
"unknown": {
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 1
},
"0.5": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 1
},
"0.75": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 1
},
"1.0": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 1
}
},
"visibility_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"visibility_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"visibility_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
}
},
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 27,
"false_negatives": 3
},
"0.5": {
"true_positives": 2,
"false_positives": 27,
"false_negatives": 3
},
"0.75": {
"true_positives": 1,
"false_positives": 28,
"false_negatives": 4
},
"1.0": {
"true_positives": 0,
"false_positives": 29,
"false_negatives": 5
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.06896551724137931,
"0.5": 0.06896551724137931,
"0.75": 0.034482758620689655,
"1.0": 0.0
},
"segmentation_recall_by_threshold": {
"0.25": 0.4,
"0.5": 0.4,
"0.75": 0.2,
"1.0": 0.0
},
"segmentation_f1_by_threshold": {
"0.25": 0.1176470588235294,
"0.5": 0.1176470588235294,
"0.75": 0.0588235294117647,
"1.0": 0.0
},
"segmentation_frame_average_precision": 0.04310344827586207,
"segmentation_frame_average_recall": 0.25,
"segmentation_frame_average_f1": 0.07352941176470587,
"segmentation_first_frame_counts": {
"true_positives": 2,
"false_positives": 27,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.06896551724137931,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 0.12903225806451613,
"segmentation_by_motion_group": {
"static": {
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 2
},
"0.5": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 2
},
"0.75": {
"true_positives": 1,
"false_positives": 3,
"false_negatives": 3
},
"1.0": {
"true_positives": 0,
"false_positives": 4,
"false_negatives": 4
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.25,
"1.0": 0.0
},
"segmentation_recall_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.25,
"1.0": 0.0
},
"segmentation_f1_by_threshold": {
"0.25": 0.5,
"0.5": 0.5,
"0.75": 0.25,
"1.0": 0.0
},
"segmentation_frame_average_precision": 0.3125,
"segmentation_frame_average_recall": 0.3125,
"segmentation_frame_average_f1": 0.3125,
"segmentation_first_frame_counts": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.5,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 0.6666666666666666
},
"unknown": {
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 1
},
"0.5": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 1
},
"0.75": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 1
},
"1.0": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 1
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_frame_average_precision": 0.0,
"segmentation_frame_average_recall": 0.0,
"segmentation_frame_average_f1": 0.0,
"segmentation_first_frame_counts": {
"true_positives": 0,
"false_positives": 25,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.0,
"segmentation_first_frame_recall": 0.0,
"segmentation_first_frame_f1": 0.0
}
}
},
"per_match_metrics": [
{
"target_id": 0,
"pred_id": 3,
"cost": 1.5,
"type_similarity": 0.2095313160480917,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 944.1857845576178,
"image_position_segmentation_acc": 0.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": 0.0,
"image_position_segmentation_acc_first_frame": 0.0,
"image_position_segmentation_visible_acc": 0.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 0.125,
"visibility_true_acc": 0.125,
"visibility_false_acc": null,
"is_in_frame_acc": 0.125,
"is_in_frame_true_acc": 0.125,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 0.125,
"vis_acc_from_px_true_acc": 0.125,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 1,
"pred_id": 19,
"cost": 1.0,
"type_similarity": 0.4801363781300524,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 458.80915147697885,
"image_position_segmentation_acc": 0.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": 0.0,
"image_position_segmentation_acc_first_frame": 0.0,
"image_position_segmentation_visible_acc": 0.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 0.20833333333333334,
"visibility_true_acc": 0.20833333333333334,
"visibility_false_acc": null,
"is_in_frame_acc": 0.20833333333333334,
"is_in_frame_true_acc": 0.20833333333333334,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 0.20833333333333334,
"vis_acc_from_px_true_acc": 0.20833333333333334,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 2,
"pred_id": 7,
"cost": 1.0,
"type_similarity": 0.42385838039243984,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 715.083519617455,
"image_position_segmentation_acc": 0.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 0.0,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 0.0,
"image_position_segmentation_visible_acc": 0.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 0.4583333333333333,
"visibility_true_acc": 0.4583333333333333,
"visibility_false_acc": null,
"is_in_frame_acc": 0.4583333333333333,
"is_in_frame_true_acc": 0.4583333333333333,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 0.4583333333333333,
"vis_acc_from_px_true_acc": 0.4583333333333333,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 3,
"pred_id": 16,
"cost": 0.2916666666666667,
"type_similarity": 0.48094260655657767,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 33.14164402788749,
"image_position_segmentation_acc": 0.7083333333333334,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 0.7083333333333334,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 0.7083333333333334,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 4,
"pred_id": 17,
"cost": 1.0416666666666667,
"type_similarity": 0.18335365780628804,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 40.34124301336984,
"image_position_segmentation_acc": 0.9583333333333334,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": 0.9583333333333334,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 0.9583333333333334,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
}
],
"interaction_metrics": {
"avg_interaction_temporal_span_giou": 1.0,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 4,
"num_unmatched_pred_interactions": 37,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0975609756097561,
"0.5": 0.0975609756097561,
"0.75": 0.0975609756097561,
"1.0": 0.0975609756097561
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.17777777777777776,
"0.5": 0.17777777777777776,
"0.75": 0.17777777777777776,
"1.0": 0.17777777777777776
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0975609756097561,
"0.5": 0.0975609756097561,
"0.75": 0.0975609756097561,
"1.0": 0.0975609756097561
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.17777777777777776,
"0.5": 0.17777777777777776,
"0.75": 0.17777777777777776,
"1.0": 0.17777777777777776
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 37,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0975609756097561,
"0.5": 0.0975609756097561,
"0.75": 0.0975609756097561,
"1.0": 0.0975609756097561
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.17777777777777776,
"0.5": 0.17777777777777776,
"0.75": 0.17777777777777776,
"1.0": 0.17777777777777776
}
},
"interaction_supports": {
"avg_interaction_temporal_span_giou": 4,
"avg_interaction_input_object_jaccard": 4,
"avg_interaction_output_object_jaccard": 4,
"avg_interaction_type_exact_match_acc": 4
},
"interaction_matching": {
"matches": [
{
"target_index": 0,
"pred_index": 34,
"target_id": 0,
"pred_id": 34,
"cost": 0.0,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 1,
"pred_index": 38,
"target_id": 1,
"pred_id": 38,
"cost": 0.0,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 2,
"pred_index": 8,
"target_id": 2,
"pred_id": 8,
"cost": 0.0,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 3,
"pred_index": 9,
"target_id": 3,
"pred_id": 9,
"cost": 0.0,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
}
],
"unmatched_target_ids": [],
"unmatched_pred_ids": [
0,
1,
2,
3,
4,
5,
6,
7,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
35,
36,
37,
39,
40
],
"target_type_counts": {
"contact": 4
},
"pred_type_counts": {
"contact": 34,
"collision": 7
},
"assignment_backend": "scipy_dense",
"cost_functions": [
"type_exact",
"temporal_span",
"input_object_set_mutual_jaccard",
"output_object_set_mutual_jaccard"
],
"cost_weights": {
"type_exact": 1.0,
"temporal_span": 1.0,
"input_object_set_mutual_jaccard": 0.5,
"output_object_set_mutual_jaccard": 0.5
}
},
"per_interaction_match_metrics": [
{
"target_index": 0,
"pred_index": 34,
"target_id": 0,
"pred_id": 34,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.0,
"temporal_span_iou": 1.0,
"temporal_span_giou": 1.0,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 1.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 1,
"pred_index": 38,
"target_id": 1,
"pred_id": 38,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.0,
"temporal_span_iou": 1.0,
"temporal_span_giou": 1.0,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 1.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 2,
"pred_index": 8,
"target_id": 2,
"pred_id": 8,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.0,
"temporal_span_iou": 1.0,
"temporal_span_giou": 1.0,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 1.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 3,
"pred_index": 9,
"target_id": 3,
"pred_id": 9,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.0,
"temporal_span_iou": 1.0,
"temporal_span_giou": 1.0,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 1.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
}
],
"interaction_metrics_by_participants": {
"object_ground": {
"avg_interaction_temporal_span_giou": 1.0,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 4,
"num_unmatched_pred_interactions": 30,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.11764705882352941,
"1.0": 0.11764705882352941
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.21052631578947367,
"1.0": 0.21052631578947367
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.11764705882352941,
"1.0": 0.11764705882352941
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.21052631578947367,
"1.0": 0.21052631578947367
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.11764705882352941,
"1.0": 0.11764705882352941
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.21052631578947367,
"1.0": 0.21052631578947367
}
},
"object_object": {
"avg_interaction_temporal_span_giou": null,
"avg_interaction_input_object_jaccard": null,
"avg_interaction_output_object_jaccard": null,
"avg_interaction_type_exact_match_acc": null,
"num_interaction_matches": 0,
"num_unmatched_pred_interactions": 7,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
}
},
"interaction_supports_by_participants": {
"object_ground": {
"avg_interaction_temporal_span_giou": 4,
"avg_interaction_input_object_jaccard": 4,
"avg_interaction_output_object_jaccard": 4,
"avg_interaction_type_exact_match_acc": 4
},
"object_object": {
"avg_interaction_temporal_span_giou": 0,
"avg_interaction_input_object_jaccard": 0,
"avg_interaction_output_object_jaccard": 0,
"avg_interaction_type_exact_match_acc": 0
}
},
"interaction_metrics_by_type": {
"collision": {
"avg_interaction_temporal_span_giou": null,
"avg_interaction_input_object_jaccard": null,
"avg_interaction_output_object_jaccard": null,
"avg_interaction_type_exact_match_acc": null,
"num_interaction_matches": 0,
"num_unmatched_pred_interactions": 7,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
},
"contact": {
"avg_interaction_temporal_span_giou": 1.0,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 4,
"num_unmatched_pred_interactions": 30,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.11764705882352941,
"1.0": 0.11764705882352941
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.21052631578947367,
"1.0": 0.21052631578947367
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.11764705882352941,
"1.0": 0.11764705882352941
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.21052631578947367,
"1.0": 0.21052631578947367
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 30,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.11764705882352941,
"0.5": 0.11764705882352941,
"0.75": 0.11764705882352941,
"1.0": 0.11764705882352941
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.21052631578947367,
"0.5": 0.21052631578947367,
"0.75": 0.21052631578947367,
"1.0": 0.21052631578947367
}
}
},
"interaction_supports_by_type": {
"collision": {
"avg_interaction_temporal_span_giou": 0,
"avg_interaction_input_object_jaccard": 0,
"avg_interaction_output_object_jaccard": 0,
"avg_interaction_type_exact_match_acc": 0
},
"contact": {
"avg_interaction_temporal_span_giou": 4,
"avg_interaction_input_object_jaccard": 4,
"avg_interaction_output_object_jaccard": 4,
"avg_interaction_type_exact_match_acc": 4
}
}
}Exact evaluation configuration
{
"predictions_jsonl": "../inference_output.jsonl",
"dataset_json": "../test_data/movi_dump_sentinel_sharegpt.json",
"scene_root": "../test_data/movi_dump",
"output_json": "../evaluation_report.json",
"max_samples": null,
"num_workers": 1,
"show_progress": true,
"log_level": "INFO",
"visibility_filter_mode": "any_frame",
"pos2d_mode": "scaled_normalized",
"pos2d_scale": 1000.0,
"image_width": null,
"image_height": null,
"type_embed_model_name": "all-MiniLM-L6-v2",
"scene_similarity_thresholds": [
0.5,
1.0,
2.0,
3.0
],
"scene_segmentation_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"matching": {
"costs": [
{
"name": "mutual_type",
"weight": 1.0
},
{
"name": "same_seg",
"weight": 1.0,
"missing_value": 1.0,
"kwargs": {
"only_use_first_frame": false
}
}
]
},
"interaction_matching": {
"costs": [
{
"name": "type_exact",
"weight": 1.0
},
{
"name": "temporal_span",
"weight": 1.0
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5
}
]
},
"return_per_sample": true
}