STRIVE · 08/28/2026
Dyn generalization · val_24 · super_clevr_1034
Everything fed into evaluation and the full per-sample output it produced.
Dataset row: exact video + user prompt + assistant target
{
"messages": [
{
"role": "user",
"content": "<video>\nAnalyze the provided video and return ONLY a valid sentinel sequence in this format:\n[SCENE-START] [OBJS-START]\n[OBJ] id \"type\" birth death [VIS] t vis ... [IN-FRAME] t is_in_frame ... [PX] t px py ...\n[OBJ] ...\n[OBJS-END]\n[INTERACTIONS-START]\n[INTERACT] id \"type\" begins_at ends_at [IN] in0 in1 ... [OUT] out0 out1 ...\n[INTERACT] ...\n[INTERACTIONS-END] [SCENE-END]\n\nConstraints:\n- Return only the sentinel text (no JSON, XML, markdown, comments, or extra prose).\n- Keep tags exactly as written:\n `[SCENE-START] [OBJS-START] [OBJ] [VIS] [IN-FRAME] [PX] [OBJS-END] [INTERACTIONS-START] [INTERACT] [IN] [OUT] [INTERACTIONS-END] [SCENE-END]`\n- Use quoted strings for `type` (double quotes).\n- `VIS` uses repeated groups of 2 values in order:\n `t vis`.\n `vis` is a boolean value.\n- `IN-FRAME` uses repeated groups of 2 values in order:\n `t is_in_frame`.\n `is_in_frame` is a boolean value.\n- `PX` uses repeated groups of 3 values in order: `t px py`.\n- `px` and `py` are normalized pixel coordinates in the range `[0, 1]`,\n scaled by `1000` and then converted to integers by dropping the decimal part.\n- For occluded or out-of-frame frames under this setting, `PX` may use `t null null`.\n- `IN` and `OUT` contain whitespace-separated integer ids.\n- Use lowercase booleans (`true`/`false`) for `vis` and `is_in_frame`."
},
{
"role": "assistant",
"content": "[SCENE-START] [OBJS-START] [OBJ] 0 \"purple cruiser\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 55 913 1 69 905 2 83 899 3 97 893 4 110 886 5 123 880 6 133 874 7 145 865 8 158 861 9 169 857 10 181 852 11 193 847 12 203 842 13 214 836 14 224 831 15 239 825 16 248 821 17 258 816 18 267 811 19 274 808 20 283 803 21 292 799 22 303 794 23 310 791 [OBJ] 1 \"purple airliner\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 319 913 [OBJ] 2 \"blue airliner\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 421 359 1 445 366 4 428 360 5 430 360 6 451 367 7 439 364 9 437 361 10 463 369 11 466 369 12 446 365 13 466 369 14 472 370 17 464 368 19 483 372 20 466 368 22 492 373 [OBJ] 3 \"blue truck\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 446 801 12 445 795 22 444 786 [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 23 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 2 \"contact\" 0 23 [IN] 3 65535 [OUT] 3 65535 [INTERACTIONS-END] [SCENE-END]"
}
],
"videos": [
"/mmfs1/gscratch/krishna/suryadv/strive_dynsuperclevr/data/lf/videos/val_24/super_clevr_1034/frames.max_frames_24.mp4"
]
}Inference record: exact prompt + prediction + label + parse debug
{
"prompt": "user\n\nAnalyze the provided video and return ONLY a valid sentinel sequence in this format:\n[SCENE-START] [OBJS-START]\n[OBJ] id \"type\" birth death [VIS] t vis ... [IN-FRAME] t is_in_frame ... [PX] t px py ...\n[OBJ] ...\n[OBJS-END]\n[INTERACTIONS-START]\n[INTERACT] id \"type\" begins_at ends_at [IN] in0 in1 ... [OUT] out0 out1 ...\n[INTERACT] ...\n[INTERACTIONS-END] [SCENE-END]\n\nConstraints:\n- Return only the sentinel text (no JSON, XML, markdown, comments, or extra prose).\n- Keep tags exactly as written:\n `[SCENE-START] [OBJS-START] [OBJ] [VIS] [IN-FRAME] [PX] [OBJS-END] [INTERACTIONS-START] [INTERACT] [IN] [OUT] [INTERACTIONS-END] [SCENE-END]`\n- Use quoted strings for `type` (double quotes).\n- `VIS` uses repeated groups of 2 values in order:\n `t vis`.\n `vis` is a boolean value.\n- `IN-FRAME` uses repeated groups of 2 values in order:\n `t is_in_frame`.\n `is_in_frame` is a boolean value.\n- `PX` uses repeated groups of 3 values in order: `t px py`.\n- `px` and `py` are normalized pixel coordinates in the range `[0, 1]`,\n scaled by `1000` and then converted to integers by dropping the decimal part.\n- For occluded or out-of-frame frames under this setting, `PX` may use `t null null`.\n- `IN` and `OUT` contain whitespace-separated integer ids.\n- Use lowercase booleans (`true`/`false`) for `vis` and `is_in_frame`.\nassistant\n",
"predict": "[SCENE-START] [OBJS-START] [OBJ] 0 \"pink and purple soccer cleat\" 0 60 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 1 908 1 null null [OBJ] 1 \"pink and purple plastic package of overnight pads\" 0 60 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 5 994 1 1 986 2 5 994 11 1 986 15 1 974 20 1 962 22 5 994 23 1 982 [OBJ] 2 \"pink and purple plastic package of overnight pads\" 0 60 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 9 986 1 null null [OBJ] 3 \"pink and purple plastic package of overnight pads\" 0 60 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 25 994 1 null null [OBJ] 4 \"pink and purple soccer cleat\" 0 60 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 107 923 1 119 916 2 134 904 3 150 892 4 166 884 5 185 880 6 201 876 7 220 873 8 236 869 9 255 865 10 271 861 11 287 857 12 306 853 13 322 849 14 333 845 15 345 841 16 357 837 17 369 833 18 380 830 19 388 826 20 396 822 21 408 818 22 416 814 23 423 810 [OBJ] 5 \"pink and purple wooden toy airplane\" 0 60 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 310 935 1 330 919 2 318 927 3 330 916 4 318 923 5 330 912 6 318 919 7 330 908 8 318 916 9 330 904 10 318 912 11 330 900 12 318 908 13 330 896 14 318 904 15 330 892 16 318 900 17 330 888 18 318 896 19 330 884 20 318 892 21 330 880 22 318 888 23 330 876 [OBJ] 6 \"light blue wooden toy airplane bed set\" 0 60 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 431 365 5 443 365 6 455 369 7 462 373 8 470 376 9 478 380 10 486 384 13 498 388 14 470 373 15 490 384 16 498 388 17 474 373 18 501 388 19 478 373 20 486 376 21 494 380 22 478 369 23 486 373 [OBJ] 7 \"light blue plastic conditioner tube\" 0 60 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 439 794 6 447 798 12 443 791 13 451 794 19 455 787 20 435 775 21 458 787 22 435 771 23 458 783 [OBJ] 8 \"brown and white cardboard latte mix box\" 0 60 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 564 1 1 null null [OBJS-END] [INTERACTIONS-START] [INTERACT] 1 \"contact\" 0 12 [IN] 0 3 [OUT] 0 3 [INTERACT] 0 \"contact\" 0 13 [IN] 0 2 [OUT] 0 2 [INTERACT] 2 \"contact\" 0 14 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 3 \"contact\" 0 23 [IN] 1 2 [OUT] 1 2 [INTERACT] 6 \"contact\" 0 23 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 7 \"contact\" 0 23 [IN] 3 65535 [OUT] 3 65535 [INTERACT] 8 \"contact\" 0 23 [IN] 4 65535 [OUT] 4 65535 [INTERACT] 9 \"contact\" 0 23 [IN] 5 65535 [OUT] 5 65535 [INTERACT] 10 \"contact\" 0 23 [IN] 6 65535 [OUT] 6 65535 [INTERACT] 11 \"contact\" 0 23 [IN] 7 65535 [OUT] 7 65535 [INTERACT] 12 \"contact\" 0 23 [IN] 8 65535 [OUT] 8 65535 [INTERACT] 4 \"collision\" 5 5 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 5 \"contact\" 6 23 [IN] 1 65535 [OUT] 1 65535 [INTERACTIONS-END] [SCENE-END]",
"label": "[SCENE-START] [OBJS-START] [OBJ] 0 \"purple cruiser\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 55 913 1 69 905 2 83 899 3 97 893 4 110 886 5 123 880 6 133 874 7 145 865 8 158 861 9 169 857 10 181 852 11 193 847 12 203 842 13 214 836 14 224 831 15 239 825 16 248 821 17 258 816 18 267 811 19 274 808 20 283 803 21 292 799 22 303 794 23 310 791 [OBJ] 1 \"purple airliner\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 319 913 [OBJ] 2 \"blue airliner\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 421 359 1 445 366 4 428 360 5 430 360 6 451 367 7 439 364 9 437 361 10 463 369 11 466 369 12 446 365 13 466 369 14 472 370 17 464 368 19 483 372 20 466 368 22 492 373 [OBJ] 3 \"blue truck\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 446 801 12 445 795 22 444 786 [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 23 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 2 \"contact\" 0 23 [IN] 3 65535 [OUT] 3 65535 [INTERACTIONS-END] [SCENE-END]\n"
}Parsed prediction
{
"objects": [
{
"id": 0,
"type": "pink and purple soccer cleat",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 1,
"y": 908
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 60
},
{
"id": 1,
"type": "pink and purple plastic package of overnight pads",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 5,
"y": 994
},
{
"t": 1,
"x": 1,
"y": 986
},
{
"t": 2,
"x": 5,
"y": 994
},
{
"t": 11,
"x": 1,
"y": 986
},
{
"t": 15,
"x": 1,
"y": 974
},
{
"t": 20,
"x": 1,
"y": 962
},
{
"t": 22,
"x": 5,
"y": 994
},
{
"t": 23,
"x": 1,
"y": 982
}
],
"birth": 0,
"death": 60
},
{
"id": 2,
"type": "pink and purple plastic package of overnight pads",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 9,
"y": 986
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 60
},
{
"id": 3,
"type": "pink and purple plastic package of overnight pads",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 25,
"y": 994
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 60
},
{
"id": 4,
"type": "pink and purple soccer cleat",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 107,
"y": 923
},
{
"t": 1,
"x": 119,
"y": 916
},
{
"t": 2,
"x": 134,
"y": 904
},
{
"t": 3,
"x": 150,
"y": 892
},
{
"t": 4,
"x": 166,
"y": 884
},
{
"t": 5,
"x": 185,
"y": 880
},
{
"t": 6,
"x": 201,
"y": 876
},
{
"t": 7,
"x": 220,
"y": 873
},
{
"t": 8,
"x": 236,
"y": 869
},
{
"t": 9,
"x": 255,
"y": 865
},
{
"t": 10,
"x": 271,
"y": 861
},
{
"t": 11,
"x": 287,
"y": 857
},
{
"t": 12,
"x": 306,
"y": 853
},
{
"t": 13,
"x": 322,
"y": 849
},
{
"t": 14,
"x": 333,
"y": 845
},
{
"t": 15,
"x": 345,
"y": 841
},
{
"t": 16,
"x": 357,
"y": 837
},
{
"t": 17,
"x": 369,
"y": 833
},
{
"t": 18,
"x": 380,
"y": 830
},
{
"t": 19,
"x": 388,
"y": 826
},
{
"t": 20,
"x": 396,
"y": 822
},
{
"t": 21,
"x": 408,
"y": 818
},
{
"t": 22,
"x": 416,
"y": 814
},
{
"t": 23,
"x": 423,
"y": 810
}
],
"birth": 0,
"death": 60
},
{
"id": 5,
"type": "pink and purple wooden toy airplane",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 310,
"y": 935
},
{
"t": 1,
"x": 330,
"y": 919
},
{
"t": 2,
"x": 318,
"y": 927
},
{
"t": 3,
"x": 330,
"y": 916
},
{
"t": 4,
"x": 318,
"y": 923
},
{
"t": 5,
"x": 330,
"y": 912
},
{
"t": 6,
"x": 318,
"y": 919
},
{
"t": 7,
"x": 330,
"y": 908
},
{
"t": 8,
"x": 318,
"y": 916
},
{
"t": 9,
"x": 330,
"y": 904
},
{
"t": 10,
"x": 318,
"y": 912
},
{
"t": 11,
"x": 330,
"y": 900
},
{
"t": 12,
"x": 318,
"y": 908
},
{
"t": 13,
"x": 330,
"y": 896
},
{
"t": 14,
"x": 318,
"y": 904
},
{
"t": 15,
"x": 330,
"y": 892
},
{
"t": 16,
"x": 318,
"y": 900
},
{
"t": 17,
"x": 330,
"y": 888
},
{
"t": 18,
"x": 318,
"y": 896
},
{
"t": 19,
"x": 330,
"y": 884
},
{
"t": 20,
"x": 318,
"y": 892
},
{
"t": 21,
"x": 330,
"y": 880
},
{
"t": 22,
"x": 318,
"y": 888
},
{
"t": 23,
"x": 330,
"y": 876
}
],
"birth": 0,
"death": 60
},
{
"id": 6,
"type": "light blue wooden toy airplane bed set",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 431,
"y": 365
},
{
"t": 5,
"x": 443,
"y": 365
},
{
"t": 6,
"x": 455,
"y": 369
},
{
"t": 7,
"x": 462,
"y": 373
},
{
"t": 8,
"x": 470,
"y": 376
},
{
"t": 9,
"x": 478,
"y": 380
},
{
"t": 10,
"x": 486,
"y": 384
},
{
"t": 13,
"x": 498,
"y": 388
},
{
"t": 14,
"x": 470,
"y": 373
},
{
"t": 15,
"x": 490,
"y": 384
},
{
"t": 16,
"x": 498,
"y": 388
},
{
"t": 17,
"x": 474,
"y": 373
},
{
"t": 18,
"x": 501,
"y": 388
},
{
"t": 19,
"x": 478,
"y": 373
},
{
"t": 20,
"x": 486,
"y": 376
},
{
"t": 21,
"x": 494,
"y": 380
},
{
"t": 22,
"x": 478,
"y": 369
},
{
"t": 23,
"x": 486,
"y": 373
}
],
"birth": 0,
"death": 60
},
{
"id": 7,
"type": "light blue plastic conditioner tube",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 439,
"y": 794
},
{
"t": 6,
"x": 447,
"y": 798
},
{
"t": 12,
"x": 443,
"y": 791
},
{
"t": 13,
"x": 451,
"y": 794
},
{
"t": 19,
"x": 455,
"y": 787
},
{
"t": 20,
"x": 435,
"y": 775
},
{
"t": 21,
"x": 458,
"y": 787
},
{
"t": 22,
"x": 435,
"y": 771
},
{
"t": 23,
"x": 458,
"y": 783
}
],
"birth": 0,
"death": 60
},
{
"id": 8,
"type": "brown and white cardboard latte mix box",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 564,
"y": 1
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 60
}
],
"interactions": [
{
"id": 1,
"inputs": [
0,
3
],
"outputs": [
0,
3
],
"begins_at": 0,
"ends_at": 12,
"type": "contact"
},
{
"id": 0,
"inputs": [
0,
2
],
"outputs": [
0,
2
],
"begins_at": 0,
"ends_at": 13,
"type": "contact"
},
{
"id": 2,
"inputs": [
0,
65535
],
"outputs": [
0,
65535
],
"begins_at": 0,
"ends_at": 14,
"type": "contact"
},
{
"id": 3,
"inputs": [
1,
2
],
"outputs": [
1,
2
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 6,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 7,
"inputs": [
3,
65535
],
"outputs": [
3,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 8,
"inputs": [
4,
65535
],
"outputs": [
4,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 9,
"inputs": [
5,
65535
],
"outputs": [
5,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 10,
"inputs": [
6,
65535
],
"outputs": [
6,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 11,
"inputs": [
7,
65535
],
"outputs": [
7,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 12,
"inputs": [
8,
65535
],
"outputs": [
8,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 4,
"inputs": [
1,
65535
],
"outputs": [
1,
65535
],
"begins_at": 5,
"ends_at": 5,
"type": "collision"
},
{
"id": 5,
"inputs": [
1,
65535
],
"outputs": [
1,
65535
],
"begins_at": 6,
"ends_at": 23,
"type": "contact"
}
]
}Parsed ground truth
{
"objects": [
{
"id": 0,
"type": "purple cruiser",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 55,
"y": 913
},
{
"t": 1,
"x": 69,
"y": 905
},
{
"t": 2,
"x": 83,
"y": 899
},
{
"t": 3,
"x": 97,
"y": 893
},
{
"t": 4,
"x": 110,
"y": 886
},
{
"t": 5,
"x": 123,
"y": 880
},
{
"t": 6,
"x": 133,
"y": 874
},
{
"t": 7,
"x": 145,
"y": 865
},
{
"t": 8,
"x": 158,
"y": 861
},
{
"t": 9,
"x": 169,
"y": 857
},
{
"t": 10,
"x": 181,
"y": 852
},
{
"t": 11,
"x": 193,
"y": 847
},
{
"t": 12,
"x": 203,
"y": 842
},
{
"t": 13,
"x": 214,
"y": 836
},
{
"t": 14,
"x": 224,
"y": 831
},
{
"t": 15,
"x": 239,
"y": 825
},
{
"t": 16,
"x": 248,
"y": 821
},
{
"t": 17,
"x": 258,
"y": 816
},
{
"t": 18,
"x": 267,
"y": 811
},
{
"t": 19,
"x": 274,
"y": 808
},
{
"t": 20,
"x": 283,
"y": 803
},
{
"t": 21,
"x": 292,
"y": 799
},
{
"t": 22,
"x": 303,
"y": 794
},
{
"t": 23,
"x": 310,
"y": 791
}
],
"birth": 0,
"death": 24
},
{
"id": 1,
"type": "purple airliner",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 319,
"y": 913
}
],
"birth": 0,
"death": 24
},
{
"id": 2,
"type": "blue airliner",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 421,
"y": 359
},
{
"t": 1,
"x": 445,
"y": 366
},
{
"t": 4,
"x": 428,
"y": 360
},
{
"t": 5,
"x": 430,
"y": 360
},
{
"t": 6,
"x": 451,
"y": 367
},
{
"t": 7,
"x": 439,
"y": 364
},
{
"t": 9,
"x": 437,
"y": 361
},
{
"t": 10,
"x": 463,
"y": 369
},
{
"t": 11,
"x": 466,
"y": 369
},
{
"t": 12,
"x": 446,
"y": 365
},
{
"t": 13,
"x": 466,
"y": 369
},
{
"t": 14,
"x": 472,
"y": 370
},
{
"t": 17,
"x": 464,
"y": 368
},
{
"t": 19,
"x": 483,
"y": 372
},
{
"t": 20,
"x": 466,
"y": 368
},
{
"t": 22,
"x": 492,
"y": 373
}
],
"birth": 0,
"death": 24
},
{
"id": 3,
"type": "blue truck",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 446,
"y": 801
},
{
"t": 12,
"x": 445,
"y": 795
},
{
"t": 22,
"x": 444,
"y": 786
}
],
"birth": 0,
"death": 24
}
],
"interactions": [
{
"id": 0,
"inputs": [
0,
65535
],
"outputs": [
0,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 1,
"inputs": [
1,
65535
],
"outputs": [
1,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 2,
"inputs": [
3,
65535
],
"outputs": [
3,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
}
]
}Complete per-sample evaluator report
{
"metrics": {
"avg_type_similarity": 0.48156556666991335,
"avg_size_diff": null,
"avg_3d_iou": null,
"avg_image_position_px_diff": 48.18261011965083,
"avg_image_position_segmentation_acc": 0.7458333333333333,
"avg_image_position_segmentation_acc_small": null,
"avg_image_position_segmentation_acc_medium": 0.75,
"avg_image_position_segmentation_acc_large": 0.7444444444444445,
"avg_image_position_segmentation_acc_first_frame": 1.0,
"avg_image_position_segmentation_visible_acc": 0.7458333333333333,
"avg_quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null,
"num_matches": 4,
"num_unmatched_pred": 5,
"num_unmatched_target": 0
},
"supports": {
"avg_type_similarity": 4,
"avg_size_diff": 0,
"avg_3d_iou": 0,
"avg_image_position_px_diff": 4,
"avg_image_position_segmentation_acc": 4,
"avg_image_position_segmentation_acc_small": 0,
"avg_image_position_segmentation_acc_medium": 1,
"avg_image_position_segmentation_acc_large": 3,
"avg_image_position_segmentation_acc_first_frame": 4,
"avg_image_position_segmentation_visible_acc": 4,
"avg_quaternion_geodesic_deg": 0,
"visibility_acc": 4,
"visibility_true_acc": 4,
"visibility_false_acc": 0,
"is_in_frame_acc": 4,
"is_in_frame_true_acc": 4,
"is_in_frame_false_acc": 0,
"vis_acc_from_px": 4,
"vis_acc_from_px_true_acc": 4,
"vis_acc_from_px_false_acc": 0
},
"matching": {
"matches": [
{
"target_index": 0,
"pred_index": 4,
"target_id": 0,
"pred_id": 4,
"cost": 1.7666666666666666,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.7666666666666667,
"weighted_cost": 0.7666666666666667,
"used_missing_value": false
}
]
},
{
"target_index": 1,
"pred_index": 5,
"target_id": 1,
"pred_id": 5,
"cost": 0.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 2,
"pred_index": 6,
"target_id": 2,
"pred_id": 6,
"cost": 0.75,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.5,
"weighted_cost": 0.5,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.25,
"weighted_cost": 0.25,
"used_missing_value": false
}
]
},
{
"target_index": 3,
"pred_index": 7,
"target_id": 3,
"pred_id": 7,
"cost": 1.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
}
],
"unmatched_target_ids": [],
"unmatched_pred_ids": [
0,
1,
2,
3,
8
],
"target_type_counts": {
"purple cruiser": 1,
"purple airliner": 1,
"blue airliner": 1,
"blue truck": 1
},
"pred_type_counts": {
"pink and purple soccer cleat": 2,
"pink and purple plastic package of overnight pads": 3,
"pink and purple wooden toy airplane": 1,
"light blue wooden toy airplane bed set": 1,
"light blue plastic conditioner tube": 1,
"brown and white cardboard latte mix box": 1
},
"assignment_backend": "scipy_dense",
"cost_functions": [
"mutual_type",
"same_seg"
],
"cost_weights": {
"mutual_type": 1.0,
"same_seg": 1.0
},
"num_frames": 60
},
"scene_metrics": {
"thresholds": [
0.5,
1.0,
2.0,
3.0
],
"counts_by_threshold": {
"0.5": {
"true_positives": 3,
"false_positives": 1,
"false_negatives": 0
},
"1.0": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 0
},
"2.0": {
"true_positives": 0,
"false_positives": 4,
"false_negatives": 0
},
"3.0": {
"true_positives": 0,
"false_positives": 4,
"false_negatives": 0
}
},
"jaccard_by_threshold": {
"0.5": 0.75,
"1.0": 0.5,
"2.0": 0.0,
"3.0": 0.0
},
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 5,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 5,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 5,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 5,
"false_negatives": 0
}
},
"visibility_precision_by_threshold": {
"0.25": 0.4444444444444444,
"0.5": 0.4444444444444444,
"0.75": 0.4444444444444444,
"1.0": 0.4444444444444444
},
"visibility_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"visibility_f1_by_threshold": {
"0.25": 0.6153846153846153,
"0.5": 0.6153846153846153,
"0.75": 0.6153846153846153,
"1.0": 0.6153846153846153
},
"visibility_by_motion_group": {
"static": {
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 0,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 0,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 0,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 0,
"false_negatives": 0
}
},
"visibility_precision_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"visibility_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"visibility_f1_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
}
},
"unknown": {
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 1,
"false_positives": 5,
"false_negatives": 0
},
"0.5": {
"true_positives": 1,
"false_positives": 5,
"false_negatives": 0
},
"0.75": {
"true_positives": 1,
"false_positives": 5,
"false_negatives": 0
},
"1.0": {
"true_positives": 1,
"false_positives": 5,
"false_negatives": 0
}
},
"visibility_precision_by_threshold": {
"0.25": 0.16666666666666666,
"0.5": 0.16666666666666666,
"0.75": 0.16666666666666666,
"1.0": 0.16666666666666666
},
"visibility_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"visibility_f1_by_threshold": {
"0.25": 0.2857142857142857,
"0.5": 0.2857142857142857,
"0.75": 0.2857142857142857,
"1.0": 0.2857142857142857
}
}
},
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 6,
"false_negatives": 1
},
"0.5": {
"true_positives": 3,
"false_positives": 6,
"false_negatives": 1
},
"0.75": {
"true_positives": 3,
"false_positives": 6,
"false_negatives": 1
},
"1.0": {
"true_positives": 2,
"false_positives": 7,
"false_negatives": 2
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.3333333333333333,
"0.5": 0.3333333333333333,
"0.75": 0.3333333333333333,
"1.0": 0.2222222222222222
},
"segmentation_recall_by_threshold": {
"0.25": 0.75,
"0.5": 0.75,
"0.75": 0.75,
"1.0": 0.5
},
"segmentation_f1_by_threshold": {
"0.25": 0.46153846153846156,
"0.5": 0.46153846153846156,
"0.75": 0.46153846153846156,
"1.0": 0.30769230769230765
},
"segmentation_frame_average_precision": 0.3055555555555556,
"segmentation_frame_average_recall": 0.6875,
"segmentation_frame_average_f1": 0.4230769230769231,
"segmentation_first_frame_counts": {
"true_positives": 4,
"false_positives": 5,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.4444444444444444,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 0.6153846153846153,
"segmentation_by_motion_group": {
"static": {
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 2,
"false_positives": 1,
"false_negatives": 1
},
"0.5": {
"true_positives": 2,
"false_positives": 1,
"false_negatives": 1
},
"0.75": {
"true_positives": 2,
"false_positives": 1,
"false_negatives": 1
},
"1.0": {
"true_positives": 2,
"false_positives": 1,
"false_negatives": 1
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.6666666666666666,
"0.5": 0.6666666666666666,
"0.75": 0.6666666666666666,
"1.0": 0.6666666666666666
},
"segmentation_recall_by_threshold": {
"0.25": 0.6666666666666666,
"0.5": 0.6666666666666666,
"0.75": 0.6666666666666666,
"1.0": 0.6666666666666666
},
"segmentation_f1_by_threshold": {
"0.25": 0.6666666666666666,
"0.5": 0.6666666666666666,
"0.75": 0.6666666666666666,
"1.0": 0.6666666666666666
},
"segmentation_frame_average_precision": 0.6666666666666666,
"segmentation_frame_average_recall": 0.6666666666666666,
"segmentation_frame_average_f1": 0.6666666666666666,
"segmentation_first_frame_counts": {
"true_positives": 3,
"false_positives": 0,
"false_negatives": 0
},
"segmentation_first_frame_precision": 1.0,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 1.0
},
"unknown": {
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 1,
"false_positives": 5,
"false_negatives": 0
},
"0.5": {
"true_positives": 1,
"false_positives": 5,
"false_negatives": 0
},
"0.75": {
"true_positives": 1,
"false_positives": 5,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 6,
"false_negatives": 1
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.16666666666666666,
"0.5": 0.16666666666666666,
"0.75": 0.16666666666666666,
"1.0": 0.0
},
"segmentation_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 0.0
},
"segmentation_f1_by_threshold": {
"0.25": 0.2857142857142857,
"0.5": 0.2857142857142857,
"0.75": 0.2857142857142857,
"1.0": 0.0
},
"segmentation_frame_average_precision": 0.125,
"segmentation_frame_average_recall": 0.75,
"segmentation_frame_average_f1": 0.21428571428571427,
"segmentation_first_frame_counts": {
"true_positives": 1,
"false_positives": 5,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.16666666666666666,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 0.2857142857142857
}
}
},
"per_match_metrics": [
{
"target_id": 0,
"pred_id": 4,
"cost": 1.7666666666666666,
"type_similarity": 0.42312697080889494,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 133.3251816568596,
"image_position_segmentation_acc": 0.23333333333333334,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": 0.23333333333333334,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 0.23333333333333334,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 1,
"pred_id": 5,
"cost": 0.0,
"type_similarity": 0.6492408606523746,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 29.840547190155927,
"image_position_segmentation_acc": 1.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": 1.0,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 1.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 2,
"pred_id": 6,
"cost": 0.75,
"type_similarity": 0.5375342888944351,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 14.85192246175735,
"image_position_segmentation_acc": 0.75,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 0.75,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 0.75,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 3,
"pred_id": 7,
"cost": 1.0,
"type_similarity": 0.31636014632394904,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 14.712789169830465,
"image_position_segmentation_acc": 1.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": 1.0,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 1.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
}
],
"interaction_metrics": {
"avg_interaction_temporal_span_giou": 1.0,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 3,
"num_unmatched_pred_interactions": 10,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.23076923076923078,
"0.5": 0.23076923076923078,
"0.75": 0.23076923076923078,
"1.0": 0.23076923076923078
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.375,
"0.5": 0.375,
"0.75": 0.375,
"1.0": 0.375
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.23076923076923078,
"0.5": 0.23076923076923078,
"0.75": 0.23076923076923078,
"1.0": 0.23076923076923078
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.375,
"0.5": 0.375,
"0.75": 0.375,
"1.0": 0.375
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 10,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.23076923076923078,
"0.5": 0.23076923076923078,
"0.75": 0.23076923076923078,
"1.0": 0.23076923076923078
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.375,
"0.5": 0.375,
"0.75": 0.375,
"1.0": 0.375
}
},
"interaction_supports": {
"avg_interaction_temporal_span_giou": 3,
"avg_interaction_input_object_jaccard": 3,
"avg_interaction_output_object_jaccard": 3,
"avg_interaction_type_exact_match_acc": 3
},
"interaction_matching": {
"matches": [
{
"target_index": 0,
"pred_index": 8,
"target_id": 0,
"pred_id": 8,
"cost": 0.0,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 1,
"pred_index": 9,
"target_id": 1,
"pred_id": 9,
"cost": 0.0,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 2,
"pred_index": 11,
"target_id": 2,
"pred_id": 11,
"cost": 0.0,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
}
],
"unmatched_target_ids": [],
"unmatched_pred_ids": [
0,
1,
2,
3,
4,
5,
6,
7,
10,
12
],
"target_type_counts": {
"contact": 3
},
"pred_type_counts": {
"contact": 12,
"collision": 1
},
"assignment_backend": "scipy_dense",
"cost_functions": [
"type_exact",
"temporal_span",
"input_object_set_mutual_jaccard",
"output_object_set_mutual_jaccard"
],
"cost_weights": {
"type_exact": 1.0,
"temporal_span": 1.0,
"input_object_set_mutual_jaccard": 0.5,
"output_object_set_mutual_jaccard": 0.5
}
},
"per_interaction_match_metrics": [
{
"target_index": 0,
"pred_index": 8,
"target_id": 0,
"pred_id": 8,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.0,
"temporal_span_iou": 1.0,
"temporal_span_giou": 1.0,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 1.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 1,
"pred_index": 9,
"target_id": 1,
"pred_id": 9,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.0,
"temporal_span_iou": 1.0,
"temporal_span_giou": 1.0,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 1.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 2,
"pred_index": 11,
"target_id": 2,
"pred_id": 11,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.0,
"temporal_span_iou": 1.0,
"temporal_span_giou": 1.0,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 1.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
}
],
"interaction_metrics_by_participants": {
"object_ground": {
"avg_interaction_temporal_span_giou": 1.0,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 3,
"num_unmatched_pred_interactions": 7,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.3,
"0.5": 0.3,
"0.75": 0.3,
"1.0": 0.3
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.4615384615384615,
"0.5": 0.4615384615384615,
"0.75": 0.4615384615384615,
"1.0": 0.4615384615384615
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.3,
"0.5": 0.3,
"0.75": 0.3,
"1.0": 0.3
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.4615384615384615,
"0.5": 0.4615384615384615,
"0.75": 0.4615384615384615,
"1.0": 0.4615384615384615
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 7,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.3,
"0.5": 0.3,
"0.75": 0.3,
"1.0": 0.3
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.4615384615384615,
"0.5": 0.4615384615384615,
"0.75": 0.4615384615384615,
"1.0": 0.4615384615384615
}
},
"object_object": {
"avg_interaction_temporal_span_giou": null,
"avg_interaction_input_object_jaccard": null,
"avg_interaction_output_object_jaccard": null,
"avg_interaction_type_exact_match_acc": null,
"num_interaction_matches": 0,
"num_unmatched_pred_interactions": 3,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 3,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
}
},
"interaction_supports_by_participants": {
"object_ground": {
"avg_interaction_temporal_span_giou": 3,
"avg_interaction_input_object_jaccard": 3,
"avg_interaction_output_object_jaccard": 3,
"avg_interaction_type_exact_match_acc": 3
},
"object_object": {
"avg_interaction_temporal_span_giou": 0,
"avg_interaction_input_object_jaccard": 0,
"avg_interaction_output_object_jaccard": 0,
"avg_interaction_type_exact_match_acc": 0
}
},
"interaction_metrics_by_type": {
"collision": {
"avg_interaction_temporal_span_giou": null,
"avg_interaction_input_object_jaccard": null,
"avg_interaction_output_object_jaccard": null,
"avg_interaction_type_exact_match_acc": null,
"num_interaction_matches": 0,
"num_unmatched_pred_interactions": 1,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 1,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
},
"contact": {
"avg_interaction_temporal_span_giou": 1.0,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 3,
"num_unmatched_pred_interactions": 9,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.25,
"0.5": 0.25,
"0.75": 0.25,
"1.0": 0.25
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.4,
"0.5": 0.4,
"0.75": 0.4,
"1.0": 0.4
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.25,
"0.5": 0.25,
"0.75": 0.25,
"1.0": 0.25
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.4,
"0.5": 0.4,
"0.75": 0.4,
"1.0": 0.4
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"0.5": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"0.75": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 9,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.25,
"0.5": 0.25,
"0.75": 0.25,
"1.0": 0.25
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.4,
"0.5": 0.4,
"0.75": 0.4,
"1.0": 0.4
}
}
},
"interaction_supports_by_type": {
"collision": {
"avg_interaction_temporal_span_giou": 0,
"avg_interaction_input_object_jaccard": 0,
"avg_interaction_output_object_jaccard": 0,
"avg_interaction_type_exact_match_acc": 0
},
"contact": {
"avg_interaction_temporal_span_giou": 3,
"avg_interaction_input_object_jaccard": 3,
"avg_interaction_output_object_jaccard": 3,
"avg_interaction_type_exact_match_acc": 3
}
}
}Exact evaluation configuration
{
"predictions_jsonl": "../inference_output.jsonl",
"dataset_json": "../test_data/movi_dump_sentinel_sharegpt.json",
"scene_root": "../test_data/movi_dump",
"output_json": "../evaluation_report.json",
"max_samples": null,
"num_workers": 1,
"show_progress": true,
"log_level": "INFO",
"visibility_filter_mode": "any_frame",
"pos2d_mode": "scaled_normalized",
"pos2d_scale": 1000.0,
"image_width": null,
"image_height": null,
"type_embed_model_name": "all-MiniLM-L6-v2",
"scene_similarity_thresholds": [
0.5,
1.0,
2.0,
3.0
],
"scene_segmentation_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"matching": {
"costs": [
{
"name": "mutual_type",
"weight": 1.0
},
{
"name": "same_seg",
"weight": 1.0,
"missing_value": 1.0,
"kwargs": {
"only_use_first_frame": false
}
}
]
},
"interaction_matching": {
"costs": [
{
"name": "type_exact",
"weight": 1.0
},
{
"name": "temporal_span",
"weight": 1.0
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5
}
]
},
"return_per_sample": true
}