STRIVE · 08/28/2026
Dyn generalization · val_24 · super_clevr_1077
Everything fed into evaluation and the full per-sample output it produced.
Dataset row: exact video + user prompt + assistant target
{
"messages": [
{
"role": "user",
"content": "<video>\nAnalyze the provided video and return ONLY a valid sentinel sequence in this format:\n[SCENE-START] [OBJS-START]\n[OBJ] id \"type\" birth death [VIS] t vis ... [IN-FRAME] t is_in_frame ... [PX] t px py ...\n[OBJ] ...\n[OBJS-END]\n[INTERACTIONS-START]\n[INTERACT] id \"type\" begins_at ends_at [IN] in0 in1 ... [OUT] out0 out1 ...\n[INTERACT] ...\n[INTERACTIONS-END] [SCENE-END]\n\nConstraints:\n- Return only the sentinel text (no JSON, XML, markdown, comments, or extra prose).\n- Keep tags exactly as written:\n `[SCENE-START] [OBJS-START] [OBJ] [VIS] [IN-FRAME] [PX] [OBJS-END] [INTERACTIONS-START] [INTERACT] [IN] [OUT] [INTERACTIONS-END] [SCENE-END]`\n- Use quoted strings for `type` (double quotes).\n- `VIS` uses repeated groups of 2 values in order:\n `t vis`.\n `vis` is a boolean value.\n- `IN-FRAME` uses repeated groups of 2 values in order:\n `t is_in_frame`.\n `is_in_frame` is a boolean value.\n- `PX` uses repeated groups of 3 values in order: `t px py`.\n- `px` and `py` are normalized pixel coordinates in the range `[0, 1]`,\n scaled by `1000` and then converted to integers by dropping the decimal part.\n- For occluded or out-of-frame frames under this setting, `PX` may use `t null null`.\n- `IN` and `OUT` contain whitespace-separated integer ids.\n- Use lowercase booleans (`true`/`false`) for `vis` and `is_in_frame`."
},
{
"role": "assistant",
"content": "[SCENE-START] [OBJS-START] [OBJ] 0 \"brown tandem\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 336 642 3 337 644 5 342 641 6 341 646 7 333 613 8 344 641 10 333 624 12 335 626 13 348 648 14 347 657 16 348 658 17 341 623 18 340 632 19 351 659 20 350 663 21 344 629 23 347 629 [OBJ] 1 \"green mountain\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 799 786 21 783 767 [OBJ] 2 \"blue minivan\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 873 927 1 864 926 2 856 925 3 848 924 4 851 922 5 839 921 6 832 920 7 825 919 8 819 918 9 812 917 10 806 916 11 800 915 12 798 914 13 791 914 15 786 910 17 784 909 18 780 907 19 776 906 20 774 905 21 787 910 22 780 907 23 774 905 [OBJ] 3 \"cyan chopper\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 948 847 1 940 845 2 933 844 3 924 842 4 916 839 5 909 836 6 903 835 7 897 835 8 890 833 9 883 831 10 876 830 11 869 827 12 862 826 13 856 825 14 849 823 15 844 822 16 837 820 17 830 819 18 824 817 19 818 816 20 813 815 21 806 813 22 801 811 23 795 809 [OBJ] 4 \"blue jet\" 2 24 [VIS] 0 false 2 true [IN-FRAME] 0 false 2 true [PX] 0 null null 2 0 331 3 5 332 4 5 341 5 8 340 6 12 338 7 14 338 11 56 339 12 66 339 13 16 339 14 26 339 15 37 339 16 48 339 17 15 333 19 16 329 20 18 324 21 21 319 22 29 316 23 40 316 [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 23 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 2 \"contact\" 0 23 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 3 \"contact\" 0 23 [IN] 3 65535 [OUT] 3 65535 [INTERACTIONS-END] [SCENE-END]"
}
],
"videos": [
"/mmfs1/gscratch/krishna/suryadv/strive_dynsuperclevr/data/lf/videos/val_24/super_clevr_1077/frames.max_frames_24.mp4"
]
}Inference record: exact prompt + prediction + label + parse debug
{
"prompt": "user\n\nAnalyze the provided video and return ONLY a valid sentinel sequence in this format:\n[SCENE-START] [OBJS-START]\n[OBJ] id \"type\" birth death [VIS] t vis ... [IN-FRAME] t is_in_frame ... [PX] t px py ...\n[OBJ] ...\n[OBJS-END]\n[INTERACTIONS-START]\n[INTERACT] id \"type\" begins_at ends_at [IN] in0 in1 ... [OUT] out0 out1 ...\n[INTERACT] ...\n[INTERACTIONS-END] [SCENE-END]\n\nConstraints:\n- Return only the sentinel text (no JSON, XML, markdown, comments, or extra prose).\n- Keep tags exactly as written:\n `[SCENE-START] [OBJS-START] [OBJ] [VIS] [IN-FRAME] [PX] [OBJS-END] [INTERACTIONS-START] [INTERACT] [IN] [OUT] [INTERACTIONS-END] [SCENE-END]`\n- Use quoted strings for `type` (double quotes).\n- `VIS` uses repeated groups of 2 values in order:\n `t vis`.\n `vis` is a boolean value.\n- `IN-FRAME` uses repeated groups of 2 values in order:\n `t is_in_frame`.\n `is_in_frame` is a boolean value.\n- `PX` uses repeated groups of 3 values in order: `t px py`.\n- `px` and `py` are normalized pixel coordinates in the range `[0, 1]`,\n scaled by `1000` and then converted to integers by dropping the decimal part.\n- For occluded or out-of-frame frames under this setting, `PX` may use `t null null`.\n- `IN` and `OUT` contain whitespace-separated integer ids.\n- Use lowercase booleans (`true`/`false`) for `vis` and `is_in_frame`.\nassistant\n",
"predict": "[SCENE-START] [OBJS-START] [OBJ] 0 \"dark brown leather wingtip shoe with orange sole\" 0 10 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 337 638 2 341 646 6 345 654 8 349 662 11 353 669 12 353 658 13 357 673 16 357 658 17 361 677 19 361 666 20 365 681 21 365 669 22 369 685 23 369 673 [OBJ] 1 \"green wooden dollhouse bedroom furniture set\" 0 10 [VIS] 0 false 3 true [IN-FRAME] 0 true [PX] 0 null null 3 458 513 4 451 509 7 455 517 10 447 521 12 458 525 13 451 521 14 458 525 15 451 521 16 458 525 21 451 529 [OBJ] 2 \"dark grey and light grey plastic monster action figure\" 0 10 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 966 865 1 974 869 2 966 873 3 958 880 4 951 884 6 943 888 8 931 892 9 919 896 11 904 900 13 888 904 14 876 904 15 865 908 17 849 908 18 837 912 19 826 912 20 818 916 21 806 916 22 794 919 23 783 919 [OBJ] 3 \"blue and white athletic shoe\" 0 10 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 994 837 1 982 837 2 970 841 3 958 841 4 947 841 6 931 845 8 912 845 9 900 845 10 892 841 11 884 845 12 876 841 13 865 841 15 853 841 16 845 837 17 833 837 18 826 833 19 818 830 20 810 826 21 802 822 22 794 818 23 787 814 [OBJ] 4 \"blue wooden toy kitchen set\" 0 10 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 998 880 1 990 884 2 986 892 3 978 896 5 966 900 7 951 904 8 943 908 9 935 912 10 923 912 11 916 916 12 904 916 13 896 919 14 884 919 15 876 916 16 865 916 17 857 912 18 845 912 19 837 908 20 826 908 21 814 908 22 806 904 23 794 904 [OBJ] 5 \"blue wooden toy train\" 0 10 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 998 900 1 994 912 2 994 892 3 990 904 4 986 912 5 986 892 6 982 904 7 978 912 8 974 919 9 974 900 10 970 912 11 966 919 12 962 927 13 962 908 14 958 916 15 955 923 16 955 904 17 951 912 18 947 919 19 943 927 20 943 908 21 939 916 22 935 923 23 931 931 [OBJ] 6 \"blue wooden toy train\" 0 10 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 998 998 1 null null [OBJ] 7 \"red plastic hair straightener\" 0 10 [VIS] 0 false 1 true [IN-FRAME] 0 true [PX] 0 null null 1 998 341 2 982 361 3 966 380 4 947 404 5 931 423 6 916 443 7 896 462 8 880 474 9 869 478 10 857 482 11 845 486 12 833 490 13 822 494 14 810 498 15 798 501 16 787 505 17 775 509 18 763 513 19 751 517 20 740 521 21 728 525 22 716 529 23 705 533 [OBJ] 8 \"blue and white high-top sneaker\" 0 10 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 998 857 1 990 861 2 982 865 3 974 869 4 966 873 5 958 876 6 947 880 7 939 884 8 927 884 9 919 888 10 908 888 11 896 888 12 888 892 13 876 892 14 865 892 15 853 892 16 845 888 17 833 888 18 822 888 19 814 884 20 802 884 21 794 880 22 783 880 23 775 876 [OBJ] 9 \"blue and white striped fabric dishtowel\" 0 10 [VIS] 0 false 1 true [IN-FRAME] 0 true [PX] 0 null null 1 998 556 2 998 541 3 994 548 4 994 537 5 990 544 6 990 533 7 986 541 8 986 529 9 982 537 10 982 525 11 978 533 12 978 521 13 974 529 14 974 517 15 970 525 16 970 513 17 966 521 18 966 509 19 962 517 20 962 505 21 958 513 22 958 498 23 955 505 [OBJ] 10 \"blue portable hard drive\" 0 10 [VIS] 0 false 1 true [IN-FRAME] 0 true [PX] 0 null null 1 998 654 2 998 642 3 994 650 4 994 638 5 990 646 6 990 634 7 986 642 8 986 630 9 982 638 10 982 626 11 978 634 12 978 623 13 974 630 14 974 619 15 970 626 16 970 615 17 966 623 18 966 611 19 962 619 20 962 607 21 958 615 22 958 603 23 955 611 [OBJ] 11 \"blue and white high-top sneaker\" 0 10 [VIS] 0 false 1 true [IN-FRAME] 0 true [PX] 0 null null 1 998 603 4 990 607 7 986 599 10 978 595 12 974 587 14 970 580 16 962 576 18 958 568 20 951 564 22 947 556 23 943 548 [OBJ] 12 \"red plastic New Super Mario Bros. Wii game case\" 0 10 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 998 884 1 994 892 2 994 880 4 990 888 5 986 896 6 986 884 7 982 892 8 978 900 9 978 888 10 974 896 11 970 904 12 970 892 13 966 900 14 962 908 15 958 916 16 958 904 17 955 912 18 951 919 19 947 927 20 943 935 21 943 923 22 939 931 23 935 939 [OBJ] 13 \"blue and white high-top sneaker\" 0 10 [VIS] 0 false 1 true [IN-FRAME] 0 true [PX] 0 null null 1 998 849 3 994 833 5 986 837 7 978 841 9 970 845 11 958 845 13 943 841 14 935 837 16 923 833 18 908 830 19 900 826 21 884 822 22 876 818 23 869 814 [OBJ] 14 \"blue and white high-top sneaker\" 0 10 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 998 978 1 994 986 3 986 990 5 978 994 7 966 994 9 955 994 10 947 990 11 935 990 12 927 986 14 912 986 15 904 982 17 892 982 18 884 978 19 876 974 21 861 974 22 853 970 23 845 966 [OBJ] 15 \"blue and red running shoe\" 0 10 [VIS] 0 true 1 false [IN-FRAME] 0 true 1 false [PX] 0 998 994 1 null null [OBJ] 16 \"red plastic New Super Mario Bros. Wii game case\" 1 10 [VIS] 0 false 1 true [IN-FRAME] 0 false 1 true [PX] 0 null null 1 1 349 2 9 361 3 17 369 4 21 357 5 29 365 6 33 353 7 37 345 8 44 349 9 52 353 10 60 357 11 68 361 12 80 361 13 91 365 14 103 369 15 115 369 16 123 373 17 134 373 18 146 373 19 158 373 20 169 373 21 181 373 22 189 376 23 201 376 [OBJS-END] [INTERACTIONS-START] [INTERACT] 18 \"contact\" 0 4 [IN] 3 5 [OUT] 3 5 [INTERACT] 0 \"contact\" 0 10 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 3 \"contact\" 0 10 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 19 \"contact\" 0 10 [IN] 3 65535 [OUT] 3 65535 [INTERACT] 20 \"contact\" 0 10 [IN] 4 12 [OUT] 4 12 [INTERACT] 21 \"contact\" 0 10 [IN] 4 65535 [OUT] 4 65535 [INTERACT] 24 \"contact\" 0 10 [IN] 5 65535 [OUT] 5 65535 [INTERACT] 25 \"contact\" 0 10 [IN] 6 65535 [OUT] 6 65535 [INTERACT] 26 \"contact\" 0 10 [IN] 8 65535 [OUT] 8 65535 [INTERACT] 27 \"contact\" 0 10 [IN] 9 65535 [OUT] 9 65535 [INTERACT] 4 \"contact\" 0 10 [IN] 10 65535 [OUT] 10 65535 [INTERACT] 5 \"contact\" 0 10 [IN] 11 65535 [OUT] 11 65535 [INTERACT] 6 \"contact\" 0 10 [IN] 12 65535 [OUT] 12 65535 [INTERACT] 7 \"contact\" 0 10 [IN] 13 65535 [OUT] 13 65535 [INTERACT] 8 \"contact\" 0 10 [IN] 14 65535 [OUT] 14 65535 [INTERACT] 9 \"contact\" 0 10 [IN] 15 65535 [OUT] 15 65535 [INTERACT] 10 \"contact\" 0 10 [IN] 16 65535 [OUT] 16 65535 [INTERACT] 17 \"collision\" 4 4 [IN] 2 8 [OUT] 2 8 [INTERACT] 1 \"collision\" 6 6 [IN] 1 16 [OUT] 1 16 [INTERACT] 11 \"collision\" 6 6 [IN] 2 3 [OUT] 2 3 [INTERACT] 12 \"collision\" 6 6 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 2 \"contact\" 7 10 [IN] 1 16 [OUT] 1 16 [INTERACT] 13 \"contact\" 7 10 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 22 \"collision\" 8 8 [IN] 4 8 [OUT] 4 8 [INTERACT] 23 \"contact\" 9 10 [IN] 4 8 [OUT] 4 8 [INTERACT] 14 \"collision\" 10 10 [IN] 2 8 [OUT] 2 8 [INTERACT] 15 \"collision\" 10 10 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 28 \"collision\" 10 10 [IN] 9 16 [OUT] 9 16 [INTERACT] 16 \"contact\" 11 12 [IN] 2 8 [OUT] 2 8 [INTERACT] 29 \"collision\" 12 12 [IN] 9 16 [OUT] 9 16 [INTERACTIONS-END] [SCENE-END]",
"label": "[SCENE-START] [OBJS-START] [OBJ] 0 \"brown tandem\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 336 642 3 337 644 5 342 641 6 341 646 7 333 613 8 344 641 10 333 624 12 335 626 13 348 648 14 347 657 16 348 658 17 341 623 18 340 632 19 351 659 20 350 663 21 344 629 23 347 629 [OBJ] 1 \"green mountain\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 799 786 21 783 767 [OBJ] 2 \"blue minivan\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 873 927 1 864 926 2 856 925 3 848 924 4 851 922 5 839 921 6 832 920 7 825 919 8 819 918 9 812 917 10 806 916 11 800 915 12 798 914 13 791 914 15 786 910 17 784 909 18 780 907 19 776 906 20 774 905 21 787 910 22 780 907 23 774 905 [OBJ] 3 \"cyan chopper\" 0 24 [VIS] 0 true [IN-FRAME] 0 true [PX] 0 948 847 1 940 845 2 933 844 3 924 842 4 916 839 5 909 836 6 903 835 7 897 835 8 890 833 9 883 831 10 876 830 11 869 827 12 862 826 13 856 825 14 849 823 15 844 822 16 837 820 17 830 819 18 824 817 19 818 816 20 813 815 21 806 813 22 801 811 23 795 809 [OBJ] 4 \"blue jet\" 2 24 [VIS] 0 false 2 true [IN-FRAME] 0 false 2 true [PX] 0 null null 2 0 331 3 5 332 4 5 341 5 8 340 6 12 338 7 14 338 11 56 339 12 66 339 13 16 339 14 26 339 15 37 339 16 48 339 17 15 333 19 16 329 20 18 324 21 21 319 22 29 316 23 40 316 [OBJS-END] [INTERACTIONS-START] [INTERACT] 0 \"contact\" 0 23 [IN] 0 65535 [OUT] 0 65535 [INTERACT] 1 \"contact\" 0 23 [IN] 1 65535 [OUT] 1 65535 [INTERACT] 2 \"contact\" 0 23 [IN] 2 65535 [OUT] 2 65535 [INTERACT] 3 \"contact\" 0 23 [IN] 3 65535 [OUT] 3 65535 [INTERACTIONS-END] [SCENE-END]\n"
}Parsed prediction
{
"objects": [
{
"id": 0,
"type": "dark brown leather wingtip shoe with orange sole",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 337,
"y": 638
},
{
"t": 2,
"x": 341,
"y": 646
},
{
"t": 6,
"x": 345,
"y": 654
},
{
"t": 8,
"x": 349,
"y": 662
},
{
"t": 11,
"x": 353,
"y": 669
},
{
"t": 12,
"x": 353,
"y": 658
},
{
"t": 13,
"x": 357,
"y": 673
},
{
"t": 16,
"x": 357,
"y": 658
},
{
"t": 17,
"x": 361,
"y": 677
},
{
"t": 19,
"x": 361,
"y": 666
},
{
"t": 20,
"x": 365,
"y": 681
},
{
"t": 21,
"x": 365,
"y": 669
},
{
"t": 22,
"x": 369,
"y": 685
},
{
"t": 23,
"x": 369,
"y": 673
}
],
"birth": 0,
"death": 10
},
{
"id": 1,
"type": "green wooden dollhouse bedroom furniture set",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 3,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 3,
"x": 458,
"y": 513
},
{
"t": 4,
"x": 451,
"y": 509
},
{
"t": 7,
"x": 455,
"y": 517
},
{
"t": 10,
"x": 447,
"y": 521
},
{
"t": 12,
"x": 458,
"y": 525
},
{
"t": 13,
"x": 451,
"y": 521
},
{
"t": 14,
"x": 458,
"y": 525
},
{
"t": 15,
"x": 451,
"y": 521
},
{
"t": 16,
"x": 458,
"y": 525
},
{
"t": 21,
"x": 451,
"y": 529
}
],
"birth": 0,
"death": 10
},
{
"id": 2,
"type": "dark grey and light grey plastic monster action figure",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 966,
"y": 865
},
{
"t": 1,
"x": 974,
"y": 869
},
{
"t": 2,
"x": 966,
"y": 873
},
{
"t": 3,
"x": 958,
"y": 880
},
{
"t": 4,
"x": 951,
"y": 884
},
{
"t": 6,
"x": 943,
"y": 888
},
{
"t": 8,
"x": 931,
"y": 892
},
{
"t": 9,
"x": 919,
"y": 896
},
{
"t": 11,
"x": 904,
"y": 900
},
{
"t": 13,
"x": 888,
"y": 904
},
{
"t": 14,
"x": 876,
"y": 904
},
{
"t": 15,
"x": 865,
"y": 908
},
{
"t": 17,
"x": 849,
"y": 908
},
{
"t": 18,
"x": 837,
"y": 912
},
{
"t": 19,
"x": 826,
"y": 912
},
{
"t": 20,
"x": 818,
"y": 916
},
{
"t": 21,
"x": 806,
"y": 916
},
{
"t": 22,
"x": 794,
"y": 919
},
{
"t": 23,
"x": 783,
"y": 919
}
],
"birth": 0,
"death": 10
},
{
"id": 3,
"type": "blue and white athletic shoe",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 994,
"y": 837
},
{
"t": 1,
"x": 982,
"y": 837
},
{
"t": 2,
"x": 970,
"y": 841
},
{
"t": 3,
"x": 958,
"y": 841
},
{
"t": 4,
"x": 947,
"y": 841
},
{
"t": 6,
"x": 931,
"y": 845
},
{
"t": 8,
"x": 912,
"y": 845
},
{
"t": 9,
"x": 900,
"y": 845
},
{
"t": 10,
"x": 892,
"y": 841
},
{
"t": 11,
"x": 884,
"y": 845
},
{
"t": 12,
"x": 876,
"y": 841
},
{
"t": 13,
"x": 865,
"y": 841
},
{
"t": 15,
"x": 853,
"y": 841
},
{
"t": 16,
"x": 845,
"y": 837
},
{
"t": 17,
"x": 833,
"y": 837
},
{
"t": 18,
"x": 826,
"y": 833
},
{
"t": 19,
"x": 818,
"y": 830
},
{
"t": 20,
"x": 810,
"y": 826
},
{
"t": 21,
"x": 802,
"y": 822
},
{
"t": 22,
"x": 794,
"y": 818
},
{
"t": 23,
"x": 787,
"y": 814
}
],
"birth": 0,
"death": 10
},
{
"id": 4,
"type": "blue wooden toy kitchen set",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 880
},
{
"t": 1,
"x": 990,
"y": 884
},
{
"t": 2,
"x": 986,
"y": 892
},
{
"t": 3,
"x": 978,
"y": 896
},
{
"t": 5,
"x": 966,
"y": 900
},
{
"t": 7,
"x": 951,
"y": 904
},
{
"t": 8,
"x": 943,
"y": 908
},
{
"t": 9,
"x": 935,
"y": 912
},
{
"t": 10,
"x": 923,
"y": 912
},
{
"t": 11,
"x": 916,
"y": 916
},
{
"t": 12,
"x": 904,
"y": 916
},
{
"t": 13,
"x": 896,
"y": 919
},
{
"t": 14,
"x": 884,
"y": 919
},
{
"t": 15,
"x": 876,
"y": 916
},
{
"t": 16,
"x": 865,
"y": 916
},
{
"t": 17,
"x": 857,
"y": 912
},
{
"t": 18,
"x": 845,
"y": 912
},
{
"t": 19,
"x": 837,
"y": 908
},
{
"t": 20,
"x": 826,
"y": 908
},
{
"t": 21,
"x": 814,
"y": 908
},
{
"t": 22,
"x": 806,
"y": 904
},
{
"t": 23,
"x": 794,
"y": 904
}
],
"birth": 0,
"death": 10
},
{
"id": 5,
"type": "blue wooden toy train",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 900
},
{
"t": 1,
"x": 994,
"y": 912
},
{
"t": 2,
"x": 994,
"y": 892
},
{
"t": 3,
"x": 990,
"y": 904
},
{
"t": 4,
"x": 986,
"y": 912
},
{
"t": 5,
"x": 986,
"y": 892
},
{
"t": 6,
"x": 982,
"y": 904
},
{
"t": 7,
"x": 978,
"y": 912
},
{
"t": 8,
"x": 974,
"y": 919
},
{
"t": 9,
"x": 974,
"y": 900
},
{
"t": 10,
"x": 970,
"y": 912
},
{
"t": 11,
"x": 966,
"y": 919
},
{
"t": 12,
"x": 962,
"y": 927
},
{
"t": 13,
"x": 962,
"y": 908
},
{
"t": 14,
"x": 958,
"y": 916
},
{
"t": 15,
"x": 955,
"y": 923
},
{
"t": 16,
"x": 955,
"y": 904
},
{
"t": 17,
"x": 951,
"y": 912
},
{
"t": 18,
"x": 947,
"y": 919
},
{
"t": 19,
"x": 943,
"y": 927
},
{
"t": 20,
"x": 943,
"y": 908
},
{
"t": 21,
"x": 939,
"y": 916
},
{
"t": 22,
"x": 935,
"y": 923
},
{
"t": 23,
"x": 931,
"y": 931
}
],
"birth": 0,
"death": 10
},
{
"id": 6,
"type": "blue wooden toy train",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 998
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 10
},
{
"id": 7,
"type": "red plastic hair straightener",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 1,
"x": 998,
"y": 341
},
{
"t": 2,
"x": 982,
"y": 361
},
{
"t": 3,
"x": 966,
"y": 380
},
{
"t": 4,
"x": 947,
"y": 404
},
{
"t": 5,
"x": 931,
"y": 423
},
{
"t": 6,
"x": 916,
"y": 443
},
{
"t": 7,
"x": 896,
"y": 462
},
{
"t": 8,
"x": 880,
"y": 474
},
{
"t": 9,
"x": 869,
"y": 478
},
{
"t": 10,
"x": 857,
"y": 482
},
{
"t": 11,
"x": 845,
"y": 486
},
{
"t": 12,
"x": 833,
"y": 490
},
{
"t": 13,
"x": 822,
"y": 494
},
{
"t": 14,
"x": 810,
"y": 498
},
{
"t": 15,
"x": 798,
"y": 501
},
{
"t": 16,
"x": 787,
"y": 505
},
{
"t": 17,
"x": 775,
"y": 509
},
{
"t": 18,
"x": 763,
"y": 513
},
{
"t": 19,
"x": 751,
"y": 517
},
{
"t": 20,
"x": 740,
"y": 521
},
{
"t": 21,
"x": 728,
"y": 525
},
{
"t": 22,
"x": 716,
"y": 529
},
{
"t": 23,
"x": 705,
"y": 533
}
],
"birth": 0,
"death": 10
},
{
"id": 8,
"type": "blue and white high-top sneaker",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 857
},
{
"t": 1,
"x": 990,
"y": 861
},
{
"t": 2,
"x": 982,
"y": 865
},
{
"t": 3,
"x": 974,
"y": 869
},
{
"t": 4,
"x": 966,
"y": 873
},
{
"t": 5,
"x": 958,
"y": 876
},
{
"t": 6,
"x": 947,
"y": 880
},
{
"t": 7,
"x": 939,
"y": 884
},
{
"t": 8,
"x": 927,
"y": 884
},
{
"t": 9,
"x": 919,
"y": 888
},
{
"t": 10,
"x": 908,
"y": 888
},
{
"t": 11,
"x": 896,
"y": 888
},
{
"t": 12,
"x": 888,
"y": 892
},
{
"t": 13,
"x": 876,
"y": 892
},
{
"t": 14,
"x": 865,
"y": 892
},
{
"t": 15,
"x": 853,
"y": 892
},
{
"t": 16,
"x": 845,
"y": 888
},
{
"t": 17,
"x": 833,
"y": 888
},
{
"t": 18,
"x": 822,
"y": 888
},
{
"t": 19,
"x": 814,
"y": 884
},
{
"t": 20,
"x": 802,
"y": 884
},
{
"t": 21,
"x": 794,
"y": 880
},
{
"t": 22,
"x": 783,
"y": 880
},
{
"t": 23,
"x": 775,
"y": 876
}
],
"birth": 0,
"death": 10
},
{
"id": 9,
"type": "blue and white striped fabric dishtowel",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 1,
"x": 998,
"y": 556
},
{
"t": 2,
"x": 998,
"y": 541
},
{
"t": 3,
"x": 994,
"y": 548
},
{
"t": 4,
"x": 994,
"y": 537
},
{
"t": 5,
"x": 990,
"y": 544
},
{
"t": 6,
"x": 990,
"y": 533
},
{
"t": 7,
"x": 986,
"y": 541
},
{
"t": 8,
"x": 986,
"y": 529
},
{
"t": 9,
"x": 982,
"y": 537
},
{
"t": 10,
"x": 982,
"y": 525
},
{
"t": 11,
"x": 978,
"y": 533
},
{
"t": 12,
"x": 978,
"y": 521
},
{
"t": 13,
"x": 974,
"y": 529
},
{
"t": 14,
"x": 974,
"y": 517
},
{
"t": 15,
"x": 970,
"y": 525
},
{
"t": 16,
"x": 970,
"y": 513
},
{
"t": 17,
"x": 966,
"y": 521
},
{
"t": 18,
"x": 966,
"y": 509
},
{
"t": 19,
"x": 962,
"y": 517
},
{
"t": 20,
"x": 962,
"y": 505
},
{
"t": 21,
"x": 958,
"y": 513
},
{
"t": 22,
"x": 958,
"y": 498
},
{
"t": 23,
"x": 955,
"y": 505
}
],
"birth": 0,
"death": 10
},
{
"id": 10,
"type": "blue portable hard drive",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 1,
"x": 998,
"y": 654
},
{
"t": 2,
"x": 998,
"y": 642
},
{
"t": 3,
"x": 994,
"y": 650
},
{
"t": 4,
"x": 994,
"y": 638
},
{
"t": 5,
"x": 990,
"y": 646
},
{
"t": 6,
"x": 990,
"y": 634
},
{
"t": 7,
"x": 986,
"y": 642
},
{
"t": 8,
"x": 986,
"y": 630
},
{
"t": 9,
"x": 982,
"y": 638
},
{
"t": 10,
"x": 982,
"y": 626
},
{
"t": 11,
"x": 978,
"y": 634
},
{
"t": 12,
"x": 978,
"y": 623
},
{
"t": 13,
"x": 974,
"y": 630
},
{
"t": 14,
"x": 974,
"y": 619
},
{
"t": 15,
"x": 970,
"y": 626
},
{
"t": 16,
"x": 970,
"y": 615
},
{
"t": 17,
"x": 966,
"y": 623
},
{
"t": 18,
"x": 966,
"y": 611
},
{
"t": 19,
"x": 962,
"y": 619
},
{
"t": 20,
"x": 962,
"y": 607
},
{
"t": 21,
"x": 958,
"y": 615
},
{
"t": 22,
"x": 958,
"y": 603
},
{
"t": 23,
"x": 955,
"y": 611
}
],
"birth": 0,
"death": 10
},
{
"id": 11,
"type": "blue and white high-top sneaker",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 1,
"x": 998,
"y": 603
},
{
"t": 4,
"x": 990,
"y": 607
},
{
"t": 7,
"x": 986,
"y": 599
},
{
"t": 10,
"x": 978,
"y": 595
},
{
"t": 12,
"x": 974,
"y": 587
},
{
"t": 14,
"x": 970,
"y": 580
},
{
"t": 16,
"x": 962,
"y": 576
},
{
"t": 18,
"x": 958,
"y": 568
},
{
"t": 20,
"x": 951,
"y": 564
},
{
"t": 22,
"x": 947,
"y": 556
},
{
"t": 23,
"x": 943,
"y": 548
}
],
"birth": 0,
"death": 10
},
{
"id": 12,
"type": "red plastic New Super Mario Bros. Wii game case",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 884
},
{
"t": 1,
"x": 994,
"y": 892
},
{
"t": 2,
"x": 994,
"y": 880
},
{
"t": 4,
"x": 990,
"y": 888
},
{
"t": 5,
"x": 986,
"y": 896
},
{
"t": 6,
"x": 986,
"y": 884
},
{
"t": 7,
"x": 982,
"y": 892
},
{
"t": 8,
"x": 978,
"y": 900
},
{
"t": 9,
"x": 978,
"y": 888
},
{
"t": 10,
"x": 974,
"y": 896
},
{
"t": 11,
"x": 970,
"y": 904
},
{
"t": 12,
"x": 970,
"y": 892
},
{
"t": 13,
"x": 966,
"y": 900
},
{
"t": 14,
"x": 962,
"y": 908
},
{
"t": 15,
"x": 958,
"y": 916
},
{
"t": 16,
"x": 958,
"y": 904
},
{
"t": 17,
"x": 955,
"y": 912
},
{
"t": 18,
"x": 951,
"y": 919
},
{
"t": 19,
"x": 947,
"y": 927
},
{
"t": 20,
"x": 943,
"y": 935
},
{
"t": 21,
"x": 943,
"y": 923
},
{
"t": 22,
"x": 939,
"y": 931
},
{
"t": 23,
"x": 935,
"y": 939
}
],
"birth": 0,
"death": 10
},
{
"id": 13,
"type": "blue and white high-top sneaker",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 1,
"x": 998,
"y": 849
},
{
"t": 3,
"x": 994,
"y": 833
},
{
"t": 5,
"x": 986,
"y": 837
},
{
"t": 7,
"x": 978,
"y": 841
},
{
"t": 9,
"x": 970,
"y": 845
},
{
"t": 11,
"x": 958,
"y": 845
},
{
"t": 13,
"x": 943,
"y": 841
},
{
"t": 14,
"x": 935,
"y": 837
},
{
"t": 16,
"x": 923,
"y": 833
},
{
"t": 18,
"x": 908,
"y": 830
},
{
"t": 19,
"x": 900,
"y": 826
},
{
"t": 21,
"x": 884,
"y": 822
},
{
"t": 22,
"x": 876,
"y": 818
},
{
"t": 23,
"x": 869,
"y": 814
}
],
"birth": 0,
"death": 10
},
{
"id": 14,
"type": "blue and white high-top sneaker",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 978
},
{
"t": 1,
"x": 994,
"y": 986
},
{
"t": 3,
"x": 986,
"y": 990
},
{
"t": 5,
"x": 978,
"y": 994
},
{
"t": 7,
"x": 966,
"y": 994
},
{
"t": 9,
"x": 955,
"y": 994
},
{
"t": 10,
"x": 947,
"y": 990
},
{
"t": 11,
"x": 935,
"y": 990
},
{
"t": 12,
"x": 927,
"y": 986
},
{
"t": 14,
"x": 912,
"y": 986
},
{
"t": 15,
"x": 904,
"y": 982
},
{
"t": 17,
"x": 892,
"y": 982
},
{
"t": 18,
"x": 884,
"y": 978
},
{
"t": 19,
"x": 876,
"y": 974
},
{
"t": 21,
"x": 861,
"y": 974
},
{
"t": 22,
"x": 853,
"y": 970
},
{
"t": 23,
"x": 845,
"y": 966
}
],
"birth": 0,
"death": 10
},
{
"id": 15,
"type": "blue and red running shoe",
"vis": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"is_in_frame": [
{
"t": 0,
"v": true
},
{
"t": 1,
"v": false
}
],
"image_positions": [
{
"t": 0,
"x": 998,
"y": 994
},
{
"t": 1,
"x": null,
"y": null
}
],
"birth": 0,
"death": 10
},
{
"id": 16,
"type": "red plastic New Super Mario Bros. Wii game case",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 1,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 1,
"x": 1,
"y": 349
},
{
"t": 2,
"x": 9,
"y": 361
},
{
"t": 3,
"x": 17,
"y": 369
},
{
"t": 4,
"x": 21,
"y": 357
},
{
"t": 5,
"x": 29,
"y": 365
},
{
"t": 6,
"x": 33,
"y": 353
},
{
"t": 7,
"x": 37,
"y": 345
},
{
"t": 8,
"x": 44,
"y": 349
},
{
"t": 9,
"x": 52,
"y": 353
},
{
"t": 10,
"x": 60,
"y": 357
},
{
"t": 11,
"x": 68,
"y": 361
},
{
"t": 12,
"x": 80,
"y": 361
},
{
"t": 13,
"x": 91,
"y": 365
},
{
"t": 14,
"x": 103,
"y": 369
},
{
"t": 15,
"x": 115,
"y": 369
},
{
"t": 16,
"x": 123,
"y": 373
},
{
"t": 17,
"x": 134,
"y": 373
},
{
"t": 18,
"x": 146,
"y": 373
},
{
"t": 19,
"x": 158,
"y": 373
},
{
"t": 20,
"x": 169,
"y": 373
},
{
"t": 21,
"x": 181,
"y": 373
},
{
"t": 22,
"x": 189,
"y": 376
},
{
"t": 23,
"x": 201,
"y": 376
}
],
"birth": 1,
"death": 10
}
],
"interactions": [
{
"id": 18,
"inputs": [
3,
5
],
"outputs": [
3,
5
],
"begins_at": 0,
"ends_at": 4,
"type": "contact"
},
{
"id": 0,
"inputs": [
0,
65535
],
"outputs": [
0,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 3,
"inputs": [
1,
65535
],
"outputs": [
1,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 19,
"inputs": [
3,
65535
],
"outputs": [
3,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 20,
"inputs": [
4,
12
],
"outputs": [
4,
12
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 21,
"inputs": [
4,
65535
],
"outputs": [
4,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 24,
"inputs": [
5,
65535
],
"outputs": [
5,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 25,
"inputs": [
6,
65535
],
"outputs": [
6,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 26,
"inputs": [
8,
65535
],
"outputs": [
8,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 27,
"inputs": [
9,
65535
],
"outputs": [
9,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 4,
"inputs": [
10,
65535
],
"outputs": [
10,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 5,
"inputs": [
11,
65535
],
"outputs": [
11,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 6,
"inputs": [
12,
65535
],
"outputs": [
12,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 7,
"inputs": [
13,
65535
],
"outputs": [
13,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 8,
"inputs": [
14,
65535
],
"outputs": [
14,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 9,
"inputs": [
15,
65535
],
"outputs": [
15,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 10,
"inputs": [
16,
65535
],
"outputs": [
16,
65535
],
"begins_at": 0,
"ends_at": 10,
"type": "contact"
},
{
"id": 17,
"inputs": [
2,
8
],
"outputs": [
2,
8
],
"begins_at": 4,
"ends_at": 4,
"type": "collision"
},
{
"id": 1,
"inputs": [
1,
16
],
"outputs": [
1,
16
],
"begins_at": 6,
"ends_at": 6,
"type": "collision"
},
{
"id": 11,
"inputs": [
2,
3
],
"outputs": [
2,
3
],
"begins_at": 6,
"ends_at": 6,
"type": "collision"
},
{
"id": 12,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 6,
"ends_at": 6,
"type": "collision"
},
{
"id": 2,
"inputs": [
1,
16
],
"outputs": [
1,
16
],
"begins_at": 7,
"ends_at": 10,
"type": "contact"
},
{
"id": 13,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 7,
"ends_at": 10,
"type": "contact"
},
{
"id": 22,
"inputs": [
4,
8
],
"outputs": [
4,
8
],
"begins_at": 8,
"ends_at": 8,
"type": "collision"
},
{
"id": 23,
"inputs": [
4,
8
],
"outputs": [
4,
8
],
"begins_at": 9,
"ends_at": 10,
"type": "contact"
},
{
"id": 14,
"inputs": [
2,
8
],
"outputs": [
2,
8
],
"begins_at": 10,
"ends_at": 10,
"type": "collision"
},
{
"id": 15,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 10,
"ends_at": 10,
"type": "collision"
},
{
"id": 28,
"inputs": [
9,
16
],
"outputs": [
9,
16
],
"begins_at": 10,
"ends_at": 10,
"type": "collision"
},
{
"id": 16,
"inputs": [
2,
8
],
"outputs": [
2,
8
],
"begins_at": 11,
"ends_at": 12,
"type": "contact"
},
{
"id": 29,
"inputs": [
9,
16
],
"outputs": [
9,
16
],
"begins_at": 12,
"ends_at": 12,
"type": "collision"
}
]
}Parsed ground truth
{
"objects": [
{
"id": 0,
"type": "brown tandem",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 336,
"y": 642
},
{
"t": 3,
"x": 337,
"y": 644
},
{
"t": 5,
"x": 342,
"y": 641
},
{
"t": 6,
"x": 341,
"y": 646
},
{
"t": 7,
"x": 333,
"y": 613
},
{
"t": 8,
"x": 344,
"y": 641
},
{
"t": 10,
"x": 333,
"y": 624
},
{
"t": 12,
"x": 335,
"y": 626
},
{
"t": 13,
"x": 348,
"y": 648
},
{
"t": 14,
"x": 347,
"y": 657
},
{
"t": 16,
"x": 348,
"y": 658
},
{
"t": 17,
"x": 341,
"y": 623
},
{
"t": 18,
"x": 340,
"y": 632
},
{
"t": 19,
"x": 351,
"y": 659
},
{
"t": 20,
"x": 350,
"y": 663
},
{
"t": 21,
"x": 344,
"y": 629
},
{
"t": 23,
"x": 347,
"y": 629
}
],
"birth": 0,
"death": 24
},
{
"id": 1,
"type": "green mountain",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 799,
"y": 786
},
{
"t": 21,
"x": 783,
"y": 767
}
],
"birth": 0,
"death": 24
},
{
"id": 2,
"type": "blue minivan",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 873,
"y": 927
},
{
"t": 1,
"x": 864,
"y": 926
},
{
"t": 2,
"x": 856,
"y": 925
},
{
"t": 3,
"x": 848,
"y": 924
},
{
"t": 4,
"x": 851,
"y": 922
},
{
"t": 5,
"x": 839,
"y": 921
},
{
"t": 6,
"x": 832,
"y": 920
},
{
"t": 7,
"x": 825,
"y": 919
},
{
"t": 8,
"x": 819,
"y": 918
},
{
"t": 9,
"x": 812,
"y": 917
},
{
"t": 10,
"x": 806,
"y": 916
},
{
"t": 11,
"x": 800,
"y": 915
},
{
"t": 12,
"x": 798,
"y": 914
},
{
"t": 13,
"x": 791,
"y": 914
},
{
"t": 15,
"x": 786,
"y": 910
},
{
"t": 17,
"x": 784,
"y": 909
},
{
"t": 18,
"x": 780,
"y": 907
},
{
"t": 19,
"x": 776,
"y": 906
},
{
"t": 20,
"x": 774,
"y": 905
},
{
"t": 21,
"x": 787,
"y": 910
},
{
"t": 22,
"x": 780,
"y": 907
},
{
"t": 23,
"x": 774,
"y": 905
}
],
"birth": 0,
"death": 24
},
{
"id": 3,
"type": "cyan chopper",
"vis": [
{
"t": 0,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": 948,
"y": 847
},
{
"t": 1,
"x": 940,
"y": 845
},
{
"t": 2,
"x": 933,
"y": 844
},
{
"t": 3,
"x": 924,
"y": 842
},
{
"t": 4,
"x": 916,
"y": 839
},
{
"t": 5,
"x": 909,
"y": 836
},
{
"t": 6,
"x": 903,
"y": 835
},
{
"t": 7,
"x": 897,
"y": 835
},
{
"t": 8,
"x": 890,
"y": 833
},
{
"t": 9,
"x": 883,
"y": 831
},
{
"t": 10,
"x": 876,
"y": 830
},
{
"t": 11,
"x": 869,
"y": 827
},
{
"t": 12,
"x": 862,
"y": 826
},
{
"t": 13,
"x": 856,
"y": 825
},
{
"t": 14,
"x": 849,
"y": 823
},
{
"t": 15,
"x": 844,
"y": 822
},
{
"t": 16,
"x": 837,
"y": 820
},
{
"t": 17,
"x": 830,
"y": 819
},
{
"t": 18,
"x": 824,
"y": 817
},
{
"t": 19,
"x": 818,
"y": 816
},
{
"t": 20,
"x": 813,
"y": 815
},
{
"t": 21,
"x": 806,
"y": 813
},
{
"t": 22,
"x": 801,
"y": 811
},
{
"t": 23,
"x": 795,
"y": 809
}
],
"birth": 0,
"death": 24
},
{
"id": 4,
"type": "blue jet",
"vis": [
{
"t": 0,
"v": false
},
{
"t": 2,
"v": true
}
],
"is_in_frame": [
{
"t": 0,
"v": false
},
{
"t": 2,
"v": true
}
],
"image_positions": [
{
"t": 0,
"x": null,
"y": null
},
{
"t": 2,
"x": 0,
"y": 331
},
{
"t": 3,
"x": 5,
"y": 332
},
{
"t": 4,
"x": 5,
"y": 341
},
{
"t": 5,
"x": 8,
"y": 340
},
{
"t": 6,
"x": 12,
"y": 338
},
{
"t": 7,
"x": 14,
"y": 338
},
{
"t": 11,
"x": 56,
"y": 339
},
{
"t": 12,
"x": 66,
"y": 339
},
{
"t": 13,
"x": 16,
"y": 339
},
{
"t": 14,
"x": 26,
"y": 339
},
{
"t": 15,
"x": 37,
"y": 339
},
{
"t": 16,
"x": 48,
"y": 339
},
{
"t": 17,
"x": 15,
"y": 333
},
{
"t": 19,
"x": 16,
"y": 329
},
{
"t": 20,
"x": 18,
"y": 324
},
{
"t": 21,
"x": 21,
"y": 319
},
{
"t": 22,
"x": 29,
"y": 316
},
{
"t": 23,
"x": 40,
"y": 316
}
],
"birth": 2,
"death": 24
}
],
"interactions": [
{
"id": 0,
"inputs": [
0,
65535
],
"outputs": [
0,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 1,
"inputs": [
1,
65535
],
"outputs": [
1,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 2,
"inputs": [
2,
65535
],
"outputs": [
2,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
},
{
"id": 3,
"inputs": [
3,
65535
],
"outputs": [
3,
65535
],
"begins_at": 0,
"ends_at": 23,
"type": "contact"
}
]
}Complete per-sample evaluator report
{
"metrics": {
"avg_type_similarity": 0.3302667003277321,
"avg_size_diff": null,
"avg_3d_iou": null,
"avg_image_position_px_diff": 190.1839939160884,
"avg_image_position_segmentation_acc": 0.325,
"avg_image_position_segmentation_acc_small": 0.375,
"avg_image_position_segmentation_acc_medium": 0.23611111111111113,
"avg_image_position_segmentation_acc_large": 0.5416666666666666,
"avg_image_position_segmentation_acc_first_frame": 0.5,
"avg_image_position_segmentation_visible_acc": 0.325,
"avg_quaternion_geodesic_deg": null,
"visibility_acc": 0.7833333333333333,
"visibility_true_acc": 0.775,
"visibility_false_acc": 0.5,
"is_in_frame_acc": 0.8083333333333333,
"is_in_frame_true_acc": 0.8,
"is_in_frame_false_acc": 0.5,
"vis_acc_from_px": 0.7833333333333333,
"vis_acc_from_px_true_acc": 0.775,
"vis_acc_from_px_false_acc": 0.5,
"num_matches": 5,
"num_unmatched_pred": 12,
"num_unmatched_target": 0
},
"supports": {
"avg_type_similarity": 5,
"avg_size_diff": 0,
"avg_3d_iou": 0,
"avg_image_position_px_diff": 4,
"avg_image_position_segmentation_acc": 5,
"avg_image_position_segmentation_acc_small": 1,
"avg_image_position_segmentation_acc_medium": 3,
"avg_image_position_segmentation_acc_large": 1,
"avg_image_position_segmentation_acc_first_frame": 4,
"avg_image_position_segmentation_visible_acc": 5,
"avg_quaternion_geodesic_deg": 0,
"visibility_acc": 5,
"visibility_true_acc": 5,
"visibility_false_acc": 1,
"is_in_frame_acc": 5,
"is_in_frame_true_acc": 5,
"is_in_frame_false_acc": 1,
"vis_acc_from_px": 5,
"vis_acc_from_px_true_acc": 5,
"vis_acc_from_px_false_acc": 1
},
"matching": {
"matches": [
{
"target_index": 0,
"pred_index": 0,
"target_id": 0,
"pred_id": 0,
"cost": 1.125,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.5,
"weighted_cost": 0.5,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.625,
"weighted_cost": 0.625,
"used_missing_value": false
}
]
},
{
"target_index": 1,
"pred_index": 1,
"target_id": 1,
"pred_id": 1,
"cost": 1.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
}
]
},
{
"target_index": 2,
"pred_index": 5,
"target_id": 2,
"pred_id": 5,
"cost": 0.4583333333333333,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.4583333333333333,
"weighted_cost": 0.4583333333333333,
"used_missing_value": false
}
]
},
{
"target_index": 3,
"pred_index": 3,
"target_id": 3,
"pred_id": 3,
"cost": 1.2916666666666667,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 0.2916666666666667,
"weighted_cost": 0.2916666666666667,
"used_missing_value": false
}
]
},
{
"target_index": 4,
"pred_index": 15,
"target_id": 4,
"pred_id": 15,
"cost": 1.0,
"cost_breakdown": [
{
"name": "mutual_type",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "same_seg",
"weight": 1.0,
"raw_cost": 1.0,
"weighted_cost": 1.0,
"used_missing_value": false
}
]
}
],
"unmatched_target_ids": [],
"unmatched_pred_ids": [
2,
4,
6,
7,
8,
9,
10,
11,
12,
13,
14,
16
],
"target_type_counts": {
"brown tandem": 1,
"green mountain": 1,
"blue minivan": 1,
"cyan chopper": 1,
"blue jet": 1
},
"pred_type_counts": {
"dark brown leather wingtip shoe with orange sole": 1,
"green wooden dollhouse bedroom furniture set": 1,
"dark grey and light grey plastic monster action figure": 1,
"blue and white athletic shoe": 1,
"blue wooden toy kitchen set": 1,
"blue wooden toy train": 2,
"red plastic hair straightener": 1,
"blue and white high-top sneaker": 4,
"blue and white striped fabric dishtowel": 1,
"blue portable hard drive": 1,
"red plastic New Super Mario Bros. Wii game case": 2,
"blue and red running shoe": 1
},
"assignment_backend": "scipy_dense",
"cost_functions": [
"mutual_type",
"same_seg"
],
"cost_weights": {
"mutual_type": 1.0,
"same_seg": 1.0
},
"num_frames": 24
},
"scene_metrics": {
"thresholds": [
0.5,
1.0,
2.0,
3.0
],
"counts_by_threshold": {
"0.5": {
"true_positives": 5,
"false_positives": 0,
"false_negatives": 0
},
"1.0": {
"true_positives": 1,
"false_positives": 4,
"false_negatives": 0
},
"2.0": {
"true_positives": 0,
"false_positives": 5,
"false_negatives": 0
},
"3.0": {
"true_positives": 0,
"false_positives": 5,
"false_negatives": 0
}
},
"jaccard_by_threshold": {
"0.5": 1.0,
"1.0": 0.2,
"2.0": 0.0,
"3.0": 0.0
},
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 13,
"false_negatives": 1
},
"0.5": {
"true_positives": 4,
"false_positives": 13,
"false_negatives": 1
},
"0.75": {
"true_positives": 4,
"false_positives": 13,
"false_negatives": 1
},
"1.0": {
"true_positives": 3,
"false_positives": 14,
"false_negatives": 2
}
},
"visibility_precision_by_threshold": {
"0.25": 0.23529411764705882,
"0.5": 0.23529411764705882,
"0.75": 0.23529411764705882,
"1.0": 0.17647058823529413
},
"visibility_recall_by_threshold": {
"0.25": 0.8,
"0.5": 0.8,
"0.75": 0.8,
"1.0": 0.6
},
"visibility_f1_by_threshold": {
"0.25": 0.3636363636363636,
"0.5": 0.3636363636363636,
"0.75": 0.3636363636363636,
"1.0": 0.2727272727272727
},
"visibility_by_motion_group": {
"static": {
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 0,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 0,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 0,
"false_negatives": 0
},
"1.0": {
"true_positives": 3,
"false_positives": 1,
"false_negatives": 1
}
},
"visibility_precision_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 0.75
},
"visibility_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 0.75
},
"visibility_f1_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 0.75
}
},
"unknown": {
"visibility_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 1
},
"0.5": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 1
},
"0.75": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 1
},
"1.0": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 1
}
},
"visibility_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"visibility_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"visibility_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
}
},
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 14,
"false_negatives": 2
},
"0.5": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 3
},
"0.75": {
"true_positives": 0,
"false_positives": 17,
"false_negatives": 5
},
"1.0": {
"true_positives": 0,
"false_positives": 17,
"false_negatives": 5
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.17647058823529413,
"0.5": 0.11764705882352941,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_recall_by_threshold": {
"0.25": 0.6,
"0.5": 0.4,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_f1_by_threshold": {
"0.25": 0.2727272727272727,
"0.5": 0.1818181818181818,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_frame_average_precision": 0.07352941176470588,
"segmentation_frame_average_recall": 0.25,
"segmentation_frame_average_f1": 0.11363636363636362,
"segmentation_first_frame_counts": {
"true_positives": 2,
"false_positives": 15,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.11764705882352941,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 0.21052631578947367,
"segmentation_by_motion_group": {
"static": {
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 3,
"false_positives": 1,
"false_negatives": 1
},
"0.5": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 2
},
"0.75": {
"true_positives": 0,
"false_positives": 4,
"false_negatives": 4
},
"1.0": {
"true_positives": 0,
"false_positives": 4,
"false_negatives": 4
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.75,
"0.5": 0.5,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_recall_by_threshold": {
"0.25": 0.75,
"0.5": 0.5,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_f1_by_threshold": {
"0.25": 0.75,
"0.5": 0.5,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_frame_average_precision": 0.3125,
"segmentation_frame_average_recall": 0.3125,
"segmentation_frame_average_f1": 0.3125,
"segmentation_first_frame_counts": {
"true_positives": 2,
"false_positives": 2,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.5,
"segmentation_first_frame_recall": 1.0,
"segmentation_first_frame_f1": 0.6666666666666666
},
"unknown": {
"segmentation_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 1
},
"0.5": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 1
},
"0.75": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 1
},
"1.0": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 1
}
},
"segmentation_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"segmentation_frame_average_precision": 0.0,
"segmentation_frame_average_recall": 0.0,
"segmentation_frame_average_f1": 0.0,
"segmentation_first_frame_counts": {
"true_positives": 0,
"false_positives": 13,
"false_negatives": 0
},
"segmentation_first_frame_precision": 0.0,
"segmentation_first_frame_recall": 0.0,
"segmentation_first_frame_f1": 0.0
}
}
},
"per_match_metrics": [
{
"target_id": 0,
"pred_id": 0,
"cost": 1.125,
"type_similarity": 0.35167219250318005,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 26.766523447167117,
"image_position_segmentation_acc": 0.375,
"image_position_segmentation_acc_small": 0.375,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 0.375,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 1,
"pred_id": 1,
"cost": 1.0,
"type_similarity": 0.2552283350418539,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 505.67001734489486,
"image_position_segmentation_acc": 0.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 0.0,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 0.0,
"image_position_segmentation_visible_acc": 0.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 0.875,
"visibility_true_acc": 0.875,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 0.875,
"vis_acc_from_px_true_acc": 0.875,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 2,
"pred_id": 5,
"cost": 0.4583333333333333,
"type_similarity": 0.46547871561669607,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 199.97275894932497,
"image_position_segmentation_acc": 0.5416666666666666,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": null,
"image_position_segmentation_acc_large": 0.5416666666666666,
"image_position_segmentation_acc_first_frame": 1.0,
"image_position_segmentation_visible_acc": 0.5416666666666666,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 3,
"pred_id": 3,
"cost": 1.2916666666666667,
"type_similarity": 0.10738896387563442,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": 28.326675922966558,
"image_position_segmentation_acc": 0.7083333333333334,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 0.7083333333333334,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": 0.0,
"image_position_segmentation_visible_acc": 0.7083333333333334,
"quaternion_geodesic_deg": null,
"visibility_acc": 1.0,
"visibility_true_acc": 1.0,
"visibility_false_acc": null,
"is_in_frame_acc": 1.0,
"is_in_frame_true_acc": 1.0,
"is_in_frame_false_acc": null,
"vis_acc_from_px": 1.0,
"vis_acc_from_px_true_acc": 1.0,
"vis_acc_from_px_false_acc": null
},
{
"target_id": 4,
"pred_id": 15,
"cost": 1.0,
"type_similarity": 0.47156529460129604,
"size_diff": null,
"3d_iou": null,
"image_position_px_diff": null,
"image_position_segmentation_acc": 0.0,
"image_position_segmentation_acc_small": null,
"image_position_segmentation_acc_medium": 0.0,
"image_position_segmentation_acc_large": null,
"image_position_segmentation_acc_first_frame": null,
"image_position_segmentation_visible_acc": 0.0,
"quaternion_geodesic_deg": null,
"visibility_acc": 0.041666666666666664,
"visibility_true_acc": 0.0,
"visibility_false_acc": 0.5,
"is_in_frame_acc": 0.041666666666666664,
"is_in_frame_true_acc": 0.0,
"is_in_frame_false_acc": 0.5,
"vis_acc_from_px": 0.041666666666666664,
"vis_acc_from_px_true_acc": 0.0,
"vis_acc_from_px_false_acc": 0.5
}
],
"interaction_metrics": {
"avg_interaction_temporal_span_giou": 0.4583333333333333,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 4,
"num_unmatched_pred_interactions": 26,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 30,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 30,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 30,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.13333333333333333,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.23529411764705882,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.13333333333333333,
"0.5": 0.13333333333333333,
"0.75": 0.13333333333333333,
"1.0": 0.13333333333333333
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.23529411764705882,
"0.5": 0.23529411764705882,
"0.75": 0.23529411764705882,
"1.0": 0.23529411764705882
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 26,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.13333333333333333,
"0.5": 0.13333333333333333,
"0.75": 0.13333333333333333,
"1.0": 0.13333333333333333
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.23529411764705882,
"0.5": 0.23529411764705882,
"0.75": 0.23529411764705882,
"1.0": 0.23529411764705882
}
},
"interaction_supports": {
"avg_interaction_temporal_span_giou": 4,
"avg_interaction_input_object_jaccard": 4,
"avg_interaction_output_object_jaccard": 4,
"avg_interaction_type_exact_match_acc": 4
},
"interaction_matching": {
"matches": [
{
"target_index": 0,
"pred_index": 0,
"target_id": 0,
"pred_id": 0,
"cost": 0.27083333333333337,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.27083333333333337,
"weighted_cost": 0.27083333333333337,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 1,
"pred_index": 3,
"target_id": 1,
"pred_id": 3,
"cost": 0.27083333333333337,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.27083333333333337,
"weighted_cost": 0.27083333333333337,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 2,
"pred_index": 24,
"target_id": 2,
"pred_id": 24,
"cost": 0.27083333333333337,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.27083333333333337,
"weighted_cost": 0.27083333333333337,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
},
{
"target_index": 3,
"pred_index": 19,
"target_id": 3,
"pred_id": 19,
"cost": 0.27083333333333337,
"cost_breakdown": [
{
"name": "type_exact",
"weight": 1.0,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "temporal_span",
"weight": 1.0,
"raw_cost": 0.27083333333333337,
"weighted_cost": 0.27083333333333337,
"used_missing_value": false
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5,
"raw_cost": 0.0,
"weighted_cost": 0.0,
"used_missing_value": false
}
]
}
],
"unmatched_target_ids": [],
"unmatched_pred_ids": [
1,
2,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
20,
21,
22,
23,
25,
26,
27,
28,
29
],
"target_type_counts": {
"contact": 4
},
"pred_type_counts": {
"contact": 21,
"collision": 9
},
"assignment_backend": "scipy_dense",
"cost_functions": [
"type_exact",
"temporal_span",
"input_object_set_mutual_jaccard",
"output_object_set_mutual_jaccard"
],
"cost_weights": {
"type_exact": 1.0,
"temporal_span": 1.0,
"input_object_set_mutual_jaccard": 0.5,
"output_object_set_mutual_jaccard": 0.5
}
},
"per_interaction_match_metrics": [
{
"target_index": 0,
"pred_index": 0,
"target_id": 0,
"pred_id": 0,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.27083333333333337,
"temporal_span_iou": 0.4583333333333333,
"temporal_span_giou": 0.4583333333333333,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 0.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 1,
"pred_index": 3,
"target_id": 1,
"pred_id": 3,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.27083333333333337,
"temporal_span_iou": 0.4583333333333333,
"temporal_span_giou": 0.4583333333333333,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 0.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 2,
"pred_index": 24,
"target_id": 2,
"pred_id": 24,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.27083333333333337,
"temporal_span_iou": 0.4583333333333333,
"temporal_span_giou": 0.4583333333333333,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 0.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
},
{
"target_index": 3,
"pred_index": 19,
"target_id": 3,
"pred_id": 19,
"target_type": "contact",
"pred_type": "contact",
"cost": 0.27083333333333337,
"temporal_span_iou": 0.4583333333333333,
"temporal_span_giou": 0.4583333333333333,
"input_object_jaccard": 1.0,
"output_object_jaccard": 1.0,
"type_exact_match": 1.0,
"temporal_span_exact_match": 0.0,
"input_object_set_exact_match": 1.0,
"output_object_set_exact_match": 1.0
}
],
"interaction_metrics_by_participants": {
"object_ground": {
"avg_interaction_temporal_span_giou": 0.4583333333333333,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 4,
"num_unmatched_pred_interactions": 14,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 18,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 18,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 18,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.2222222222222222,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.3636363636363636,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.2222222222222222,
"0.5": 0.2222222222222222,
"0.75": 0.2222222222222222,
"1.0": 0.2222222222222222
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.3636363636363636,
"0.5": 0.3636363636363636,
"0.75": 0.3636363636363636,
"1.0": 0.3636363636363636
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 14,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.2222222222222222,
"0.5": 0.2222222222222222,
"0.75": 0.2222222222222222,
"1.0": 0.2222222222222222
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.3636363636363636,
"0.5": 0.3636363636363636,
"0.75": 0.3636363636363636,
"1.0": 0.3636363636363636
}
},
"object_object": {
"avg_interaction_temporal_span_giou": null,
"avg_interaction_input_object_jaccard": null,
"avg_interaction_output_object_jaccard": null,
"avg_interaction_type_exact_match_acc": null,
"num_interaction_matches": 0,
"num_unmatched_pred_interactions": 12,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 12,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
}
},
"interaction_supports_by_participants": {
"object_ground": {
"avg_interaction_temporal_span_giou": 4,
"avg_interaction_input_object_jaccard": 4,
"avg_interaction_output_object_jaccard": 4,
"avg_interaction_type_exact_match_acc": 4
},
"object_object": {
"avg_interaction_temporal_span_giou": 0,
"avg_interaction_input_object_jaccard": 0,
"avg_interaction_output_object_jaccard": 0,
"avg_interaction_type_exact_match_acc": 0
}
},
"interaction_metrics_by_type": {
"collision": {
"avg_interaction_temporal_span_giou": null,
"avg_interaction_input_object_jaccard": null,
"avg_interaction_output_object_jaccard": null,
"avg_interaction_type_exact_match_acc": null,
"num_interaction_matches": 0,
"num_unmatched_pred_interactions": 9,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 9,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
}
},
"contact": {
"avg_interaction_temporal_span_giou": 0.4583333333333333,
"avg_interaction_input_object_jaccard": 1.0,
"avg_interaction_output_object_jaccard": 1.0,
"avg_interaction_type_exact_match_acc": 1.0,
"num_interaction_matches": 4,
"num_unmatched_pred_interactions": 17,
"num_unmatched_target_interactions": 0,
"interaction_temporal_iou_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_object_jaccard_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"interaction_temporal_iou_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
},
"0.5": {
"true_positives": 0,
"false_positives": 21,
"false_negatives": 0
},
"0.75": {
"true_positives": 0,
"false_positives": 21,
"false_negatives": 0
},
"1.0": {
"true_positives": 0,
"false_positives": 21,
"false_negatives": 0
}
},
"interaction_temporal_iou_precision_by_threshold": {
"0.25": 0.19047619047619047,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_recall_by_threshold": {
"0.25": 1.0,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_temporal_iou_f1_by_threshold": {
"0.25": 0.32,
"0.5": 0.0,
"0.75": 0.0,
"1.0": 0.0
},
"interaction_input_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
}
},
"interaction_input_object_jaccard_precision_by_threshold": {
"0.25": 0.19047619047619047,
"0.5": 0.19047619047619047,
"0.75": 0.19047619047619047,
"1.0": 0.19047619047619047
},
"interaction_input_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_input_object_jaccard_f1_by_threshold": {
"0.25": 0.32,
"0.5": 0.32,
"0.75": 0.32,
"1.0": 0.32
},
"interaction_output_object_jaccard_counts_by_threshold": {
"0.25": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
},
"0.5": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
},
"0.75": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
},
"1.0": {
"true_positives": 4,
"false_positives": 17,
"false_negatives": 0
}
},
"interaction_output_object_jaccard_precision_by_threshold": {
"0.25": 0.19047619047619047,
"0.5": 0.19047619047619047,
"0.75": 0.19047619047619047,
"1.0": 0.19047619047619047
},
"interaction_output_object_jaccard_recall_by_threshold": {
"0.25": 1.0,
"0.5": 1.0,
"0.75": 1.0,
"1.0": 1.0
},
"interaction_output_object_jaccard_f1_by_threshold": {
"0.25": 0.32,
"0.5": 0.32,
"0.75": 0.32,
"1.0": 0.32
}
}
},
"interaction_supports_by_type": {
"collision": {
"avg_interaction_temporal_span_giou": 0,
"avg_interaction_input_object_jaccard": 0,
"avg_interaction_output_object_jaccard": 0,
"avg_interaction_type_exact_match_acc": 0
},
"contact": {
"avg_interaction_temporal_span_giou": 4,
"avg_interaction_input_object_jaccard": 4,
"avg_interaction_output_object_jaccard": 4,
"avg_interaction_type_exact_match_acc": 4
}
}
}Exact evaluation configuration
{
"predictions_jsonl": "../inference_output.jsonl",
"dataset_json": "../test_data/movi_dump_sentinel_sharegpt.json",
"scene_root": "../test_data/movi_dump",
"output_json": "../evaluation_report.json",
"max_samples": null,
"num_workers": 1,
"show_progress": true,
"log_level": "INFO",
"visibility_filter_mode": "any_frame",
"pos2d_mode": "scaled_normalized",
"pos2d_scale": 1000.0,
"image_width": null,
"image_height": null,
"type_embed_model_name": "all-MiniLM-L6-v2",
"scene_similarity_thresholds": [
0.5,
1.0,
2.0,
3.0
],
"scene_segmentation_thresholds": [
0.25,
0.5,
0.75,
1.0
],
"matching": {
"costs": [
{
"name": "mutual_type",
"weight": 1.0
},
{
"name": "same_seg",
"weight": 1.0,
"missing_value": 1.0,
"kwargs": {
"only_use_first_frame": false
}
}
]
},
"interaction_matching": {
"costs": [
{
"name": "type_exact",
"weight": 1.0
},
{
"name": "temporal_span",
"weight": 1.0
},
{
"name": "input_object_set_mutual_jaccard",
"weight": 0.5
},
{
"name": "output_object_set_mutual_jaccard",
"weight": 0.5
}
]
},
"return_per_sample": true
}