# BUTom-ST21 "A spatial-temporal tomato dataset for video instance segmentation (VIS) and multi-object tracking (MOT), with pseudo-labelled training/validation sequences and a hand-corrected evaluation set." BUTom-ST21 is the video counterpart of the still-image dataset "BUTom21". It contains short image sequences of tomato plants together with per-frame id consistent instance annotations. Training and validation annotations are pseudo-labels produced with PAg-NeRF (from two segmentation backbones, Mask2Former and YOLO26); the evaluation annotations are "hand-corrected". Download instructions and minimal PyTorch dataloaders: https://github.com/Agricultural-Robotics-Bonn/BUTom21-ST21 --- ## Download & extract The dataset is distributed as per-component compressed archives. Large components ("rgb", "depth", "pseudo_labels_m2f", "pseudo_labels_Yolo26", "poses") are split into ~4 GB parts named ".tar.gz.aa", ".tar.gz.ab", ... ; smaller components are a single ".tar.gz". The loose ".yaml" and this README are not compressed. Archives: """ butomst21_rgb.tar.gz.aa .ab .ac ... (~27 GB after extraction) butomst21_depth.tar.gz.aa .ab ... (~8 GB after extraction) butomst21_pseudo_labels_m2f.tar.gz(.aa ...) (~285 GB after extraction) butomst21_pseudo_labels_Yolo26.tar.gz(.aa ...) (~308 GB after extraction) butomst21_eval_annotations.tar.gz (~41 GB after extraction) butomst21_poses.tar.gz(.aa ...) (~10 GB after extraction) """ To extract a split (multi-part) archive, concatenate the parts and untar in one step: """ cat butomst21_rgb.tar.gz.* | tar -xzf - """ To extract a single-part archive: """ tar -xzf butomst21_eval_annotations.tar.gz """ Each archive extracts to its own folder ("rgb/", "depth/", "eval_annotations/", ...), recreating the structure shown above. Place the extracted folders and the loose ".yaml" files together in one "BUTom-ST21/" directory. ## Sequences, splits, and the 83 vs 63 frame distinction The dataset contains "217 sequences", split into: - "train" — 123 sequences - "valid" — 72 sequences - "eval" — 22 sequences Each sequence is centered on a single "BUTom21 GT frame" (the still-image annotated frame), which sits at the middle of the sequence and links BUTom-ST21 back to BUTom21. Two frame counts appear throughout the dataset, and it is important to keep them apart: - "83 frames per sequence" — the frames provided as "rgb/", "depth/", and "poses/". PAg-NeRF requires these 83 frames as input because it discards the first and last 10 "warm-up" frames. Releasing all 83 lets users reproduce the pseudo-label generation themselves. - "63 usable frames per sequence" — the frames that carry annotations ("pseudo_labels_*" and "eval_annotations"). These are the 83 frames minus the 20 warm-up frames (10 at each end), with the GT frame at the centre. "annotated_seq_frame_list.yaml" lists the 63 annotated frames per sequence; "pagnerf_seq_frames.yaml" lists the full 83 frames per sequence. --- ## Folder structure """ BUTom-ST21/ ├── rgb/ 000/ .png ... # 83 frames per sequence ├── depth/ 000/ .png ... # 83 frames per sequence ├── pseudo_labels_m2f/ 000/ .pkl ... # train + valid (Mask2Former) ├── pseudo_labels_Yolo26/ 000/ .pkl ... # train + valid (YOLO26) ├── eval_annotations/ 197/ .pkl ... # 22 eval sequences, hand-corrected ├── poses/ │ ├── COLMAP/ 000/ {cameras.txt, images.txt, points3D.txt} │ └── HLoc/ 000/ {cameras.txt, images.txt, points3D.txt} ├── camera_parameters.yaml ├── image_camera_row_identity.yaml ├── train_val_eval_splits.yaml ├── annotated_seq_frame_list.yaml # 63 usable frames per sequence └── pagnerf_seq_frames.yaml # 83 frames per sequence """ Sequences are numbered folders ("000", "001", ...). Images are "720 × 1280" (width × height, portrait); each RGB image has a matching depth image with the "same file name" (".png"). The "" is the capture timestamp and is the unique identifier used throughout. --- ## Annotations (pickle format) Every annotated frame is a single ".pkl" file. It stores a Python dictionary keyed by the "track id" (int) of each tomato in that frame. The same physical tomato keeps the same track id throughout a sequence, so instances can be tracked over time by matching keys between consecutive frames — this is what makes the dataset suitable for VIS and MOT. Each track id maps to a dictionary with four fields: | field | description |------------------|--------------------------------------------------------------------| | "bbox" | bounding box in COCO format "[x, y, w, h]" | "instance_mask" | full-resolution binary mask, "1280 × 720" (bool) | "area" | mask area in pixels | "semantic_label" | ripeness subclass as a string (e.g. "red", "mixed-red", "green", "orange", "mixed_orange") Track ids are "guaranteed consistent" in the hand-corrected "eval_annotations", and "weakly consistent" in the pseudo-labelled "pseudo_labels_*" (training/validation). ### Pseudo-labels: Mask2Former vs YOLO26 "pseudo_labels_m2f/" and "pseudo_labels_Yolo26/" contain the training and validation pseudo-labels produced with two different segmentation backbones, so users can compare them. Both are generated by training on BUTom21, refining with PAg-NeRF, and post-processing. The eval set is not pseudo-labelled — it is hand-corrected and lives in "eval_annotations/". --- ## Poses For each sequence we provide camera poses in COLMAP text format, under "poses/COLMAP//" and "poses/HLoc//", each containing "cameras.txt", "images.txt", and "points3D.txt". The poses cover the same "83 frames" as "rgb/" and "depth/". - "HLoc" poses are provided for "all 217" sequences. - "COLMAP" poses are provided for the "204" sequences where COLMAP successfully registered all 83 frames. For the following "13" sequences COLMAP did not register the frames reliably, so only HLoc is provided: "038, 081, 084, 086, 089, 099, 102, 122, 144, 150, 179, 188, 201". Intrinsics are fixed during reconstruction, so each "cameras.txt" carries the exact intrinsics used for that sequence (see "camera_parameters.yaml"). --- ## Metadata files ### "train_val_eval_splits.yaml" Three lists ("train", "val", "eval") of sequence ids — 123 / 72 / 22. ### "annotated_seq_frame_list.yaml" Per sequence, the list of the "63 usable (annotated)" frame ids. These are the frames the "pseudo_labels_*" and "eval_annotations" pickles correspond to. ### "pagnerf_seq_frames.yaml" Per sequence, the list of the "83" frame ids present in "rgb/", "depth/", and "poses/" (GT frame at the centre). ### "image_camera_row_identity.yaml" Records which camera each sequence was captured with, grouped as "date → row → camera → {seq_id: center_frame_id}". The "center_frame_id" is the BUTom21 GT frame at the centre of the sequence, providing the link to the still-image dataset ("CKA_tomato_2021.json"). ### "camera_parameters.yaml" Intrinsics per camera ("cam_1", "cam_2"); they depend only on the camera and are constant across dates and rows. Values are PINHOLE intrinsics, given as a 3×3 matrix with "image_width" and "image_height". --- ## Categories (ripeness subclasses) | id | name | |-----|----------------| | 0 | "red" | | 1 | "mixed_red" | | 2 | "green" | | 3 | "orange" | | 4 | "mixed_orange" | In the pickles the ripeness is stored under "semantic_label" as the class "name" (string). --- ## Link to BUTom21 Each sequence is built around one BUTom21 GT image at its centre. Using "image_camera_row_identity.yaml" you can find the "center_frame_id" of a sequence, which is the BUTom21 annotated image; this connects the spatial-temporal dataset to the still-image dataset BUTom21. --- ## Reproducing the pseudo-labels Because the full 83-frame "rgb/", "depth/", and "poses/" are provided, users can re-run the pseudo-label pipeline: train Mask2Former / YOLO26 on BUTom21, run PAg-NeRF using the provided poses and 83-frame sequences (PAg-NeRF drops the 10 warm-up frames at each end), and post-process to obtain track-consistent instance masks over the 63 usable frames. --- ## Citation If you use this dataset, please cite both the associated paper and the dataset itself as below: ## Paper citation ## @misc{halstead2026imagespatialtemporaltomatodata, title={Still image and spatial-temporal tomato data enabling detection, segmentation, tracking, and video-instance segmentation using strong and weak labels}, author={Michael Halstead and Esra Guclu and Mohamed Farag and Enrico Pallotta and Christian Hund and Ribana Roscher and Maren Bennewitz and Juergen Gall and Cyrill Stachniss and Chris McCool}, year={2026}, eprint={2607.14934}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2607.14934}, } ## Dataset citation ## @data{FK2/TTPCNV_2026, author = {Halstead, Michael and Guclu, Esra and Farag, Mohamed and Pallotta, Enrico and Hund, Christian and Roscher, Ribana and Bennewitz, Maren and Gall, Juergen and McCool, Chris}, publisher = {bonndata}, title = {BUTom-ST21: Spatial-temporal tomato data enabling detection, segmentation, tracking, and video-instance segmentation using strong and weak labels, year = {2026}, version = {V1}, doi = {10.60507/FK2/TTPCNV}, url = {https://doi.org/10.60507/FK2/TTPCNV} } ---