# How to Use BUP-ST20

Below is a brief guide about the BUP-ST20 dataset.  
In total we provide 16,240 RGB/depth image pairs split across 275 sequences (train, valid, and eval).  
We provide weak labels for 12,430 frames in the train and validation sequences.  
The evaluation set has 3,810 hand-labelled ground-truth annotations.

## RGB and Depth Data
The data provided includes RGB and associated depth images in `.tiff` format.

## Annotations
The annotations are provided for each frame in a pickle file `pkl`.  
If there are no labels for a given `frameid`, we do not provide the `pkl` file.  
If a `pkl` file exists but contains an empty list, this means no label is available for that frame.

Each pickle file contains a dictionary where each key is an `instanceid` (unique per object in the sequence).  
In the evaluation set, these IDs are consistent across frames.  
In the training and validation sets, the annotations are weakly labelled and therefore ID consistency across frames is not guaranteed.

Each instance contains:
 * `bbox`: this is the bounding box defined as `[x, y, w, h]`, and `area`
 * `semlabel`: this is a number representing the semantic label for the sweet peppers (0: 'red', 1: 'yellow', 2: 'green', 3: 'mixed_red', 4: 'mixed_yellow')
 * `instance_mask`:  this is the instance mask for this object and is a binary mask

The dataset includes annotations for the train, validation, and evaluation sets.


## Odometry
We provide wheel odometry from a robotic platform for each sequence as a single CSV file located under:
 - BUPST20/odoms/{sequence_id}.csv

Each CSV file contains odometry and pose information for corresponding frames of that sequence.  
The columns in csv files are:

- `timestamp_us`: frameid
- `parent`: The robotic platform's odom frame (e.g., `odom`)
- `child`: The robotic platform's base frame (e.g., `base_link`)
- `tx`, `ty`, `tz`: Translation (in millimeters)
- `qx`, `qy`, `qz`, `qw`: Orientation

These values can be used to construct a homogeneous transformation matrix for pose estimation.

- Example rows from sequence 105:
```
timestamp_us, parent, child, tx, ty, tz, qx, qy, qz, qw
1600936849728261, odom, base_link, 5.3655536864665, 0, 0, 0, 0, 0, 1
1600936849794921, odom, base_link, 5.37639056151826, 0, 0, 0, 0, 0, 1
.
.
.
```

## Camera Parameters
The dataset includes a YAML file containing camera intrinsic and extrinsics in the following file:
    - cam_params.yaml

## Data Splits: train, valid, eval
The data splits are defined in the following file:
  - train_valid_eval_splits.yaml

which assigns each video sequence to one of the three sets: train, valid, or eval.
Each split (train, valid, eval) is assigned at the sequence level—not at the frame level.
We ask that you respect these splits and only use the train set for model training.


### Frame ID Consistency and Data Access ###
Each frame in the dataset is uniquely identified by its frameid (for example `1600936832187439`), which is consistent across all data types.
This allows you to access the corresponding data across modalities using the frameid:

RGB image:      BUPST20/rgb/{sequence_id}/{frame_id}.tiff  
Depth image:    BUPST20/depth/{sequence_id}/{frame_id}.tiff  
Annotation:     BUPST20/annotations/{sequence_id}/{frame_id}.pkl  
Odometry:       BUPST20/odoms/{sequence_id}.csv  → match with timestamp_us

This consistency allows for precise temporal and spatial alignment across RGB, depth, annotation, and robot pose information.


### From 2D Image Plane to 3D Coordinates (Optional) ###
For users interested in re-projection from 2D pixel coordinates to 3D space can use the provided depth, intrinsics, extrinsics, and odometry files.
Here is the recommended pipeline:

1. Start with:
  * A 2D pixel location in the RGB image
  * The depth value at that pixel (from the depth image)
  * The camera intrinsics (in cam_params.yaml)

2. Apply re-projection to compute the 3D point in the camera reference frame.

3. To transform the 3D point from the camera reference frame to the robot’s base frame:
  * Use the camera extrinsics matrix from cam_params.yaml.

4. To transform the 3D point from the robot's base frame to the robot’s odom frame:
  * Use the odometry information (tx, ty, tz, qx, qy, qz, qw) from the corresponding .csv file for the sequence.


- In short:
  pixel (u,v) + depth → 3D point in camera frame (re-projection)
  → transform with extrinsics → robot base frame
  → transform with wheel odometry → robot odom frame

This enables spatial reasoning tasks, such as tracking, mapping, or localization, using the provided RGB-D and odometry data.


## Citation
If you use the dataset in your work we ask that you cite the following paper.

@Article{2025bupst20,
  author   = {Guclu, Esra and Halstead, Michael and Denman, Simon and McCool, Chris},
  title    = {Weakly Labelled Spatial-Temporal Sweet Pepper Data: enabling higher quality detection, segmentation, and tracking.},
  journal  = {The International Journal of Robotics Research},
  year     = {2025},
  volume   = {},
  pages    = {},
  month    = {},
  doi      = {10.1177/02783649251379093},
  publisher= {SAGE Publications Sage UK: London, England}
}