# Benchmarking PySceneDetect

This page contains the results of benchmarking PySceneDetect's detection accuracy againts public
shot-boundary-detection datasets. Scoring follows the [TRECVID-SBD][trecvid] convention
(greedy 1-to-1 nearest-neighbor matching, with a configurable frame tolerance for hard cuts;
point-in-interval matching for fade transitions; mean absolute frame offset on matched events)
so numbers are comparable to published SBD results.

[trecvid]: https://www-nlpir.nist.gov/projects/tv2007/pastdata/shot_boundary.07.html

Supported datasets:

- [BBC Planet Earth](https://zenodo.org/records/14865504):
  11 long-form broadcast clips; hard cuts only
- [AutoShot](https://drive.google.com/file/d/17diRkLlNUUjHDooXdqFUTXYje2-x4Yt6/view?usp=sharing):
  Short-form web clips; hard cuts only
- [ClipShots](https://github.com/Tangshitao/ClipShots):
  Short-form web clips; hard cuts and typed gradual transitions (fades/dissolves)

## Usage

```bash
# Single detector x single dataset:
python -m benchmark --detector detect-content --dataset BBC
```

Pass `--help` for `--dataset-root`, `--backend`, `--tolerance`, and `--out` options.

### Parameter sweeps

`python -m benchmark.sweep` runs a grid over detector parameters and reports the
top cells by F1 plus the Pareto front across tolerances. One decode is shared by up to
`--workers` parallel detectors via an internal fan-out wrapper, so the cost scales with
`ceil(cells / workers)` decodes per video rather than `cells` decodes.

```bash
python -m benchmark.sweep \
  --detector detect-content --dataset BBC \
  --params "threshold=15:35:1;min_scene_len=0.0:1.0:0.1" \
  --tolerance 0,1 --workers 16 \
  --out sweep-content-bbc.json
```

`--params`: assignments joined by `;`. Each is either `key=v1,v2,v3` (enumerated values) or
`key=start:stop:step` (numeric range, inclusive when `stop` lands on a step). Omitted keys
use the detector's default.

Time-valued kwargs (`min_scene_len`, etc.) accept `TimecodeLike` - integers are
frames, floats are seconds, and strings like `"0.1s"` / `"00:00:00.500"` also work.
Prefer floats so the same sweep is comparable across datasets with different
framerates. Use `--quick N` to limit to the first N samples for iteration; published
numbers should always come from the full corpus.

## Dataset Download

### BBC

```bash
# annotations
wget -O BBC/fixed.zip https://zenodo.org/records/14873790/files/fixed.zip
unzip BBC/fixed.zip -d BBC
rm -rf BBC/fixed.zip

# videos
wget -O BBC/videos.zip https://zenodo.org/records/14873790/files/videos.zip
unzip BBC/videos.zip -d BBC
rm -rf BBC/videos.zip
```

### AutoShot

Download `AutoShot_test.tar.gz` from
[Google Drive](https://drive.google.com/file/d/17diRkLlNUUjHDooXdqFUTXYje2-x4Yt6/view?usp=sharing).

```bash
tar -zxvf AutoShot_test.tar.gz
rm AutoShot_test.tar.gz
```

### ClipShots

ClipShots is gated behind a dataset request form; direct `wget`-style download links are not
published. See [the download instructions](https://github.com/Tangshitao/ClipShots#downloads) to
obtain the annotations and videos. The expected on-disk layout is:

```
ClipShots/
  annotations/{train,test,only_gradual}.json
  video_lists/{train,test,only_gradual}.txt
  videos/*.mp4
```

The loader defaults to the test split (500 videos). The full corpus is ~46 GB.

Set `--dataset-root /path/to/datasets` to override. The default dataset location assumes they are
all placed in the benchmark folder (e.g. `benchmark/BBC`, `benchmark/AutoShot`, `benchmark/ClipShots`).

## Results (defaults)

Generated by `scripts/benchmark_defaults.sh` at `tolerance=0` (frame-exact matching).
Elapsed is mean wall-clock seconds per video.

#### BBC

|      Detector     | Recall | Precision |   F1  | Mean s/video |
|:-----------------:|:------:|:---------:|:-----:|:------------:|
|  AdaptiveDetector |  87.12 |   96.55   | 91.59 |    36.12     |
|  ContentDetector  |  84.70 |   88.77   | 86.69 |    37.02     |
|    HashDetector   |  92.30 |   75.56   | 83.10 |    25.51     |
| HistogramDetector |  89.84 |   72.03   | 79.96 |    22.29     |
| ThresholdDetector |   0.06 |    0.70   |  0.11 |    16.05     |

#### AutoShot

|      Detector     | Recall | Precision |   F1  | Mean s/video |
|:-----------------:|:------:|:---------:|:-----:|:------------:|
|  AdaptiveDetector |  70.59 |   77.46   | 73.86 |     3.52     |
|  ContentDetector  |  63.49 |   76.19   | 69.26 |     4.80     |
|    HashDetector   |  56.48 |   76.11   | 64.84 |     4.14     |
| HistogramDetector |  63.27 |   53.23   | 57.82 |     3.76     |
| ThresholdDetector |   0.75 |   38.64   |  1.47 |     3.28     |

#### ClipShots (hard cuts)

|      Detector     | Recall | Precision |   F1  | Mean s/video |
|:-----------------:|:------:|:---------:|:-----:|:------------:|
|  AdaptiveDetector |  85.97 |   41.25   | 55.75 |     1.81     |
|  ContentDetector  |  81.93 |   42.36   | 55.84 |     2.52     |
|    HashDetector   |  81.34 |   30.14   | 43.98 |     1.04     |
| HistogramDetector |  72.20 |   11.47   | 19.80 |     0.71     |
| ThresholdDetector |   0.08 |    0.58   |  0.14 |     0.64     |

#### ClipShots (fades)

|      Detector     | Recall | Precision |   F1  |
|:-----------------:|:------:|:---------:|:-----:|
|  AdaptiveDetector |  13.65 |   98.12   | 23.96 |
|  ContentDetector  |  26.03 |   98.04   | 41.14 |
|    HashDetector   |  18.77 |   94.53   | 31.33 |
| HistogramDetector |  69.67 |   81.99   | 75.33 |
| ThresholdDetector |   5.69 |   99.24   | 10.77 |

## Parameter sweep results

The tables above use each detector's v0.7 defaults. A grid sweep over the key parameters
scored by hard-cut F1 at 1-frame tolerance, averaged across BBC / AutoShot / ClipShots gives the
best single parameter set for this corpus mix:

|      Detector     | Best mean F1 | Best params                                                | v0.7 default                           |
|:-----------------:|:------------:|:-----------------------------------------------------------|:---------------------------------------|
|  ContentDetector  |     73.4     | threshold=31, min_scene_len=0.6s                           | threshold=27                           |
|  AdaptiveDetector |     76.3     | adaptive_threshold=3.5, window_width=3, min_scene_len=0.6s | adaptive_threshold=3.0, window_width=2 |
|    HashDetector   |     69.8     | threshold=0.35, size=8                                     | threshold=0.395, size=16               |
| HistogramDetector |     66.3     | threshold=0.20, bins=128                                   | threshold=0.05, bins=256               |
| ThresholdDetector |      --      | detects fades, not hard cuts (validation only)             | threshold=12                           |

Full per-dataset breakdowns are in [`SWEEP_REPORT.md`](SWEEP_REPORT.md), and can be generated
with `python -m benchmark.report_sweep`. The full grids (all detectors and datasets) are driven
by `scripts/benchmark_sweep.sh`.

## Citations

### BBC

```
@InProceedings{bbc_dataset,
  author    = {Lorenzo Baraldi and Costantino Grana and Rita Cucchiara},
  title     = {A Deep Siamese Network for Scene Detection in Broadcast Videos},
  booktitle = {Proceedings of the 23rd ACM International Conference on Multimedia},
  year      = {2015},
}
```

### AutoShot

```
@InProceedings{autoshot_dataset,
  author    = {Wentao Zhu and Yufang Huang and Xiufeng Xie and Wenxian Liu and Jincan Deng and Debing Zhang and Zhangyang Wang and Ji Liu},
  title     = {AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  year      = {2023},
}
```

### ClipShots

```
@InProceedings{clipshots_dataset,
  author    = {Shitao Tang and Litong Feng and Zhanghui Kuang and Yimin Chen and Wei Zhang},
  title     = {Fast Video Shot Transition Localization with Deep Structured Models},
  booktitle = {Asian Conference on Computer Vision (ACCV)},
  year      = {2018},
}
```
