DP-Bench_benchmarks.md

DP-Bench Benchmarks

DP-Bench on HuggingFace

Create DPBench evaluation datasets:

# Make the ground-truth
docling-eval create-gt --benchmark DPBench --output-dir ./benchmarks/DPBench-gt/ 

# Make predictions for different modalities.
docling-eval create-eval \
  --benchmark DPBench \
  --gt-dir ./benchmarks/DPBench-gt/gt_dataset/ \
  --output-dir ./benchmarks/DPBench-e2e/ \
  --prediction-provider Docling # use full-document predictions from docling
  
docling-eval create-eval \
  --benchmark DPBench \
  --gt-dir ./benchmarks/DPBench-gt/gt_dataset/ \
  --output-dir ./benchmarks/DPBench-tables/ \
  --prediction-provider TableFormer # use tableformer predictions only

Layout Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality layout \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-e2e/ 

Layout evaluation json

Visualize the report:

docling-eval visualize \
  --modality layout \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-e2e/ 

mAP[0.5:0.95] report

TableFormer Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality table_structure \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-tables/ 

Visualize the report:

Tableformer evaluation json

Visualize the report:

docling-eval visualize \
  --modality table_structure \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-tables/ 

TEDS struct only report

TEDS struct with text report

Reading order Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality reading_order \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-e2e/ 

Reading order json

Visualize the report:

docling-eval visualize \
  --modality reading_order \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-e2e/ 

ARD report

Weighted ARD report

Markdown text Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality markdown_text \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-e2e/ 

Markdown text json

Visualize the report:

docling-eval visualize \
  --modality markdown_text \
  --benchmark DPBench \
  --output-dir ./benchmarks/DPBench-e2e/ 

Markdown text report