Skip to content

[Bug] benchmark files (latency, throughput) allow basis/angle selections without cross-framework encoding parity #1252

@viiccwen

Description

@viiccwen

The main cross-framework benchmark scripts accept:

  • --encoding-method amplitude
  • --encoding-method angle
  • --encoding-method basis

But only the Mahout leg actually uses the selected encoding method.

The PennyLane and Qiskit legs are still hard-coded to amplitude-style state preparation, so a run such as:

uv run --project qdp/qdp-python python qdp/qdp-python/benchmark/benchmark_throughput.py \
  --frameworks mahout,pennylane \
  --encoding-method basis

does not produce a fair basis benchmark.

Evidence

  • qdp/qdp-python/benchmark/benchmark_throughput.py
    • Mahout path uses .encoding(encoding_method)
    • PennyLane path uses qml.AmplitudeEmbedding(...)
    • Qiskit path uses qc.initialize(...)
  • qdp/qdp-python/benchmark/benchmark_latency.py
    • Mahout path uses .encoding(encoding_method)
    • PennyLane path uses qml.AmplitudeEmbedding(...)
    • Qiskit paths use qc.initialize(...) / Statevector(...)

Why this matters

This makes the benchmark output potentially misleading:

  • Mahout may run basis or angle
  • competitor frameworks still run amplitude-style preparation
  • the script still prints a single comparison table as if all legs measured the same workload

That is a benchmark validity bug, not just a docs problem.

Proposed fix

Short term:

  • reject non-amplitude cross-framework combinations
  • allow angle / basis only for Mahout-only runs
  • update benchmark docs/examples to state that cross-framework parity currently exists only for amplitude

Longer term:

  • add true angle / basis parity for competitor frameworks where semantics can be matched reliably

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions