A European Benchmarking Framework for hybrid quantum-classical computing

Expected Outcome

• Enhanced evidence-based decision making through comprehensive system comparison that improves the procurement process for quantum-accelerated supercomputers (hybrid HPC-QC). This will enable more informed choices regarding the acquisition of new systems and upgrades of existing ones
• Overall improved operation and fine-tuning of hybrid HPC-QC systems leading to improved performance, throughput and energy optimization, and improved end-user experience
• Enhancing the competitiveness of EU quantum suppliers through transparent benchmarking of industrially relevant workflows
• A unified, extensible and well-documented benchmarking framework to easily accommodate new, community-contributed benchmarks with common standards, versioning and control
• A well-maintained and continuously updated benchmarking suite for QC- and hybrid HPC-QC workflows with open-source reference implementations for every benchmark, SDK-agnostic specifications and sample datasets
• Active contributions to European and global standardisation on QC-, and hybrid HPC-QC benchmarking, certification and sustainability via standardisation bodies such as IEEE P7130/P7131, CEN-CENELEC JTC22-WG3, IEC/ISO JTC3 etc.

Scope

A.) Development of a benchmarking framework addressing the following aspects:

• offer a fine grained and fair comparison methodology among different systems and integration approaches, i.e. all benchmarks, benchmark run rules[1] and benchmark submission rules must be designed to ensure reproducibility, repeatability and replicability of metrics on the same system, ensuring fairness and comparability of metrics across different systems
• define and continuously update a precise and relevant performance metrics catalogue
• standardise all benchmarking input- and output formats
• collect and report all benchmarking results while offering statistically sound result analyses
• ensure that all benchmarks are executable on the respective target environment(s)
• offer a standardized structured workflow capturing and streamlining the entire benchmarking process
• offer a standardised repository with transparent version control
• provide at least one reference implementation based on an EU SDK for each benchmark
• is of production-quality and ready to assess all of EuroHPC’s QC- and hybrid HPC-QC systems
• provide all required templates with relevant input data to properly execute the benchmarking suite on every EuroHPC system.

The benchmarking framework along with its workflows will be realised in a software implementation that offers to the end-user a dynamic workspace for the entire workflow.

B.) Establishing a comprehensive QC- and hybrid HPC-QC benchmarking suite utilizing the framework developed in the first objective. This benchmarking suite, with its associated performance metrics, will be designed to measure and assess the performance of QC- and hybrid HPC-QC systems.

The envisioned benchmarking suite is expected to:

• be generally hardware agnostic, enabling cross-device verification (e.g. via entangled-state comparisons, or distributed inner-product tests)
• provide thorough documentation for developers and end-users
• develop novel hybrid HPC-QC benchmarks within the following three key areas:
• Throughput: capturing true end-to-end performance including queue-to-result latency, provisioning delays and data movement overhead
• Latency: queue-to-solution time, end-to-end (i.e. submission-to-result)
• Energy: per-shot energy, cryogenic overhead, control-electronics draw, classic-compute share, energy-to-solution (incl. all contributions, i.e. quantum-core, cryogenic, control electronics, classical compute)
• catalogue relevant, well-established QC benchmarks from across the entire software stack, e.g.:
• Component level/ Gate level: gate fidelities, connectivity, energy-state relaxation time, coherence (T1/T2), randomized benchmarking (RB), cross-talk RB, error-per-layered-gate (EPLG), logical-error rate, logical error scaling (Λ)
• System level: Clifford-volume, cross-entropy (XEB), CLOPS/ rQOPS, MegaQuOp, GHZ-state fidelity (single shot read-out)
• Software/ Compiler level: error-syndrome mapping, decoding-throughput metrics, quantum compilation volumetric benchmarks
• Application layer: VQE, QAOA, MaxCut, Q-Score, quantum-enhanced ML, Monte Carlo sampling, workload-specific benchmarks, quantum application score
• support of both noisy-intermediate-scale-quantum (NISQ) and fault-tolerant-quantum-computing (FTQC) regimes (e.g. logical-error rates, logical-clock rate, decoding throughput)
• ensure that each benchmark produces at least one metric
• define reliable and common metrics to compare different qubit modalities, as well as the depth of the HPC-QC integration based on pre-defined criteria (e.g. efficiency)
• ensure the scalability of each benchmark by identifying relevant scale parameters

Proposals should provide a thorough justification for the selection of each benchmark and performance metric, clearly explaining how they align with the specific requirements and priorities of the European HPC-QC landscape. The inclusion or integration of existing benchmarks under the umbrella of this initiative is clearly foreseen, provided there are prior agreements with the benchmark owners and compatibility with licensing conditions.

The proposal must also outline a clear IP plan and licensing strategy under the European Union Public Licence (EUPL-1.2) to safeguard openness and promoting European industrial uptake enabling first exploitation within EU- and Participating States.

The project will also propose and maintain a detailed strategic development roadmap for the action, which:

1. anticipates future developments in QC and hybrid HPC-QC, including emerging technologies
2. foresees the hardware agnostic and hardware inclusive support of hybrid HPC-QC systems

The consortium will actively engage with industry and research communities through workshops, working groups, and feedback loops to receive continuous feedback ensuring that all benchmarks are relevant and up to date.

Requirements:

1. The proposal shall take into consideration the state of the art of development quantum computing benchmarking
2. Define a mechanism to monitor the benchmarking framework and pool appropriate existing benchmark suites, relevant for architectures of all participating HPC-QC centres for deployment in a common data repository
3. The benchmarking framework and the encompassing benchmarking suite will be made available to the user communities under the European Union Public Licence (EUPL).

[1] run rules define required and forbidden hardware, software, optimization, tuning, and procedures.