RequantTESTNET
Requant research

GPU Board Energy, Batching and Tail Latency

Abstract

Across three exact-output kernels, batching reduced board energy by 2.82–4.30% and raised offered-to-delivered p99 to about 8.1 ms. All four held-out latency objectives passed while the kernel energy rankings changed.

Study 15 of the Consumer Blackwell programme. See the research methodology for timing and replay boundaries.

Batching reduced measured board energy by 2.82–4.30% in all 12 accepted kernel pairs on the RTX 5060 Ti. It raised offered-to-delivered p99 to about 8.1 ms. The cohort-1 selector used singletons for the 2 ms objective and batching for the 12 ms objective. All four held-out objectives passed. The selected kernel in each cell used 0.11–0.52% more energy than the held-out best eligible kernel for that objective. The experiment supports a bounded batching decision and provides no consistent winner across the tested kernels and arithmetic implementations.

Workload, energy and boundaries

Each arm runs 1,024 requests whose two 4,096-word unsigned 16-bit vectors have independently computed exact 64-bit dot products. The CUDA integer tree is the original shared-memory control. The integer warp kernel changes its reduction and the FP64 warp kernel changes its multiply-add arithmetic with the same shuffle reduction. The maximum exact result is below 2^44, so the binary64 multiply-add and summation preserve the required integer value. The two new binaries were inspected for shuffle instructions and the FP64 binary for double FMA. The source and binary digests are in the kernel receipt. See NVIDIA's shuffle intrinsic reference.

Each kernel uses identical input and output formats in both arms. Single-request submission competes with a FIFO policy that can dispatch up to eight requests with a fixed 8 ms maximum batching wait. The offered schedule is either steady at 3.25 ms per request or irregular with 2.5 ms within-group spacing and group periods of 22, 28, 24 or 30 ms. Within each pattern, two independent input cohorts have reversed arm order. The four pattern/cohort blocks interleave three variants. The accepted 12 pairs represent 24,576 independently checked deliveries. An earlier separately accepted run of the integer tree adds four paired measurements and 8,192 deliveries. The earlier run is reported separately from the kernel contrast.

Board energy comes from NVML's cumulative GPU energy counter, using within-arm differences across a nominal 4 s active window. Four NVML call intervals bracket the idle and active boundaries. The owner wait, exact output, complete FIFO dispatch, bounded interarrival variation, calibration and source/device identity are checked before using a pair. Idle energy before and after the active interval remains separate in the receipts. The board was shared and the energy includes its idle and background activity. These figures give neither isolated kernel energy nor total system energy.

Matched measurements

p99 uses nearest rank over the 1,024 offered-to-delivered intervals in each arm, including host admission delay. The largest observed admission lateness across the kernel comparison was 0.209 ms. The shared-host epoch allowed bounded arrival variation without assuming identical realized spacing across arms.

Pattern/cohortKernelSingleton (J)Batch (J)Batch savingSingleton p99 (ms)Batch p99 (ms)
steady-1Integer tree108.458105.2692.94%0.0888.089
steady-1Integer warp108.776104.1564.25%0.0838.161
steady-1FP64 warp108.246105.0772.93%0.1368.149
steady-2Integer tree111.192108.0592.82%0.2008.182
steady-2Integer warp111.806107.7313.64%0.1918.184
steady-2FP64 warp111.478107.1723.86%0.1938.192
irregular-1Integer tree110.110106.1903.56%0.0768.080
irregular-1Integer warp108.957104.9493.68%0.2238.174
irregular-1FP64 warp110.113105.7014.01%0.1628.133
irregular-2Integer tree110.598106.8093.43%0.1998.163
irregular-2Integer warp111.005106.7163.86%0.1838.165
irregular-2FP64 warp111.385106.6014.30%0.0788.084

The two kernel changes are useful exact-output controls. Contrasts between integer tree and integer warp, and between integer warp and FP64 warp, changed sign across measured arms or blocks. These are descriptive whole-board comparisons on the shared host. The table does not characterize energy per instruction or infer a general arithmetic precision preference.

Held-out mode selection

For each pattern and p99 objective, choose the eligible mode with the lowest measured energy in cohort 1. Evaluate that exact kernel and batching policy in cohort 2 without choosing again.

PatternObjective (ms)Selected modeHeld-out p99 (ms)Held-out board energy (J)
Steady2FP64 warp / Singleton0.193111.478
Steady12Integer warp / Batch8.184107.731
Irregular2Integer warp / Singleton0.183111.005
Irregular12Integer warp / Batch8.165106.716

All four rows met their stated p99 limits. Recalculating the winner with held-out measurements confirms the same batching choices but changes the kernel choice in every cell. The earlier integer-tree pairs also had lower measured energy under batching in all four cases (3.86–4.73%). They independently selected the same batching policies at 2 and 12 ms.

Rejected attempts and reproduction

Earlier four-pair tightly timed experiments delivered all expected outputs but exceeded the 40 microsecond gate on 12 of 8,192 observed arrivals. A separate prospective paced epoch also remained ineligible. Its rejected cases exposed schedule delay, an NVML read that ended before its scheduled boundary, and a one-bit change to the source file of one cohort. That cohort was restored from the independently checked pre-change archive. The source of the changed bit is unresolved. The later offered-load protocol fixed its timing and input admission before collection. None of the rejected energy differences is reused.

The compact offered-load receipt and kernel receipt retain the measured cells and selector inputs. The offered-load protocol and kernel protocol specify the two accepted series. The acquisition source and independent energy and FIFO trace replays cover exact plans and raw captures. Reproduction requires the native counter feasibility receipt, compatible CUDA/NVML libraries and the exact workload fixtures. Source and output proofs with host-specific identity remain in the private, checksum-pinned archives.

The final kernel archive SHA-256 is e9760b1834707bb9fed217f2535b9d91056dff9aa627d73e24d5833e3a7ce616.

Evidence and source revision

Cite this report

Requant. “GPU Board Energy, Batching and Tail Latency.” Technical report, 19 November 2026. Evidence revision d12bcf1c3270f28fe85828725fb2f2c5c790b986. Canonical report.

This is an organizational technical report, not a peer-reviewed publication.