Night campaign with the interrupt sweep, mutants and wave 4 (2026-11-12/13)

Stress campaign on all six boards (lab/tools/night-all.sh, each board with its own copy of the repository at commit 25c64d2), 18:45 to about 07:00 UTC (the nRF5340 until 08:45: it was waiting in a run that was too slow, see below). New since the campaign of 2026-11-11/12:

No errors in any real library in any valid run, on any board: 422 valid runs of 427. The errors are where they are expected: the two variants their own projects call unsafe (wizard97 as shipped, Arduino RingBufferN as shipped), on every board and in every mode, and on the Pico 2 in SMP mode the single-core designs whose interrupt lock does nothing against the other core (ETL queue_spsc_isr and queue_spsc_locked, wizard97 with PRIMASK macros, Arduino SafeRingBufferN).

Per board

Board Runs Valid Invalid (control) Did not finish Too slow (time limit) Build failed Retries Power cycles Errors in real libraries
nrf5340dk 30 26 0 4 0 0 2 2 0
nucleo-g0b1re 38 38 0 0 0 0 0 0 0
nucleo-g474re 113 113 0 0 0 0 0 0 0
nucleo-h723zg 62 62 0 0 0 0 0 0 0
sam-d20-xpro 24 23 1 0 0 0 0 0 0
pico2 160 160 0 0 0 0 0 0 0

The four nRF5340 runs under 'Did not finish' were too slow, not hung (see below); the campaign's run.sh could not yet tell the two apart.

Valid runs per configuration (valid / started)

Configuration nrf5340dk nucleo-g0b1re nucleo-g474re nucleo-h723zg sam-d20-xpro pico2
single p0 cache on flash 2/2 3/3 7/7 4/4 2/2 5/5
block p0 cache on flash 2/2 3/3 7/7 4/4 3/3 5/5
single p2 cache on flash 2/2 3/3 7/7 4/4 3/3 10/10
sweep cache off flash 2/2 3/3 7/7 4/4 2/2 -
sweep cache on flash 2/2 2/2 7/7 4/4 2/2 20/20
block p1 cache on flash 2/2 2/2 7/7 4/4 3/3 5/5
single p2 cache off flash 2/2 2/2 7/7 4/4 2/2 -
sweep-block cache on flash 2/2 2/2 7/7 4/4 2/2 18/18
single p2 cache on sram 2/2 2/2 7/7 4/4 - 5/5
block p0 cache on sram 2/2 2/2 7/7 4/4 - 5/5
sweep-block cache off flash 0/4 2/2 7/7 4/4 1/1 -
block p1 cache on sram 1/1 2/2 6/6 3/3 - 5/5
dma p0 cache on flash 1/1 2/2 6/6 3/3 0/1 9/9
dma p2 cache on flash 1/1 2/2 6/6 3/3 1/1 9/9
dma p1 cache on sram 1/1 2/2 6/6 3/3 - 9/9
sweep-dma cache on flash 1/1 2/2 6/6 3/3 1/1 9/9
sweep-block cache on sram 1/1 2/2 6/6 3/3 - 9/9
dma p1 cache on flash - - - - 1/1 -
smp p0 cache on flash - - - - - 5/5
smp-block p0 cache on flash - - - - - 5/5
smp p2 cache on flash - - - - - 10/10
smp-block p1 cache on flash - - - - - 5/5
smp p2 cache on sram - - - - - 4/4
smp-block p0 cache on sram - - - - - 4/4
smp-block p1 cache on sram - - - - - 4/4

Bold: at least one run of that configuration was invalid or did not finish.

Errors per library (valid runs)

Library nrf5340dk nucleo-g0b1re nucleo-g474re nucleo-h723zg sam-d20-xpro pico2
Arduino RingBufferN as shipped (C++) (not claimed ISR-safe) 46,353 60,644 194,811 138,764 40,211 152,281
Arduino RingBufferN as shipped (C++) [SMP, single-core design] (not claimed ISR-safe) 0 0 0 0 0 90,294
Arduino RingBufferN as shipped (C++) [sweep] (not claimed ISR-safe) 471,808 1,550,967 3,980,265 1,715,698 771,130 3,534,429
Arduino SafeRingBufferN (C++) [SMP, single-core design] 0 0 0 0 0 81,765
ETL queue_spsc_isr (C++) [SMP, single-core design] 0 0 0 0 0 102,031
ETL queue_spsc_locked (C++) [SMP, single-core design] 0 0 0 0 0 90,123
wizard97 + PRIMASK macros (C++) [SMP, single-core design] 0 0 0 0 0 840,538
wizard97 as shipped (C++) (not claimed ISR-safe) 217,944 348,946 580,807 139,256 297,213 209,253
wizard97 as shipped (C++) [SMP, single-core design] (not claimed ISR-safe) 0 0 0 0 0 1,378,431
wizard97 as shipped (C++) [sweep] (not claimed ISR-safe) 474,486 1,457,091 2,821,526 940,085 812,417 2,868,307

Rows without a tag count as errors in real libraries, except 'single-core design' rows (SMP runs of interrupt-only designs).

Interrupt sweep (valid sweep runs)

Board Runs Trials Interrupt inside the call Library functions with instructions not interrupted
nrf5340dk 8 222,648,964 77.6% 260
nucleo-g0b1re 13 454,578,120 73.9% 138
nucleo-g474re 40 1,247,266,170 72.7% 304
nucleo-h723zg 22 524,629,714 73.8% 349
sam-d20-xpro 8 249,306,478 75.9% 125
pico2 56 982,911,563 70.0% 321

'Not interrupted' includes paths the sweep never executes (see README, Interrupt sweep); the per-run lists are in each record under sweep.functions.

Mutants: which mode finds which defect

Valid runs in which each mutant was caught, interrupt→main / main→interrupt direction, all boards together.

Mutant block single smp smp-block sweep sweep-block
drain re-reads head 81 / 0 of 84 39 / 0 of 89 16 / 19 of 19 18 / 18 of 18 55 / 0 of 55 70 / 0 of 70
full test misses the wrap 84 / 67 of 84 89 / 88 of 89 19 / 19 of 19 18 / 18 of 18 55 / 55 of 55 70 / 70 of 70
shared count, no lock 84 / 84 of 84 89 / 89 of 89 19 / 19 of 19 18 / 18 of 18 55 / 55 of 55 70 / 70 of 70
shared count, pop without lock 84 / 0 of 84 89 / 0 of 89 19 / 19 of 19 18 / 18 of 18 55 / 0 of 55 70 / 0 of 70

Runs that did not count

Time per run

sweep-block at 256 start positions is by far the longest entry of the rotation: 10 to 14 minutes on the fast boards, 38 to 88 on the slow ones (cache on, flash; up to 95 with the cache off), against 2 to 18 for a random block run. The slow boards got 23 to 38 runs in the night, the fast ones 62 to 160. Open: fewer start positions for sweep-block, on all boards alike.

After the campaign: board check of the work since 25c64d2

Short runs on all six boards at 76c9581 (20 000 bytes or 16 start positions; records in ~/lab-runs/val-20261113/): the six stress modes, the benchmark, and the multi-producer/multi-consumer stress (MPSC and SPMC), plus the SMP modes on the Pico 2. All 61 runs valid, no errors in real libraries in their documented mode. Records name their source commit.

How many start positions: 16 to 256 (NUCLEO-G474RE, sweep-block)

Question: the sweep runs with 256 start positions take up to 1.7 h on the slowest board; would fewer do? Measured on one board at 4e8e856: 16, 32, 64, 128 and 256 start positions, three seed pairs each, cache on, code in flash, -O2 (15 runs, all valid, 0 errors in the real libraries; records in ~/lab-runs/reps-20261113/nucleo-g474re).

Start positions Time per run Mutants, wizard97 as shipped, Arduino RingBufferN as shipped RT-Thread rt_ringbuffer, no lock: errors isr→main in the three runs
16 2.4 min caught in 3 of 3 3 / 3 / 6
32 2.4 min caught in 3 of 3 0 / 0 / 0
64 3.1 min caught in 3 of 3 6 / 0 / 0
128 5.9 min caught in 3 of 3 39 / 42 / 63
256 11.2 min caught in 3 of 3 84 / 75 / 78

Decision (owner, 2026-11-13): 256 start positions stay, the same on every board; the default time limit of run.sh goes from 300 to 600 polls (at least 2.5 h), so the slowest configuration (about 1.7 h) fits.