Fix psd_array_welch for good data spans shorter than n_per_seg (#13039)

[CedricConday]

Reference issue

Fixes #13039.

What does this implement/fix?

When bad_* annotations split the data into good spans, a span shorter than n_per_seg previously raised from SciPy:

ValueError: noverlap must be less than nperseg.

Following @CarinaFo's review, such spans are now dropped from the PSD estimate with a warning rather than analyzed with a shrunken window — a single Welch window does not fit them, and shrinking the window per-span would mix incompatible estimates. If every good span is shorter than n_per_seg, a clear ValueError is raised pointing the user to lower n_per_seg / n_fft.

Test

test_psd_welch_short_span_dropped covers both the warning-and-drop path (one short span among long ones) and the all-spans-too-short ValueError. Full test_psd.py is green under -W error.

---

Disclosure: I'm an AI engineer and use Claude Code in my workflow. All code here is hand-reviewed and the reasoning is mine; I'm happy to walk through any of it.

[welcome]

Hello! 👋 Thanks for opening your first pull request here! ❤️ We will try to get back to you soon. 🚴

[CarinaFo]

Hi Cedric, thank you for your contribution.

Could you disclose your AI usage according to our adopted policy.

I ran into this same issue in my analysis recently and ended up deciding that dropping short spans was the safer approach, so I would be rather cautious about shrinking noverlap, particularly without warning the user about this behaviour.
I suggest we sync with @drammock on the right approach here (I haven't looked at the "old" behaviour but I think it would be important to know why noverlap < nperseg did not throw a ValueError in older MNE versions.)

[CedricConday]

Hi Carina, thanks for the careful review.

On the AI disclosure — I've added it to the PR description per the policy: I'm an AI engineer and use Claude Code in my workflow; I find, fix, and test, then review and verify everything before opening it under my name. Happy to walk through any part of the reasoning.

On the approach — I agree, silently shrinking noverlap didn't sit right with me either. I've pushed an update that instead drops good-data spans shorter than n_per_seg with a warning, and raises a clear ValueError if every span is too short. Glad that matches where you landed.

Happy to sync with @drammock before this is finalized. On your open question — I don't want to guess at why noverlap < n_per_seg didn't raise in older MNE, so I'll dig into the history and report back here, so we're deciding on the real prior behavior rather than assumptions.

Thanks!

[CedricConday]

Quick follow-up on the history question — I traced it. In 2015 (ca23ab6) _check_nfft silently clamped both: oversized n_fft down to the data length, and n_overlap >= n_fft down to n_fft - 1 — nothing raised. The ValueError for n_overlap >= n_per_seg was added in 2017 with #4003 (the PR that introduced n_per_seg). That same PR kept the segment-length clamp silent — n_per_seg = n if n_per_seg > n — and that line is still on main. So the short-span case never raised: the window just gets quietly shrunk to the span length, which is exactly the behavior here. Only the overlap check ever became strict; the segment-length clamp stayed silent throughout.

[drammock]

So if I'm understanding correctly, the problem is:

• we effectively hard-code the values of n_overlap, n_per_seg, and n_fft that we pass to scipy.signal.spectrogram, by baking them into a functools.partial, here:

  mne-python/mne/time_frequency/psd.py

  Lines 247 to 256 in 31a7326

  	_func = partial(
  	spectrogram,
  	detrend=detrend,
  	noverlap=n_overlap,
  	nperseg=n_per_seg,
  	nfft=n_fft,
  	fs=sfreq,
  	window=window,
  	mode=mode,
  	)

• we do that before we've sorted out which spans of the signal need to be skipped / which spans we're going to analyze (AKA x_splits, here and here)
• so if one of the spans is shorter than n_per_seg, scipy will change n_per_seg internally but will not change n_overlap, and then it will error out a bit further on

I lean toward considering this a SciPy bug: the user passed input that ought to work, and then scipy-internal code changed one value, then chokes because that value is no longer OK relative the other passed value. Would love @larsoner's opinion as to whether upstreaming this report makes sense.

As for what to do about it on our end (mostly repeating what others have said here), we could:

1. error out (current behavior)
2. drop short segments (with a warning)
3. catch this problem before passing a span to scipy.signal.spectrogram and adjust the values of both n_per_seg and n_overlap in the partial (workaround)
4. report upstream and (wait for a) fix; this is a severe enough bug though that even if this is the long term approach I think we should also do (3) in the meantime.

I lean toward (3) or (4 & 3)

[CedricConday]

Thanks @drammock — your root-cause read is exactly right: n_per_seg/n_overlap/n_fft get baked into the functools.partial before spans are split, so a span shorter than n_per_seg makes SciPy clamp nperseg internally but leave noverlap, then error.

I've gone with option 3. Instead of dropping short spans, each span now gets a per-span partial that overrides the baked-in values: nperseg shrunk to the span length and noverlap clamped below it. n_fft is untouched, so every span is zero-padded to the same length and shares one frequency grid — short spans just get coarser spectral resolution, which I surface with a warning. No data is discarded. The all-long path is unchanged (the test_bad_annot_handling equivalence test still matches at rtol=1e-15).

Two things worth your call:

1. Array windows. For string/tuple windows SciPy regenerates the window at the shrunk nperseg, so this is transparent. An explicit ndarray window has a fixed length and can't be shrunk — that combination (array window + a span shorter than n_per_seg) isn't handled here. Prefer I drop-with-warning in just that sub-case, or raise a clearer error?
2. Upstream (option 4). Happy to open a SciPy issue for the clamp-nperseg-but-not-noverlap behavior. Want me to, and link it here?

[drammock]

Happy to open a SciPy issue for the clamp-nperseg-but-not-noverlap behavior. Want me to, and link it here?

Yeah I think it's worth asking them if they consider it bug-ish and want it fixed.

An explicit ndarray window has a fixed length and can't be shrunk

I don't have a clear sense of what users would prefer here (it might vary). drop-segment-plus-warning is a bit more friendly I guess, but would still be catastrophic if a particular dataset was mostly/all short spans... but I guess in either case the remedy is "pass in a shorter custom window array" so maybe that's fine.

[CedricConday]

Pushed ae6fcbf handling the array-window case.

Array/ndarray window + short span — went with the clear-error route. A named/tuple window is regenerated by SciPy at the shrunk nperseg, so those spans stay transparent; but a fixed-length window array can't be shortened, so instead of letting SciPy raise a cryptic length-mismatch, psd_array_welch now raises an actionable ValueError pointing the user to pass a shorter window array or reduce n_per_seg/n_fft. I preferred this over drop-with-warning since silently dropping is catastrophic on a mostly-short dataset (your concern), and the remedy you named — "pass a shorter custom window array" — is exactly what the error tells them to do. Regression test added; full test_psd.py green locally (22/22).

Upstream (option 4) — I'll open a SciPy issue for the clamp-nperseg-but-not-noverlap behavior and link it here.

[CedricConday]

Opened the upstream SciPy issue: scipy/scipy#25608 — clamp-nperseg-but-not-noverlap for short input, with a minimal reproducer. Our per-span workaround here stands regardless of how they decide to handle it upstream.