Skip to content

ngmix: give the PSF observation a finite weight map (#749)#779

Open
cailmdaley wants to merge 2 commits into
refactor/psf-column-grammarfrom
fix/psf-obs-weight
Open

ngmix: give the PSF observation a finite weight map (#749)#779
cailmdaley wants to merge 2 commits into
refactor/psf-column-grammarfrom
fix/psf-obs-weight

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Closes #749

What

make_ngmix_observation built the PSF observation weightless
(psf_obs = Observation(psf, jacobian=psf_jacob)), while the galaxy
observation right below it carries a full weight map. With no weight, ngmix
defaults to unit weight on the unit-flux PSF stamp, so the galaxy
GPriorBA(0.4) prior handed to the PSF fitter swamps the (tiny) likelihood and
the recovered PSF shape collapses toward the prior (zero ellipticity).

This PR gives the PSF observation a finite flat weight map
psf_wt = 1/PSF_NOISE**2 (PSF_NOISE = 1e-5, the esheldon/aguinot value
Axel's #749 reproduction used) — the PSF analogue of the galaxy weight built
just below. Weight only: PSFEx/MCCD models already carry a little noise, so no
explicit stamp-noise injection is added (Axel's guidance on #749). The galaxy
prior is left in place (option B from the thread) — once a real weight is
present the likelihood reasserts and the prior no longer bites.

Closes #749

One root, two issues

The single weightless line was the root of both #749 (the galaxy shape prior
prior-dominates the PSF fit, collapsing the recovered ellipticity) and the now
-retired #774 (the small finite-weight noise budget of the pre-v2.0
esheldon/aguinot code was dropped in the rewrite). They are the same bug from
two angles; the finite 1/psf_noise**2 weight is the substance of the old
noise-injection, so adding the weight resolves both. #774 carried no separate
deliverable and has been retired as redundant.

Evidence — end to end through do_ngmix_metacal / average_original_psf

A sheared-PSF stamp (psf_shear = (0.05, -0.03), HSM moment truth
g = [+0.0477, -0.0286]) pushed through the real production path; the
*_psf_orig column is the original-PSF fit:

build g_psf_orig frac recovered
weightless (bug) [+0.0001, -0.0001] 0.00
weighted (this PR) [+0.0477, -0.0286] 1.00

g_psf_reconv ≈ 0 in both builds — the reconvolution kernel is round by
construction, independent of this weight (and is the column the pre-#761
g1_psfo_ngmix name was mis-sourcing).

  • Calibration is provably untouched. Every metacal-type shear
    (noshear/1p/1m/2p/2m) is bit-for-bit identical (Δ = 0) between the
    unit-weight (pre-fix) and weighted builds, and the fit is flat across
    PSF_NOISE ∈ {1e-4, 1e-5, 1e-6}. Metacal fits this same psf_obs with a
    prior-free AdmomFitter (MetacalFitGaussPSF._do_psf_fit), which is
    insensitive to a flat weight's absolute scale — so the diagnostic weight
    never reaches calibration. It is the galaxy prior on the diagnostic fit
    (average_original_psf), not weight-independence, that makes *_psf_orig
    sensitive while the calibration is not.
  • Weight-only vs weight+noise (ngmix: restore (or justify dropping) the PSF noise-injection / PSF-observation weight #774's question). Injecting explicit
    PSF-stamp noise (σ = 1e-6, 1e-5) on top of the weight moves g_psf, T_psf,
    and R by ≲ 1e-6 — explicit injection is justifiably dropped.

The faint-end reduced-χ² / star-response coupling #774 also speculated about is
tracked separately under the χ²-anomaly investigation (#769), not here.

Tests

tests/module/test_ngmix_weight_validation.py gains three guards: a structural
check that psf_obs carries the finite weight (never unit weight), the #749
end-to-end recovery contrast through average_original_psf (weighted recovers
to 1e-3, weightless collapses below 10%), and the bit-for-bit metacal-shear
invariance.

— Claude on behalf of Cail

@cailmdaley cailmdaley force-pushed the fix/psf-obs-weight branch from e06497d to 339fa5a Compare June 30, 2026 22:10
@cailmdaley cailmdaley changed the base branch from ngmix_v2.0 to refactor/psf-column-grammar June 30, 2026 22:10
@cailmdaley cailmdaley changed the title ngmix: give the PSF observation a finite weight map (#749, #774) ngmix: give the PSF observation a finite weight map (#749) Jun 30, 2026
cailmdaley and others added 2 commits July 1, 2026 00:16
make_ngmix_observation built psf_obs weightless, so ngmix defaulted to
unit weight on the unit-flux PSF stamp and the galaxy GPriorBA(0.4) prior
handed to the diagnostic PSF fitter swamped the likelihood. The recovered
PSF shape collapsed toward the prior (zero ellipticity), corrupting the
*_psf_orig diagnostic columns (#749), and the small finite-weight noise
budget present in the pre-v2.0 esheldon/aguinot code was lost in the
rewrite (the now-retired #774 — the same bug from the noise-budget angle).

Build psf_obs with a flat weight = 1/PSF_NOISE**2 (PSF_NOISE = 1e-5, the
esheldon/aguinot value Axel's reproduction used), forced float for parity
with the galaxy weight — the PSF analogue of the galaxy weight built just
below. Weight only; PSFEx/MCCD models already carry a little noise, so no
explicit stamp-noise injection is needed (Axel's guidance on #749). The
exact value is non-critical: validated flat across 1e-4..1e-6 on the twin.

Calibration is untouched: metacal fits this same psf_obs with a prior-free
AdmomFitter, which is insensitive to a flat weight's absolute scale, so the
calibrated shear is invariant (verified bit-for-bit on the twin).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017KQn6mXoL4XMgscZDsaTkB
Three assertions in test_ngmix_weight_validation.py, on the real production
path now that this sits on the column grammar (#761):

- structural: make_ngmix_observation builds psf_obs with the finite
  1/PSF_NOISE**2 weight, never unit weight;
- recovery (#749 done-condition): through do_ngmix_metacal /
  average_original_psf, the *_psf_orig column recovers the injected PSF moment
  ellipticity to 1e-3 with the weight, and collapses below 10% without it (the
  contrast that proves the weight is load-bearing for the diagnostic columns);
- invariance: every metacal-type shear is bit-for-bit identical between the
  unit-weight (pre-fix) and shipped PSF_NOISE builds (metacal's prior-free
  AdmomFitter is insensitive to the flat weight's scale).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017KQn6mXoL4XMgscZDsaTkB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] ngmix module passes the galaxy shape prior to the PSF fit, which is prior-dominated

1 participant