Audit remediation: Worker security, content quality, and generated assets

.gitattributes

@@ -1,7 +1,11 @@
-# `src/asset_manifest.py` is generated by `scripts/fingerprint_assets.py`.
+# `src/asset_manifest.py` is generated by `scripts/fingerprint_assets.py`,
+# `src/example_sources_data.py` by `scripts/embed_example_sources.py`, and
+# `src/editorial_registry_data.py` by `scripts/embed_editorial_registry.py`.
 # On merge/rebase, keep our side of the conflict — the post-merge and
-# post-rewrite hooks regenerate the file deterministically afterwards.
+# post-rewrite hooks regenerate these files deterministically afterwards.
 # This works once `scripts/install-git-hooks.sh` has been run locally,
 # which registers `merge.ours.driver = true` and points `core.hooksPath`
 # at `.githooks/`.
 src/asset_manifest.py merge=ours
+src/example_sources_data.py merge=ours
+src/editorial_registry_data.py merge=ours

.githooks/post-merge

@@ -1,9 +1,13 @@
 #!/usr/bin/env bash
-# Regenerate the asset manifest after a merge or pull so the digest
-# reflects the merged tree, not whichever parent won the conflict.
+# Regenerate generated files after a merge or pull so embedded data,
+# asset manifests, fingerprints, and prototype pages reflect the merged
+# tree, not whichever parent won the conflict.
 set -e
 cd "$(git rev-parse --show-toplevel)"
-uv run python scripts/fingerprint_assets.py >/dev/null
-if ! git diff --quiet src/asset_manifest.py public/_headers; then
-  echo "post-merge: asset manifest regenerated; stage and amend if needed"
+make build >/dev/null
+if ! git diff --quiet src/example_sources_data.py src/editorial_registry_data.py src/asset_manifest.py public/_headers public/prototyping; then
+  echo "post-merge: generated files regenerated; stage and amend if needed"
+fi
+if ! make check-social-cards >/dev/null; then
+  echo "post-merge: social-card provenance is stale; run make social-cards before committing"
 fi

.githooks/post-rewrite

@@ -1,9 +1,13 @@
 #!/usr/bin/env bash
-# Regenerate the asset manifest after rebase/amend so the digest matches
-# the rewritten history, not whichever commit happened to win each step.
+# Regenerate generated files after rebase/amend so embedded data,
+# asset manifests, fingerprints, and prototype pages match the rewritten
+# history, not whichever commit happened to win each step.
 set -e
 cd "$(git rev-parse --show-toplevel)"
-uv run python scripts/fingerprint_assets.py >/dev/null
-if ! git diff --quiet src/asset_manifest.py public/_headers; then
-  echo "post-rewrite: asset manifest regenerated; stage and amend if needed"
+make build >/dev/null
+if ! git diff --quiet src/example_sources_data.py src/editorial_registry_data.py src/asset_manifest.py public/_headers public/prototyping; then
+  echo "post-rewrite: generated files regenerated; stage and amend if needed"
+fi
+if ! make check-social-cards >/dev/null; then
+  echo "post-rewrite: social-card provenance is stale; run make social-cards before committing"
 fi

.githooks/pre-commit

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Generated artifacts are committed. Rebuild and refuse the commit when the
+# index would leave embedded data, fingerprints, prototypes, or card provenance
+# stale. CI enforces the same contract for contributors without this local hook.
+set -euo pipefail
+cd "$(git rev-parse --show-toplevel)"
+make check-generated

.github/pull_request_template.md

@@ -1,14 +1,30 @@
 ## Summary
 
-<!-- What changed? -->
+<!-- What changed and why? Link the issue or explain the concrete motivation. -->
 
 ## Verification
 
+<!-- Check only evidence actually run. Explain any applicable unchecked item above. -->
+
 - [ ] `make verify`
-- [ ] `scripts/check_example_migration_parity.py`
 - [ ] `scripts/format_examples.py --check`
 - [ ] `make verify-python-version VERSION=3.13`
 - [ ] `git diff --check`
+- [ ] Added or updated regression tests and verified they fail when the fix is reverted
+- [ ] Manual verification: named route/flow, environment, and observed result in Summary
+
+## Visual evidence (required for UI changes)
+
+<!-- Delete this section only when the PR has no visual impact. Caption each
+image/recording with the observable change and record how it was captured. -->
+
+**Before — [specific observable behavior]:**
+<!-- image or recording -->
+
+**After — [specific observable behavior]:**
+<!-- image or recording -->
+
+**Reproduction:** `[command]` (base: `[sha]`; head: `[sha]`; viewport/state: `[details]`)
 
 ## Example changes
 

.github/workflows/preview.yml

@@ -38,21 +38,29 @@ jobs:
         env:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
-        run: npx --yes wrangler whoami
+        run: npx --yes wrangler@4.90.0 whoami
       - name: Sync Python Workers vendor
         run: uv run pywrangler sync
+      # Workflow inputs are passed through env vars rather than interpolated
+      # into the script, so a crafted input cannot inject shell commands into
+      # steps that hold the Cloudflare API token.
       - name: Upload Cloudflare Preview
         env:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
+          PREVIEW_NAME: ${{ inputs.name || 'preview' }}
+          PREVIEW_MESSAGE: ${{ github.sha }}
         run: |
           set -x
           uv run pywrangler preview \
-            --name "${{ inputs.name || 'preview' }}" \
-            --message "${{ github.sha }}" \
+            --name "$PREVIEW_NAME" \
+            --message "$PREVIEW_MESSAGE" \
             --json
       - name: Smoke test deployed Preview
-        run: scripts/smoke_deployment.py "https://${{ inputs.name || 'preview' }}-pythonbyexample.adewale-883.workers.dev"
+        env:
+          PREVIEW_NAME: ${{ inputs.name || 'preview' }}
+          PBE_SMOKE_BYPASS_SECRET: ${{ secrets.PBE_SMOKE_BYPASS_SECRET }}
+        run: scripts/smoke_deployment.py "https://${PREVIEW_NAME}-pythonbyexample.adewale-883.workers.dev"
       - name: Dump wrangler logs on failure
         if: failure()
         run: |

.github/workflows/verify.yml

@@ -2,8 +2,17 @@ name: Verify
 
 on:
   push:
+    branches:
+      - main
   pull_request:
 
+permissions:
+  contents: read
+
+concurrency:
+  group: verify-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   verify:
     runs-on: ubuntu-latest
@@ -42,7 +51,6 @@ jobs:
           CHROME_PATH: /usr/bin/google-chrome
         run: |
           make verify
-          scripts/check_example_migration_parity.py
           scripts/format_examples.py --check
           make verify-python-version VERSION=3.13
           git diff --check

CHANGELOG.md

@@ -18,11 +18,16 @@ The format is inspired by [Keep a Changelog](https://keepachangelog.com/en/1.1.0
 - Skip-to-content link on every page.
 - Per-example social-card images composed from each example's marginalia figure (`make social-cards`), referenced by `og:image`/`twitter:card` on home and example pages and checked by the SEO linter.
 - Learner-behavior report (`scripts/learner_report.py` + `docs/learner-analytics.md`) aggregating exported Worker wide events into most-read pages, most-run examples with edited/error shares and execution percentiles, journey traffic, and missing-example 404s.
+- Content gates for program/cell parity, prose duplication, and internal inline links, wired into `make quality-checks`.
+- Shared inline-prose rendering, figure XML/caption contracts, regenerated figure-review prototypes, and behavioral tests for the Turnstile and ASGI paths.
 
 ### Changed
 
 - Journeys now reference every example: `iterator-vs-iterable`, `classmethods-and-staticmethods`, `bound-and-unbound-methods`, `abstract-base-classes`, and `structured-data-shapes` joined their natural sections, with journey-outcome support lists updated.
 - Journey meta descriptions no longer claim gap placeholders; all previously declared gaps are filled.
+- Python tooling is pinned to 3.13 through `uv`; CI and preview workflows are hardened, generated-file recovery retries from a clean worktree, and quality gates enforce live registry/score invariants.
+- Worker runtime hardening adds body limits, cache-key normalization, constant-time smoke-bypass comparison, fail-closed Turnstile modes, latin-1 ASGI headers, and once-per-isolate lifespan startup.
+- Content corrections across 29 example pages improve Python accuracy, executable demonstrations, sandbox-boundary framing, and link/prose quality.
 
 ## 2026-05-16
 

CONTRIBUTING.md

@@ -49,7 +49,6 @@ Then run:
 
 ```bash
 make verify
-scripts/check_example_migration_parity.py
 scripts/format_examples.py --check
 make verify-python-version VERSION=3.13
 git diff --check
@@ -64,17 +63,40 @@ make check-generated
 
 ## Quality checks
 
-Five scripts enforce the catalog-level rules from `docs/example-quality-rubric.md`. Run them together with `make quality-checks`.
+These scripts enforce the catalog-level rules from `docs/example-quality-rubric.md`. Run them together with `make quality-checks`.
 
 | Script | What it gates |
 | --- | --- |
-| `scripts/check_registry_integrity.py` | Every owner slug in `docs/quality-registries.toml` exists in `manifest.toml`; tokens are present. |
-| `scripts/check_confusable_pairs.py` | Each confusable pair's owning page contains every token that signals the contrast. |
-| `scripts/check_broad_surface_tours.py` | Each broad-title page either covers every required form or sets `scope_first_pass = true` with `see_also` links to focused neighbors. |
+| `scripts/check_registry_integrity.py` | Every owner slug in `docs/quality-registries.toml` exists in `manifest.toml`; tokens are present; each `paired_pages` pair is discoverable through `see_also` and has registry-named cell tokens demonstrated inside a teaching cell. |
+| `scripts/check_confusable_pairs.py` | Each confusable pair's owning page contains every token inside teaching cells; a token shadowed inside a longer sibling token (plain `def` inside `async def`) does not count. |
+| `scripts/check_broad_surface_tours.py` | Each broad-title page either covers every required form or sets `scope_first_pass = true` with registry-named `focused_neighbors` linked through `see_also`. |
 | `scripts/check_footgun_coverage.py` | Each canonical Python footgun has a page that contains both broken-form and fixed-form tokens. |
-| `scripts/check_notes_supported.py` | Every `:::note` bullet shares at least one keyword with the page body, so notes cannot assert behavior the page never demonstrates. |
+| `scripts/check_notes_supported.py` | Every `:::note` bullet (across all note blocks) shares at least one keyword with the page body outside the notes, so notes cannot assert behavior the page never demonstrates. |
+| `scripts/check_program_covers_cells.py` | Every executable cell shares substantive code with the `:::program` block, so the editor reproduces what the cells teach; `standalone_cells = true` opts out visibly. |
+| `scripts/check_prose_duplication.py` | No verbatim repeated paragraphs, no cell prose copied from the intro, no duplicate note bullets. |
+| `scripts/check_inline_links.py` | Inline `[text](target)` links in prose resolve to real `/examples` or `/journeys` pages. |
+| `scripts/score_example_criteria.py` | Heuristic criterion scores per example; fails when a curated score exceeds the heuristic by more than the delta bound, so the score registry cannot inflate without the page changing. |
+| `scripts/check_quality_scores.py` | Curated scores meet the 9.0 target / 8.5 hard minimum, waivers carry future expiry dates and are dropped when stale, and the journey-section average stays above its floor. |
+| `scripts/check_no_figure_rationales.py` | No-figure opt-outs carry a reason and an unexpired `review_after` date. |
+| `scripts/check_journey_outcomes.py` | Every journey section declares learner outcomes. |
+| `scripts/audit_example_graph.py --check` | The `see_also` graph has no broken targets, self-links, over-linked pages, or orphaned examples. |
 
-The single source of truth for the registries is `docs/quality-registries.toml`. Add a new pair, broad tour, or footgun there, then update the owning page so the tokens appear in cells or prose.
+The single source of truth for the registries is `docs/quality-registries.toml`. Add confusable pairs, broad tours, footguns, journey metadata, figure attachments, captions, and curated scores there, then update the owning page so the tokens appear in cells or prose.
+
+## Adding a new example end to end
+
+1. Write `src/example_sources/<slug>.md` (frontmatter, intro prose, one `:::program`, cells, notes) and add the slug to `manifest.toml`'s `order`.
+2. Add `see_also` links in both directions — `scripts/audit_example_graph.py --check` fails orphaned pages.
+3. Score the page against `docs/example-quality-rubric.md` and add the `[[example_quality_scores]]` entry to `docs/quality-registries.toml`.
+4. Attach a figure by adding `[[figure_attachments]]` and `[[example_figure_scores]]` entries to `docs/quality-registries.toml` (or add a `no_figure_rationales` entry with a review date); keep only new paint functions in `src/marginalia.py`. Review the result on `public/prototyping/marginalia-gestalt.html`, which shows the production caption under every figure.
+5. Run `make build`, `make verify-examples`, `make quality-checks`, and `scripts/format_examples.py --check`.
+6. Run the full `make verify` with a local Worker running, then commit — including the regenerated `src/example_sources_data.py`, `src/editorial_registry_data.py`, `src/asset_manifest.py`, and fingerprinted assets.
+
+## Secrets and deploy configuration
+
+Deployment uses repository secrets: `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` (Preview workflow), plus optional `PBE_SMOKE_BYPASS_SECRET` so deploy smoke tests can run edited-code POSTs past the Turnstile challenge. Runtime Worker secrets (`TURNSTILE_SECRET_KEY`, `TURNSTILE_CLEARANCE_SECRET`, `PBE_SMOKE_BYPASS_SECRET`) are managed with `wrangler secret put`; see `docs/turnstile-runner-protection-spec.md`.
+
+Generated output is prevented from drifting before merge: install the local hooks with `scripts/install-git-hooks.sh`, and keep `main` protected so pull requests require the `verify` status check to pass against the current base. CI enforces the same `make check-generated` contract for contributors without local hooks.
 
 ## Style expectations