# Evaluation — Cubic-Hermite FEM (Euler–Bernoulli Beam) Benchmark Run: ```bash python grade.py # full grading python grade.py --budget 900 # allow a slower solver python grade.py --strict # refuse to certify anything inconclusive python grade.py --no-exec # diagnostics only -> UNGRADED, no score ``` The grader is self-contained: it knows the manufactured exact solution, recomputes it analytically, recomputes every order of convergence from the submitted arrays, **and re-runs the submitted solver**. It does not trust `results.json`, and it does not trust `reference_arrays.npz` either — both must agree with each other *and* with what `fem_solver.py` actually produces when executed. ## Score behavior | Score | Condition | |-------|-----------| | **1.0** | Tier 1, Tier 2, and Tier 3 all pass. | | **0.5** | Tier 1, Tier 2 and all provenance checks pass, but Tier 3 is missing or fails. | | **0.0** | Required files missing (including `fem_solver.py`), array schema violations, `results.json` inconsistent with the arrays, wrong mesh sequence, non-finite or non-positive errors, a disallowed dependency, a solver that cannot reproduce its own submitted numbers or returns a nonconforming grid, or Tier 1 / Tier 2 fail. | | *(none)* | **`--no-exec`**: no provenance established, so the run reports `RESULT: UNGRADED` with no `SCORE:` line and exit code 2. Diagnostics only. | ## Visible benchmark thresholds Unlike the sibling ADI task, this specification **fixes the method completely** — `C¹` cubic-Hermite elements, a prescribed manufactured load, a prescribed mesh sequence, a prescribed dense evaluation grid. There is one correct answer, so a band around the gold reference's dense errors is a statement about the *problem*, not a preference among several admissible schemes. The bands are deliberately tight because the signature failure mode — a dense reconstruction that omits the Hermite slope DOFs — yields *perfect nodal values* and an `O(h)` dense error, and only the dense band plus the dense order check can see it. - **Tier 1** (assembly + load + reconstruction): dense error `e*_n` at `n = 32, 64, 128` within **±25 %** of the gold reference (`5.402330e-07`, `3.459441e-08`, `2.188513e-09`), and the two coarse dense orders in `[3.8, 4.2]`. - **Tier 2** (the two error measures, the core check), all of: - *order 4:* at least two of the three clean-regime dense orders (`n = 32 → 256`) in `[3.8, 4.2]`; - *nodal superconvergence:* `e_n(32) < 1e-8` **and** `< e*_n(32)/100` — the nodal values are exact up to round-off, far below the dense (field) error; - *round-off floor:* `e*_n(2048) > 10 · e*_n(256)` — refining past the clean regime stops helping, because the fourth-order operator conditions like `n⁴`. - **Tier 3** (the `n = 256` artifact): `x_dense` strictly increasing, running from exactly `0.0` to exactly `1.0`, and equal to `i/2560` to `1e-15` absolute; `uexact_dense` equal to the analytic `e^x x²(1−x)²` on that grid; max-norm error recomputed from `uh_dense` below `1e-9` (reference `1.38e-10`); `error_dense == uh_dense − uexact_dense`; and that recomputed max-norm error equal to `dense_errors[3]`, so the artifact and the convergence table's `n = 256` row provably come from the same run. - **Provenance**: `fem_solver.py` present, importable without side effects, exposing `solve(n)`, importing nothing outside the standard library + NumPy + SciPy, returning the prescribed grids, and regenerating both error measures to within `1e-6` relative of the submitted values. ## What the evaluator checks 1. **Array schema** — all nine arrays present (`n_values`, `nodal_errors`, `dense_errors`, `nodal_eoc`, `dense_eoc`, `x_dense`, `uh_dense`, `uexact_dense`, `error_dense`), correctly shaped `(7,)` / `(2561,)`, and finite; `n_values` exactly `2^p`, `p = 5..11`; errors strictly positive; `nodal_eoc[0]` and `dense_eoc[0]` NaN and both tails recomputed from the submitted errors. 2. **Sidecar consistency** — `results.json` `nodal_study.table` and `dense_study.table` (`n`, `h`, `e_n` / `e_star_n`, `beta`) against the arrays, with `h == 1/n` and every `beta` recomputed; `dofs_per_node == ["value", "slope"]`; `exact_solution` present; `asymptotic_order == 4`; `dense_study.observed_order_clean_regime` supported by the submitted errors (least-squares fit over `n = 32…256`, tolerance `0.25`); and the `artifact` block (`n`, `points`, `dense_max_error`) consistent with the `n = 256` row. 3. **Numerical tiers** — as tabulated above, recomputing the exact solution and every order analytically rather than reading the agent's claims. 4. **Dependency allowlist** — an AST pass over `fem_solver.py` rejecting any top-level import outside stdlib + NumPy + SciPy (so any FEM/PDE package fails, including ones not enumerated anywhere). 5. **Scheme fidelity — does the solver actually do finite elements?** Reproducing one's own numbers proves the arrays came from the submitted code, not that the code assembles or solves anything. The grader therefore carries its **own** cubic-Hermite implementation (5-point Gauss, where the bundled reference uses 3-point) and compares the returned dense field pointwise at **every** mesh `n = 32 … 2048` to `1e-9`. Calibration: two legitimate implementations differing only in quadrature agree to `~5e-12`, while the method's own error at `n = 32` is `5.4e-07` — five orders of separation. This catches wrong fields and hybrid implementations, but is not by itself proof of assembly: for this problem the exact cubic-Hermite interpolant can agree with the Ritz field to round-off. 6. **Reproducibility — every mesh is mandatory.** `fem_solver.py` is imported in an isolated subprocess and `solve(n)` is called to regenerate **all seven** meshes within the time budget (`--budget`, default 600 s per phase). Nothing is best-effort: the reference solves `n = 512, 1024, 2048` in 0.11 s combined, and `n = 2048` carries the Tier 2 round-off-floor requirement, so those rows cannot be accepted unverified. Regenerated nodal *and* dense errors must match the submitted ones, and both returned grids must be the prescribed `j/n` and `i/(10n)`. 7. **Artifact provenance** — the mandatory phase includes `n = 256`, the artifact mesh, so the grader also compares the submitted `x_dense` / `uh_dense` pointwise against the freshly generated field. 8. **Assembly and causal provenance.** At every mesh the isolated runner observes a library linear solve of dimension `2(n−1)`, exactly the free value/slope DOF count required by the statement. It then reruns `n = 32` after replacing the results of intercepted NumPy/SciPy solve or factorization calls with non-finite values. The returned field must change, become non-finite, or raise; an unchanged-field comparison uses `rtol=0`, `atol=1e-12`. Thus a solver cannot certify an analytic interpolant by making and discarding a dummy solve. An unobservable hand-written solve is routed to manual review and scores `0.0` under `--strict`. ### Known limits of the automated checks Stated plainly so reviewers know where manual judgement is still required: - The dependency allowlist is **static**. A submission using `importlib` or `__import__` is flagged for manual review, not auto-failed. - The "no FEM library" rule is enforced statically by the import allowlist. The runtime solve audit proves that an allowed NumPy/SciPy global solve has the published size and causally feeds the result, but it cannot identify a wholly hand-written linear solver; that case is explicitly sent to manual review. - **Pointwise artifact reproduction.** Re-running the same code in the same pinned environment reproduces the `n = 256` field bit for bit — the gold reference deviates by exactly `0.0`. Any deviation above `1e-11` is therefore unexplained, and **Tier 3 fails (0.5)**; above the artifact's whole accuracy budget (`1e-9`) the run fails outright (0.0). A deviation in between is *not* waved through with a warning: a synthetic field constructed to match `dense_errors[3]` and satisfy every analytic check lands in exactly that band, and it must not reach 1.0. If your grading environment genuinely uses a different BLAS/LAPACK, the 0.5 outcome is a prompt to adjudicate manually, not a silent pass. - If the solver is too slow for the budget, the run **fails** rather than accepting unverified rows. All seven meshes are mandatory and the reference solves the three largest in 0.11 s combined, so a timeout indicates a problem with the submission, not an honest cost. Raise `--budget` if a submission is merely slow. ## Gold reference (verified) | n | nodal e_n | β | dense e*_n | β* | |---|---|---|---|---| | 32 | 4.48e-12 | — | 5.40e-07 | — | | 64 | 5.98e-14 | (super) | 3.46e-08 | 3.965 | | 128 | 1.01e-11 | (super) | 2.19e-09 | 3.983 | | 256 | 1.10e-11 | (super) | 1.38e-10 | 3.991 | | 512 | 5.38e-10 | — | 5.38e-10 | (round-off) | | 1024 | 8.40e-09 | — | 8.40e-09 | (round-off) | | 2048 | 1.22e-07 | — | 1.22e-07 | (round-off) | Asymptotic order **4** (dense). Nodal values are superconvergent (exact up to round-off); both measures are eventually dominated by linear-solve round-off from the `O(n⁴)` conditioning of a fourth-order operator. ### Threshold calibration Measured on complete candidate submissions, each graded end to end: | implementation | e*(32) | dense order (clean) | e_n(32) | verdict | |---|---|---|---|---| | gold (3-pt Gauss load, full Hermite reconstruction) | 5.402e-07 | 3.965 / 3.983 / 3.991 | 4.48e-12 | **accept** | | slope DOFs omitted from the reconstruction | 1.159e-03 | 1.013 / 1.003 / 1.001 | 4.48e-12 | reject (Tier 1 band, order 1) | | 2-point Gauss–Legendre load | 5.495e-07 | ≈ 4 | 8.96e-08 | reject (no nodal superconvergence) | | midpoint-rule load | 4.917e-07 | ≈ 4 | 4.71e-07 | reject (no nodal superconvergence) | The slope-omitting reconstruction is the decisive case: its **nodal** study is indistinguishable from the gold one (`4.48e-12`, identical — the DOFs are the same), and only the dense measure exposes it, by three orders of magnitude at `n = 32` and by collapsing the order from 4 to 1. The two under-integrated loads show why the specification requires at least 3-point Gauss–Legendre: the discrete solution is no longer exact at the nodes, so the superconvergence the task asks the agent to *observe* is destroyed. ## What a correct run looks like Correct `C¹` Hermite assembly and load (3-point Gauss–Legendre or exact); a dense reconstruction that uses **all four** local DOFs (values *and* slopes); clean order 4 on `e*_n` in the pre-round-off regime; nodal superconvergence; a visible round-off floor at small `h`; `fem_solver.py` exposing `solve(n)` and reproducing the submitted errors and the `n = 256` field when re-executed; and a consistent `n = 256` artifact on the exact `i/2560` grid. ## What a wrong run looks like Missing `fem_solver.py` or outputs; a `C⁰` Lagrange element (non-conforming for a fourth-order form); wrong stiffness entries or slope-DOF scaling; an under-integrated load (nodal superconvergence gone); a dense reconstruction that drops the slope terms (nodal error fine, dense error order ≈ 1 — the common bug, and the real bug in the original MATLAB solution); a hand-edited `nodal_eoc` / `dense_eoc`; `results.json` inconsistent with the arrays; an artifact on a grid that is not exactly `i/2560`; a `uexact_dense` that is not the analytic solution; an artifact spliced from a different run than the convergence table's `n = 256` row; a solver that cannot reproduce the numbers submitted alongside it; or a disallowed FEM library.