{"claim":{"id":"82a41faf-194f-43fa-9e07-b5a1341d9262","submission_id":"31db7b63-d1fa-4600-8f0c-e39b29abae99","submission_revision_id":"879c4747-bd02-4eea-b589-0d82796a10ac","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:10:26.489814Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"31db7b63-d1fa-4600-8f0c-e39b29abae99","submission_revision_id":"879c4747-bd02-4eea-b589-0d82796a10ac","revision_version":3,"submission_status":"under_review","submitted_at":"2026-08-22T02:07:36.199535Z","submitter_name":"Zhenning Yang","submitter_email":"znyang@umich.edu","task":{"status":"submitted","version":3,"extra_fields":{"difficulty_self_test":{"estimated_score":"40–60%"}},"submitted_at":"2026-08-22T02:07:36.199535+00:00","industry_domain":"Cloud Operations","task_short_name":"vpc_reachability_remediation","operating_system":"Linux","task_description":"You are a network/cloud engineer. The broken three-tier VPC Terraform has already been copied to `output/infra/`. Using the intent in `input/connectivity_policy.yaml` and the reachability model in `input/solver/reachability_solver.py`, edit `output/infra/*.tf` to restore the intended connectivity — the minimum change to security groups, NACLs, and routes, without renaming any resource — and write your per-flow verdicts to `output/reachability_report.json`.\n\nTreat `input/` as read-only; keep edits under `output/infra/`; don't modify the solver; no network (use the offline provider mirror).\n\n## What you're working with\n\n- `input/infra/` — the broken three-tier VPC as ~12 `.tf` files (read-only).\n- `input/connectivity_policy.yaml` — the tier-level intent plus two prose invariants.\n- `input/solver/reachability_solver.py` — the frozen reachability model (the contract).\n- `output/infra/` — where you work; the broken Terraform has already been copied here for you to edit in place.\n- A `software/` mirror (staged by the harness) with Terraform 1.7.5 and the `hashicorp/aws` 5.31.0 provider vendored on disk, so you can `terraform init -plugin-dir=…` and `plan` fully offline — no registry, no credentials, no network.\n\nA few semantics worth remembering while you edit: security groups are **stateful** (the return traffic is implicit), NACLs are **stateless** (you must explicitly allow the ephemeral return range `1024–65535`), and a subnet's `0.0.0.0/0 → igw` vs `→ nat` default route is what decides internet reachability.\n\n## What to hand back (`output/`)\n\n1. Corrected Terraform under `output/infra/` that plans cleanly and does **not** rename any logical address (`aws_*.<name>`) relative to `input/infra/` — every resource the fix keeps stays at its original address. Change only the minimum SG rules, NACL entries, and routes.\n2. `output/reachability_report.json` — a JSON object mapping each flow id under `flows:` in `connectivity_policy.yaml` to your computed verdict:\n\n   ```json\n   {\n     \"flows\": [\n       {\"flow_id\": \"alb_from_internet_443\", \"verdict\": \"ALLOW\"},\n       {\"flow_id\": \"db_from_internet_any\",  \"verdict\": \"DENY\"}\n     ]\n   }\n   ```\n\nThe grader recomputes every verdict from your config with the frozen solver, so the report is a self-consistency artifact — an entry that disagrees with the solver's own verdict scores that flow as if fabricated.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Three hard gates first: `terraform init -plugin-dir=… ` + `plan` from `output/infra/` has to succeed; no logical-address rename (renaming a resource the fix preserves scores 0); and the plan-derived world-state has to be structurally valid. Then:\n\n```\nscore = 0.85 * flow_fraction + 0.15 * inventory_fidelity\n```\n\n`flow_fraction` is the fraction of scored flows whose solver verdict over your config matches the reference — 14 visible policy flows (which also require your report to agree) plus **14 held-out flows** you can't see. ALLOW and DENY weigh equally, so \"open everything\" and \"close everything\" each fail about half the flows. `inventory_fidelity` is the Jaccard overlap of your resource addresses with the reference, rewarding a minimal diff.\n\n## Running the grader\n\nThe whole grader is one self-contained file, `evaluate.py` (stdlib-only Python 3 — the frozen solver, the `terraform show -json` → world_state reader, the scorer, and the self-test all live inside it):\n\n```bash\n# grade output/infra (needs a terraform binary + the offline provider mirror):\npython3 evaluate.py --terraform-bin software/terraform \\\n                    --provider-mirror software/provider-mirror \\\n                    --terraformrc software/.terraformrc\n\n# prove the grader reproduces real-AWS ground truth — no terraform, no network:\npython3 evaluate.py --selftest\n```\n\n`reference/` holds the hidden ground truth: the flow verdicts, the held-out flows, the resource inventory (with the `must_keep` set the rename gate enforces), and — under `reference/testdata/` — three committed `terraform show -json` plan fixtures for the broken / partial / correct states. The self-test replays those fixtures through the exact same pure functions the real grader uses and asserts the plan-derived verdicts equal what was measured on real AWS during calibration (broken 16/28 with an identical fail-set, partial 20/28, correct 28/28), that the score is monotonic, and that a fabricated report can never beat an honest one.","software_and_version":"Python 3.9+, Terraform 1.7.5","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"aee09cb5-18a3-4aa1-b7a3-52bbcf0e269e","filename":"input.zip","category":"input_materials","content_type":"application/zip","size_bytes":14855},{"file_id":"6e5a5eb6-8214-4a3b-b8fc-c685828a7f1a","filename":"output-eval.zip","category":"reference_output","content_type":"application/zip","size_bytes":30417}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally consistent, professionally scoped, safely reachable, and supported by a rigorous offline evaluator and coherent reference evidence.\n\n**Completeness Check:**\n- Instruction: Complete — The required Terraform remediation, resource-address constraint, allowed edit surfaces, report schema, and end state are explicit.\n- Inputs: Complete — The broken Terraform, tier-level policy, and frozen reachability model are readable, mutually consistent, and sufficient for deterministic remediation.\n- Toolchain: Complete — Exact Terraform and AWS provider versions are specified, with a declared offline mirror and no live cloud dependency.\n- Reference output: Complete — Golden Terraform, the expected report, world state, resource inventory, hidden flows, verdicts, and broken/partial/correct plan fixtures are present and structurally consistent.\n- Rubric: Complete — Hard gates, flow scoring, report-consistency behavior, inventory fidelity, weights, passing threshold, and self-test expectations are objective and clearly defined.\n\n**File and Task Reachability:**\nThe attachment set is reachable, and the end-to-end task is reachable. Both `input.zip` and `output-eval.zip` were inspected, including every archive member: documentation, Terraform, YAML policy, solver and evaluator source, JSON references, golden output, and all three plan fixtures. The archives match their declared formats and sizes, parse cleanly, are unencrypted, and contain no unsafe paths, links, device entries, or excessive expansion. JSON schemas and identifiers are consistent: 14 visible flows map exactly to the golden report, 14 held-out flows map into 28 unique reference IDs, and the correct plan's 49 addresses exactly match the reference inventory while preserving all required addresses. No archive content was left unreviewed. Safe-review constraints precluded executing submitted source, and the separately staged Terraform/provider binaries were outside the attachment manifest; the pinned plan fixtures and evaluator source nevertheless establish a coherent offline completion and evaluation path from Terraform edits and report generation through planning, world-state construction, hard gates, and scoring. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nThe submission already provides unusually strong task, reference, and evaluator coverage. No changes are required.","reviewed_at":"2026-08-22T04:46:20.502968Z"},"claim":{"id":"82a41faf-194f-43fa-9e07-b5a1341d9262","submission_id":"31db7b63-d1fa-4600-8f0c-e39b29abae99","submission_revision_id":"879c4747-bd02-4eea-b589-0d82796a10ac","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:10:26.489814Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}