{"submission_id":"7bb1723e-0adb-4ee1-b20a-5c982293f823","submission_revision_id":"f75c9a1f-1ac5-4445-82a6-2e94e7fc9327","revision_version":3,"submission_status":"under_review","submitted_at":"2026-08-22T02:02:21.968658Z","submitter_name":"Zhenning Yang","submitter_email":"znyang@umich.edu","task":{"status":"submitted","version":3,"extra_fields":{"difficulty_self_test":{"estimated_score":"40–60%"}},"submitted_at":"2026-08-22T02:02:21.968658+00:00","industry_domain":"Cloud Operations","task_short_name":"iam_least_privilege_reconstruction","operating_system":"Linux","task_description":"You are a cloud security engineer. Read the 90 days of CloudTrail-style access logs in `input/access_logs.jsonl`, the role's current over-broad policy, and the fixed permission boundary + SCP, then author a new **least-privilege** identity policy set for the `etl-pipeline-role` that:\n\n1. **Allows** every action the role genuinely needs — reconstructed from the *successful* calls in the logs, and generalized so it still permits the same behaviour on held-out resources of the same kind (new object keys under a prefix already used, the next rotated secret, another table index, etc.).\n2. **Denies** everything else — scope every statement to what the evidence supports; `AccessDenied` events and the shared-namespace note in the metadata mark boundaries you must keep denying.\n3. **Closes** every privilege-escalation path the current policy leaves open (role-passing, IAM writes, role assumption, resource-policy edits the role never legitimately used).\n4. Stays **within** the permission boundary and SCP (they only remove permissions).\n\nRead `input/reconstruction_spec.md` first — it is the precise engine semantics and scoring contract. Then write your two deliverables to `output/` (below). Treat `input/` as read-only; do not attempt network access.\n\n## What you're working with (`input/`, read-only)\n\nThe heart of it is `access_logs.jsonl` — about 90 days of CloudTrail-style events for the role. The successful calls (no `errorCode`) are your ground truth for what the role legitimately does; the `AccessDenied` events mark lines the correct policy should keep denying. Treat the logs as a *representative sample*, not an exhaustive checklist: your policy has to generalize to the same behaviour on resources of the same kind it hasn't happened to touch yet (new object keys under a prefix it already uses, the next rotated secret, another table index, and so on). Deny something the pipeline actually needs and you've broken it; that costs you.\n\nAlongside the logs:\n\n- `current_identity_policy.json` — the over-broad policy you're replacing. Copying it verbatim is rejected outright.\n- `permission_boundary.json` and `scp.json` — the fixed ceilings. Both only *remove* permissions; neither grants anything. A request the boundary or SCP denies stays denied no matter what your identity policy says.\n- `role_metadata.json` — account id, region, role ARN, and the relevant resource ARNs/prefixes. Watch out: some ARN namespaces are shared across teams, so a matching prefix doesn't prove entitlement. The logs decide the exact sub-scope.\n- `reconstruction_targets.json` — the self-check targets you report a verdict for.\n- `reconstruction_spec.md` — the precise engine semantics and scoring contract. Read this first; it's the rulebook the grader plays by.\n\n## What to hand back (`output/`)\n\n1. `reconstructed_policies.json` — `{\"policies\": [ <IAM policy document>, ... ]}`. This is the deliverable that's actually graded: the grader attaches it to the role and evaluates authorization requests against it with a frozen IAM engine.\n2. `reconstruction_report.json` — your predicted `ALLOW`/`DENY` for every id in `reconstruction_targets.json`, either a list of `{\"id\": ..., \"verdict\": ...}` or an `{id: verdict}` map. This is a self-consistency artifact; it never overrides what the engine actually computes from your policy.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Three hard gates first — miss any one and the score is zero: the policy has to be valid IAM grammar, it can't be a verbatim copy of the broken policy, and it can't be a degenerate deny-all that allows almost nothing. Clear those and the score is a weighted blend of four components:\n\n```\nscore = 0.40 * coverage + 0.30 * tightness + 0.20 * escalation_safety + 0.10 * minimality\n```\n\n- **coverage** (0.40) — of the held-out requests the role genuinely needs, how many your policy still allows. This is the \"did you break the pipeline\" axis.\n- **tightness** (0.30) — of the requests it should *not* have, how many you deny.\n- **escalation_safety** (0.20) — how many of the escalation probes (role-passing, IAM writes, role assumption, resource-policy edits it never legitimately used) you shut down.\n- **minimality** (0.10) — how close your effective footprint is to the reference's, penalizing over-grants.\n\nThese weights live in `scripts/scorer.py` (`W_COVERAGE` / `W_TIGHTNESS` / `W_ESCALATION` / `W_MINIMALITY`).\n\nEvery decision is recomputed by the frozen engine over held-out request contexts you can't see, so you can't game it by pattern-matching the visible logs, and the numbers in your report are never taken on faith. The two failure modes the rubric is built to punish: copy the broad policy (or grant wildcards) to ace coverage, and you surrender tightness, escalation-safety, and minimality — roughly half the score or worse.\n\n## Running the grader\n\nThe whole grader is one self-contained file, `evaluate.py` (stdlib-only Python 3 — the frozen IAM engine, the rubric, and the self-test all live inside it):\n\n```bash\npython3 evaluate.py            # grade output/reconstructed_policies.json against reference/\npython3 evaluate.py --selftest # prove the grader reproduces the reference gradient\n```\n\n`reference/` holds the hidden ground truth — the gold policy, the held-out required / not-needed / escalation contexts, the audit grid, and the reference effective grants. The self-test drives the scorer with the gold policy (must score 1.0) plus a battery of adversarial and degenerate policies, and checks the documented score bands still hold: enumerate-exactly-the-logs lands high but under 1.0, admin `*:*` collapses to ~0.42, and deny-all / verbatim-copy / invalid-grammar each trip their hard gate.","software_and_version":"Python 3.9+","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"608d4026-1942-43fd-80aa-c73ee0814e63","filename":"input.zip","category":"input_materials","content_type":"application/zip","size_bytes":38334},{"file_id":"12811ceb-c2c1-465b-8b14-04042f139286","filename":"output-eval.zip","category":"reference_output","content_type":"application/zip","size_bytes":30700}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally consistent, safely reachable, and supports deterministic end-to-end least-privilege reconstruction and evaluation.\n\n**Completeness Check:**\n- Instruction: Complete — Clearly defines the evidence model, generalization requirements, security boundaries, and two output artifacts.\n- Inputs: Complete — Includes 276 valid access-log records spanning approximately 88 days, current and boundary policies, SCP, role metadata, reconstruction specification, and 100 visible targets.\n- Toolchain: Complete — Requires only freely available Python 3.9+ standard-library tooling on Linux, with no network or cloud account dependency.\n- Reference output: Complete — Includes the reference policy, golden deliverables, required and negative contexts, escalation probes, audit grid, expected verdicts, and effective-grant manifest.\n- Rubric: Complete — Implements the documented hard gates and 40/30/20/10 weighted scoring contract with deterministic authorization semantics and calibration cases.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. I inspected `input.zip` and `output-eval.zip`, including all contained documentation, JSON/JSONL evidence, golden outputs, reference corpora, and Python evaluator sources; no content was left unreviewed. Both archives matched their declared sizes, passed container integrity checks, were unencrypted, contained no unsafe paths or special file types, and expanded to modest sizes. All structured files were valid UTF-8 and parseable, corpus counts and identifiers were consistent, and an independent read-only check reproduced all 908 expected corpus verdicts and all 148 reference effective grants. The submitted self-test was reviewed statically but could not be executed because Python was unavailable in the review environment; this is a reviewer-tooling limitation rather than a submission defect. Solver-visible inputs are separated from evaluator-only references, and the workflow supports producing both required outputs and grading them objectively on one ordinary machine. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. No affirmative evidence supports an AI-synthetic, public-benchmark, or synthetic-defect-fixture label. Public-benchmark status could not be independently searched because external access was prohibited.\n\n**Recommended Changes:**\nThe submission provides coherent evidence, reference artifacts, and objective evaluation coverage. No changes are required.","reviewed_at":"2026-08-22T02:12:14.732677Z"},"claim":null}