{"submission_id":"f66c4c1d-6380-48ce-8ac3-5e0809342a9c","submission_revision_id":"69524c8e-a6ff-4b80-a408-3f4e40b6621b","revision_version":6,"submission_status":"under_review","submitted_at":"2026-08-22T19:52:20.926644Z","submitter_name":"Zhenning Yang","submitter_email":"znyang@umich.edu","task":{"status":"submitted","version":6,"extra_fields":{"difficulty_self_test":{"estimated_score":"40–60%"}},"submitted_at":"2026-08-22T19:52:20.926644+00:00","industry_domain":"Cloud Operations","task_short_name":"k8s_gomaxprocs_cascading_oom_rca","operating_system":"Linux","task_description":"You are the on-call SRE for a Kubernetes production incident.\n\n## Task Directory\n`base`\n\n## The Incident\nThe Go-based `checkout-api` Deployment (namespace `prod`) is OOMKilling pods in a CrashLoop and its HPA is pinned at maxReplicas. The incident is still open. You are given a STATIC capture of the cluster state and change history taken while it was in progress. There is NO live cluster, kubectl, or network available - diagnose from the artifacts only.\n\n## Visible Inputs (read-only, in `base/input`)\n- Cluster state capture: `base/input/cluster_state.txt` (kubectl get/describe pods, events, top, hpa)\n- Current Deployment + HPA YAML: `base/input/deployment.yaml`\n- Rollout history: `base/input/rollout_history.txt` (per-revision CHANGE-CAUSE, rev 36..41)\n- ConfigMap revision diff: `base/input/configmap_revisions.diff` (unified diff of checkout-api-runtime `.data`)\n- Prometheus metrics: `base/input/prometheus_metrics.csv` (timestamp,metric,value timeseries)\n- Crashing pod log: `base/input/app_pod.log` (Go runtime crash logs across restarts)\n- Incident timeline: `base/input/incident_timeline.md` (sanitized on-call narrative)\n- Mitigation attempts: `base/input/mitigation_attempts.md` (ops-channel record of what responders changed during the incident and what each change did)\n\n## Your Task\nDetermine the root cause of the OOM CrashLoop, rule out the explanations the evidence does not support, and ground every claim in verbatim text from the inputs. Write exactly one JSON file:\n\n```text\nbase/output/root_cause_analysis.json\n```\n\nRequired top-level keys:\n\n```text\nincident_metadata, root_causes, affected_resources, remediation_plan, alternatives_ruled_out, causal_chain, summary\n```\n\n## Constraints\n- Treat `base/input` as read-only; write only inside `base/output`.\n- Do NOT invent resource names, timestamps, revisions, or metric values - every number and evidence string must appear literally in an input file.\n- No external web sources or network access.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Deterministic weighted JSON scoring, stdlib only, offline. The grader is two files: evaluate.py (wires input/, output/, reference/ together and holds the self-test) and scripts/scorer.py (the frozen weighted scorer plus the embedded answer key).\n\nHARD GATES (score=0.0 on any): output/root_cause_analysis.json parses as a JSON object; all 7 required top-level keys present (incident_metadata, root_causes, affected_resources, remediation_plan, alternatives_ruled_out, causal_chain, summary); root_causes non-empty. If gates pass, 10 weighted components (weights sum to 1.0) are summed:\n\n- primary_root_cause 0.20 - binary 1.0 iff a FULLY-GROUNDED cause names the diagnosis in EITHER its type label OR its prose description (~{gomaxprocs,configmapremoval,runtimeconfigregression}, normalised) AND severity=critical AND layer=config AND its evidence contains BOTH the removed diff line (regex `-\\s*GOMAXPROCS:\\s*\"2\"`) AND a rev-38 change-cause substring (containing both `rev 38` and `configmap/checkout-api-runtime`); multiplied by a PREREQUISITE that regression_revision==38 (else 0); partial 0.5 for one of the two evidence substrings.\n- secondary_traffic 0.10 - fully-grounded cause with layer=traffic, type/description ~{upload,requestmix,concurrency}, severity in {high,medium}.\n- regression_revision 0.13 - ==38 -> 1.0.\n- trigger_timestamp 0.12 - 1.0 at 2026-07-05T08:40 exact; 0.5 correct day+hour wrong minute; 0.0 otherwise, and explicitly 0.0 at the rev-41 deploy time 03:10 or the paging time 09:12.\n- causal_chain_order 0.08 - graded: fraction of the 4 canonical stages present in correct relative order (config-removal -> runtime-default-8 -> traffic-shift-08:40 -> heap-overshoot/OOM). Non-copyable.\n- extracted_numbers 0.06 - fraction of 6 numbers correct (exact for cpu/node/limit, +/-1 for concurrency, +/-0.03 for gc fraction). Keys are matched on MEANING, not spelling: the canonical names are never published to the solver (default_gomaxprocs would give the diagnosis away), so each number is claimed by the tokens its key must carry (node+cpu, cpu+limit, gomaxprocs, mem+limit, upload+concurren, gc+fraction) in any casing or separator, searched in incident_metadata.extracted_numbers first and then elsewhere in incident_metadata; millicore (2000m) and byte (536870912) renderings are accepted alongside cores and Mi. Values are still checked strictly, so a wrong number scores 0.\n- affected_resources_f1 0.09 - F1 vs the 5-item hidden reference. Pod entries are validated against POD_HASH_RE (checkout-api-[a-z0-9]+-[a-z0-9]{5}); a malformed pod name is dropped rather than counted against precision. Resources that are genuinely part of the blast radius but not required naming (the ReplicaSets checkout-api-6f8c7d9b4 and checkout-api-mitigate, the nodes ip-10-4-2-118 and ip-10-4-9-31) are neutral: listing them neither earns nor costs anything, so a MORE complete answer is never punished. Invented or irrelevant resources still cost precision.\n- remediation_regex 0.09 - three 1/3 checks: (a) restore GOMAXPROCS/automaxprocs; (b) get it onto the running pods, in any ordinary phrasing - kubectl apply/replace/patch/rollout restart, redeploy, roll the deployment, recreate or restart the pods, edit the manifest/configmap - gated on (a), since restarting pods without a corrected setting is not a fix here; and (c) a NEGATIVE guard that fails if the only fix is raise-memory-limit with no GOMAXPROCS action.\n- evidence_grounding_gate 0.07 - fraction of fully-grounded causes (every evidence string a verbatim substring of the concatenated inputs).\n- alternatives_ruled_out 0.06 - the array must contain one entry that BOTH names the memory-limit explanation ((raise|increase|bump|expand).*(memory|limit)) AND gives a grounded reason it fails (e.g. still/anyway/delay/longer/1Gi/812/unchanged/mask), and one entry that BOTH names the rev-41/JVM explanation ((rev.?41|jvm|xmx|heap)) AND gives a reason (e.g. inert/Go binary/no change/unrelated/no effect/still). Naming an alternative without justifying it scores 0.\n\npassed iff score >= 0.999999. Discrimination between depths of investigation: the diagnosis JSON is not an input, so copy-input fails; a symptom-level report (labels gomaxprocs/critical/config, cites only the log banner, copies obvious numbers, lists most resources, but never opens the diff+rollout history and never pins rev 38) is capped ~0.21 because primary+regression+trigger+secondary+alternatives all zero; a report that chases the memory limit and rev-41 is capped ~0.19; the 10:14Z positive control is deliberately NOT a shortcut - a solver who reads only mitigation_attempts.md + the pod log, reads both runtime banners, correctly labels the mechanism gomaxprocs/critical/config and even rules the alternatives out on the negative controls is still capped ~0.35, because primary (needs the removed diff line AND the rev-38 tie), regression_revision, trigger_timestamp and secondary_traffic all zero without the diff, the rollout history and the CSV; the ordered causal_chain appears nowhere verbatim, and no artifact states the trigger minute in narrative form (the timeline says only 'earlier this morning'), so the CSV must be read. Calibration invariants, asserted by `python3 evaluate.py --selftest`: pos==1.0, wrong_change in [0.10,0.20], banner in [0.15,0.22], positive_control_only in [0.25,0.45], empty==0.0, strictly monotonic empty < wrong_change < banner < positive_control_only < pos. Observed: 1.0 / 0.1975 / 0.21 / 0.38 / 0.0. The same self-test asserts EQUIVALENCE in the other direction - a paraphrased cause label, extra genuinely-involved resources, and equivalent remediation wording each still score exactly 1.0 - and asserts that this tolerance is not a hole: irrelevant extra resources (0.985), restart-only remediation (0.91) and an undiagnosed cause (0.80) all stay below 1.0. `python3 evaluate.py --calibration` regenerates reference/calibration.md from the scorer so the documented gradient cannot drift.","software_and_version":"Python 3.9+","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"09be28f0-c570-41b9-aed5-aa0731de555b","filename":"input.zip","category":"input_materials","content_type":"application/zip","size_bytes":11348},{"file_id":"9b872ffa-1511-4a77-8068-5586241e26d6","filename":"output-eval.zip","category":"reference_output","content_type":"application/zip","size_bytes":24342}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, coherent, safely inspectable, objectively graded, and reachable end to end with the declared offline toolchain.\n\n**Completeness Check:**\n- Instruction: Complete — Clearly defines the incident, evidence constraints, required JSON path, schema, and grounding rules.\n- Inputs: Complete — All eight declared incident artifacts are present, readable, and mutually consistent.\n- Toolchain: Complete — The task and evaluator require only Linux and Python 3.9+ standard-library functionality.\n- Reference output: Complete — A valid gold JSON, answer key, calibration record, and identical golden-output copy are included outside solver-visible inputs.\n- Rubric: Complete — The deterministic weighted scorer implements the documented hard gates and ten components, with calibrated positive, partial, equivalence, and negative controls.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. Both ZIP attachments matched their manifest byte counts, passed archive-integrity checks, were unencrypted, and contained only regular text files and directories without traversal paths, links, device entries, or excessive expansion.\n\nInspected input content comprised `README.md`, `configmap_revisions.diff`, `deployment.yaml`, `app_pod.log`, `prometheus_metrics.csv`, `rollout_history.txt`, `incident_timeline.md`, `cluster_state.txt`, and `mitigation_attempts.md`. The CSV has 224 complete data rows, 32 timestamps, seven consistently represented metrics, three columns, and no missing fields. Inspected evaluation content comprised `root_cause_analysis.json`, `answer_key.json`, `calibration.md`, the identical golden-output JSON, `scorer.py`, and `evaluate.py`. JSON files parse as objects, and evaluator imports are standard-library-only apart from its bundled scorer. No content was left unreviewed. Submitted Python was inspected statically rather than executed, as required for safe review; the committed calibration supports arithmetic validation of the scoring path.\n\nA solver can inspect the static evidence, produce the required JSON, and have it evaluated offline against the hidden assets. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. No affirmative evidence supports a screening label. Public identifiability could not be independently searched because no approved search capability was available.\n\n**Recommended Changes:**\nThe incident evidence, output contract, reference diagnosis, weighted evaluation, and calibration are aligned and professionally complete. No changes are required.","reviewed_at":"2026-08-23T03:09:41.610740Z"},"claim":null}