{"submission_id":"e460fff0-490c-4149-b777-53be185f2d0c","submission_revision_id":"be9291d1-752a-4bac-8ce7-988cc45fc631","revision_version":1,"submission_status":"under_review","submitted_at":"2026-09-29T19:59:06.890754Z","submitter_name":"HAIXIANG TANG","submitter_email":"hansont2@illinois.edu","task":{"status":"submitted","version":1,"extra_fields":{},"submitted_at":"2026-09-29T19:59:06.890754+00:00","industry_domain":"Government Statistics / Public Administration / Reproducibility and Uncertainty Auditing","task_short_name":"acs-poverty-audit","operating_system":"Windows","task_description":"Conduct an independent reproducibility and uncertainty audit of the six poverty tables in Census report ACSBR-016 (2021/2022 ACS 1-year data). Use the attached solver_inputs.zip, whose TASK.md is the complete output contract. The published report is an explicit audit comparand, not a withheld answer key.\n\nReconstruct poverty counts, rates, year-to-year changes, and seven income-to-poverty bands from B17001/B17002. Reconcile national/state/Puerto Rico scope, the 25 largest metropolitan areas, the separate all-metro aggregate, and available metros versus Figure 5's 65,000 population threshold using the supplied geography and B01003 data. Apply the supplied official count-based MOE approximations without inventing covariance. Compare all 1,560 keyed estimate/MOE records with actual published values and change flags; retain rounded-zero markers, calculate discrepancy magnitudes, and distinguish direct published count MOEs from approximate matches and mismatches.\n\nDeliver audit.json with the exact required nested schema in TASK.md: scope, row-level calculations, discrepancy register, reconciled summaries, evidence-supported identifiability limitations, and source hashes. An optional memo.md can summarize Findings, Methods, and Limits, but has no subjective points. All scoring is deterministic. No network, credentials, S1701 request, historical-series reconstruction, map drawing, or unavailable exact covariance is required. This is a six-table audit, not exact reproduction of every result in the full report. Do not treat approximation disagreement as a proven source error or rounded agreement as exact uncertainty recovery.\n\nUse the sources offline. Preserve original files, use appropriate Census attribution/nonendorsement, and do not attempt respondent identification. Return audit.json as the graded output.\n\nThe frozen input archive contains TASK.md, a source manifest and 18 unchanged official files: five DAT count tables, two geography TXT files, five schema JSONs, four source/method/policy PDFs and two policy HTML files. The separate evaluator archive contains the complete reference output, scorer, reference builder, recorded tests and hashes. Source data are authentic; this is an independent audit adaptation, not a claim to have authored the original Census report. No independent-model difficulty result or human-expert review is claimed.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"Unpack the evaluator separately from solver inputs. With Python 3.12.14, run python score.py PATH_TO_SOLVER/audit.json. The scorer reads inert JSON only and never executes solver code. The perfect control is reference_outputs/audit.json. Run python score.py reference_outputs/audit.json and python -B test_score.py. The supplied control scores 100; test_results.json records 46 passing local validation tests. Portable reference rebuilding from the official inputs is documented in README.md.\n\nCompare all 1,560 keyed records, exact required nested structures, geography membership, independently calculated point estimates and specified approximate MOEs, discrepancy magnitudes/register, summaries, report/approximate change flags, identifiability conclusions and source hashes.\n\nWeights out of 100: geographic scope 10; point estimates 25; uncertainty 25; change flags 10; discrepancy register 10; summaries 10; structured limitations 5; source integrity 5. Within components, required fields receive equal weight. Discrepancy sets use intersection/union. Numeric absolute tolerance is 0.000001; count point estimates and count deltas, summary integers, identifiers, booleans, hashes and membership sets require exact agreement. Full pass requires every checked field/set, not merely a displayed score rounded to 100. Missing rows lose their field points; valid partial work receives a continuous score. The optional memo has no subjective points.\n\nMissing, unreadable, empty or malformed JSON; duplicate object keys/row IDs; unexpected row IDs; invalid required nested schemas; nonfinite/out-of-binary64-range numbers; or invalid source-hash filename schema cause a hard failure and zero points. Artifacts must be regular nonsymlink files, at most 50 MB. Extra explanatory fields cannot override required nested fields. Damaged protected reference material gives an evaluator error with score=null/exit 2, not a solver penalty.\n\nDo not grant credit for claims of unavailable exact covariance. Approximation/report disagreements are audit findings, not automatic Census errors. The full schema, identifier conventions, formulas, rounding and null/flag rules are solver-visible in TASK.md.","software_and_version":"Python 3.12.14; pdfplumber 0.11.9; pypdf 6.10.0. The scorer uses only the Python standard library. PDF libraries support reference construction and rebuilding. Tested on Windows; offline workflow.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"0d3d4715-9895-4804-bc2e-1cbdd751f682","filename":"solver_inputs.zip","category":"input_materials","content_type":"application/x-zip-compressed","size_bytes":8444253},{"file_id":"c9723087-faf9-4b0e-a9e3-5e780f01b06d","filename":"evaluator.zip","category":"reference_output","content_type":"application/x-zip-compressed","size_bytes":176351}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, professionally substantive, and reachable offline, with independently reconciled reference calculations and no material defect found.\n\n**Completeness Check:**\n\n- Instruction: Complete — TASK.md defines the six-table scope, all 1,560 records, formulas, rounding, flags, uncertainty classifications, and exact output schema.\n- Inputs: Complete — All 18 source files are readable and support the required calculations, geography selection, report comparisons, and policy requirements.\n- Toolchain: Complete — The declared open-source workflow is suitable for one ordinary machine; scoring requires only Python’s standard library.\n- Reference output: Complete — The perfect-output audit.json equals the protected reference; all 1,560 records reconcile with independent calculations and report transcription.\n- Rubric: Complete — Deterministic weights, tolerances, partial credit, hard failures, and full-pass conditions are explicitly defined and reflected in the scorer.\n\n**File and Task Reachability:**\n\nAttachments: `reachable`. End-to-end task: `reachable`.\n\nBoth solver_inputs.zip and evaluator.zip were inspected completely at the attachment level. Inspected contents include TASK.md, source_manifest.json, all five DAT tables, Geos20211YR.txt, Geos20221YR.txt, all five schema JSONs, acsbr-016.pdf, 2022_Instructions_for_Stat_Testing_ACS.pdf, ACS_Accuracy_of_Data_2022.pdf, ds027.pdf, census_api_terms.html, and census_citation_policy.html. Evaluator inspection covered score.py, reference.json, the perfect-output audit.json, reference_builder.py, test_score.py, test_results.json, README.md, and package_inventory.json.\n\nArchive sizes match the attachment manifest; both containers pass integrity checks without unsafe paths or encryption barriers. Source and evaluator-member hashes match their supplied inventories. Geography membership, estimates, MOEs, deltas, published flags, Z markers, discrepancy records, and summaries reconcile. The reference correctly distinguishes 392 available metros from 389 threshold-eligible metros.\n\nThe completion path is offline source parsing and report transcription, independent calculation, production of audit.json, and deterministic evaluation using the separately retained evaluator. No required content remained inaccessible. Submitted programs were inspected but not executed; the recorded 46 passing tests were not independently rerun. External provenance was not independently verified online.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\nNone. The inspected evidence supports an audit of official aggregate statistics, not fabricated core inputs or injected defects. Public Census provenance alone does not establish a public benchmark; external benchmark status remains unverified.\n\n**Recommended Changes:**\n\nThe submission preserves source integrity, separates evaluator answers from solver inputs, and handles approximation and identifiability limits explicitly. No changes are required.","reviewed_at":"2026-10-14T19:48:57.871863Z"},"claim":null}