{"claim":{"id":"80bb3464-e227-434d-b241-508861a58277","submission_id":"ad394f27-35d8-4ba8-94da-a218520520ae","submission_revision_id":"a015d168-8be6-4e1d-bc82-2a2008a96fec","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:35:21.714532Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"ad394f27-35d8-4ba8-94da-a218520520ae","submission_revision_id":"a015d168-8be6-4e1d-bc82-2a2008a96fec","revision_version":1,"submission_status":"under_review","submitted_at":"2026-10-01T05:15:05.883966Z","submitter_name":"Qiming Bao","submitter_email":"bqmbill714@gmail.com","task":{"status":"submitted","version":1,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 5.5","harness":"Claude Code","estimated_score":"20–40%"}},"submitted_at":"2026-10-01T05:15:05.883966+00:00","industry_domain":"AI/ML Evaluation, Agent Reliability Testing, and Automated Decision Systems","task_short_name":"Hidden-State Control Lab","operating_system":"Linux","task_description":"The participant is given a custom command-line control environment containing calibration levels and five previously unseen evaluation levels. The environment is partially observable: some state variables and target information are hidden, and additional observations must be acquired through actions that consume the same limited action budget used for solving the task.\nThe participant must interact with the environment, form and revise hypotheses about its hidden transition dynamics, reason about action history and latent state, and complete each evaluation level as efficiently as possible. The same visible action may have different effects depending on previously observed or unobserved context, so simply memorizing a fixed state-to-action mapping is insufficient.\nThe participant may use only the documented command-line interface. Reverse-engineering the executable, inspecting or modifying hidden state, fabricating audit records, or otherwise bypassing the interaction protocol is prohibited.\nThe final deliverables are an environment-generated audit log, a scorecard summarizing performance on all evaluation levels, and strategy notes documenting observed evidence, hypotheses, uncertainty, and how the participant adapted its strategy during the task.\nPerformance is evaluated not only by whether the levels are completed, but also by action efficiency, resets, failed commitments, and use of high-cost information-gathering actions.","software_licensing":"In-house/Proprietary","evaluation_criteria":null,"verification_method":"Multiple valid solution trajectories are expected. The task intentionally allows participants to discover the hidden mechanics using different experiments and to reach the same successful terminal states through different action sequences.\nEvaluation should therefore not require an exact match to the reference action path. The authoritative environment-generated audit should instead be used to verify level completion, action count, resets, failed commits, use of high-cost sensing actions, and the final score.\nStrategy notes may also differ substantially between successful submissions because a participant can complete a level using a locally sufficient model without fully identifying every hidden rule. Claims in the strategy notes should be evaluated against the recorded observations rather than against exact reference wording.\nNumerical scores and level-completion status are deterministic for a given recorded action trajectory.","software_and_version":"CONTROL-LAB v15; custom statically linked Linux x86-64 CLI benchmark environment","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"1384bfab-6492-4533-a649-69fb2a169c84","filename":"ALE_control_lab_v15_task.zip","category":"input_materials","content_type":"application/x-zip-compressed","size_bytes":332121},{"file_id":"eae8a36f-8b28-420c-bd7d-ed0ecc821baa","filename":"ALE_control_lab_v15_reviewer_ONLY.zip","category":"reference_output","content_type":"application/x-zip-compressed","size_bytes":3637},{"file_id":"4cd3b00a-bfac-47b5-bb20-d8bf532fa775","filename":"submission.zip","category":"difficulty_self_test_evidence","content_type":"application/x-zip-compressed","size_bytes":5215}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally coherent, safely packaged, objectively scored, and demonstrably reachable through both a 100/100 reference run and an independent 35/100 self-test.\n\n**Completeness Check:**\n- Instruction: Complete — The participant-facing material defines the objective, level order, permitted interface, integrity restrictions, resource constraints, and three required deliverables.\n- Inputs: Complete — The package includes the Linux x86-64 environment, interface documentation, environment notes, and output templates without exposing reviewer-only mechanics.\n- Toolchain: Complete — A statically linked x86-64 Linux executable is supplied and requires neither network access nor external runtime dependencies.\n- Reference output: Complete — Reviewer-only materials include hidden mechanics, valid reference paths, a complete 100/100 audit, and a manual review guide.\n- Rubric: Complete — The scoring formula specifies action, reset, deep-scan, and failed-commit penalties, while the review guide covers completion, integrity, and reasoning quality.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. All three manifest files were inspected: `ALE_control_lab_v15_task.zip`, `ALE_control_lab_v15_reviewer_ONLY.zip`, and `submission.zip`. Their downloaded sizes match the manifest, and all are readable, unencrypted ZIP archives with valid relative paths, ordinary files/directories, successful integrity tests, and modest expansion. Text, CSV, and JSON members use readable ASCII-compatible encodings; both audits have valid, consistent schemas. The supplied executable was confirmed through non-executing metadata inspection as a stripped, statically linked 64-bit x86-64 Linux ELF; its internals were not executed or reverse-engineered. No archive content remained materially unreviewed.\n\nA participant can read the documented interface, complete C1 and C2, solve E1–E5 through the supplied CLI, export the environment audit, and prepare the matching scorecard and strategy notes. Evaluation can verify completion and penalties from the audit, recompute scores using the disclosed formula, and qualitatively assess the notes. Reference paths replay correctly against the disclosed mechanics, all reference scores recompute to 20 per evaluation level, and the self-test scorecard exactly matches its audit and recomputes to 35/100. Reviewer-only assets are clearly separated from participant inputs. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone. No affirmative evidence supports an `ai_synthetic`, `public_benchmark`, or `synthetic_defect_fixture` label; public identifiability was not externally checked.\n\n**Recommended Changes:**\nThe submission already provides a complete interaction contract, working evidence, deterministic scoring, and appropriate reviewer guidance. No changes are required.","reviewed_at":"2026-10-01T05:27:33.230725Z"},"claim":{"id":"80bb3464-e227-434d-b241-508861a58277","submission_id":"ad394f27-35d8-4ba8-94da-a218520520ae","submission_revision_id":"a015d168-8be6-4e1d-bc82-2a2008a96fec","builder":"builder-01","status":"active","claimed_at":"2026-10-15T15:35:21.714532Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}