{"submission_id":"0a1e7f40-9411-4540-9831-6e5c4438f246","submission_revision_id":"6e1fb55e-5974-40ab-887f-c59eede1ffef","revision_version":3,"submission_status":"under_review","submitted_at":"2026-08-16T08:07:30.007383Z","submitter_name":"Ziao Yang","submitter_email":"ziaoyang@brandeis.edu","task":{"status":"submitted","version":3,"extra_fields":{},"submitted_at":"2026-08-16T08:07:30.007383+00:00","industry_domain":"Machine Learning Systems / LLM Post-Training","task_short_name":"preference_training_release_qualification_1","operating_system":"Linux","task_description":"You own the numerical release gate for a preference-training platform. Audit the staged, runnable CPU candidate and independently validate it as the contract for accelerator backends. For each example define s+=<theta,x+> and s-=<theta,x-> as full-tree inner products; r+=reference_chosen and r-=reference_rejected are fixed frozen-reference scalar sequence log-scores. DPO uses z=beta*((s+-s-)-(r+-r-)), loss=softplus(-z), grad=-beta*sigmoid(-z)*(x+-x-). This release's sanitized linear-policy ORPO uses margin=s+-s-, loss=softplus(-s+)+beta*softplus(-margin), grad=(-sigmoid(-s+)-beta*sigmoid(-margin))*x+ + beta*sigmoid(-margin)*x-; reference scores are ignored for ORPO. Copy the candidate to output/qualified_repo, execute the complete DPO/ORPO qualification matrix, add mutation-sensitive release-contract tests, and write a truthful release_report.json plus validation_metrics.json. A valid release_report.json, a valid validation_metrics.json, and a passing mutation-sensitive regression suite are mandatory pass gates: if any is missing, invalid, or failing, the release fails and its score is capped at 0.84, below the 0.85 pass threshold; weighted numerical results cannot compensate. Preserve run_training_window(payload), treat inputs as read-only, and approve only if every measured gate passes.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"The evaluator first makes its hidden bundle root-private, then snapshots a fixed whitelist of candidate files through bounded O_NOFOLLOW/O_NONBLOCK descriptors and securely removes the official references before candidate execution. Candidate Python runs as an unprivileged user with CPU, memory, process, file, descriptor, and wall-clock limits. An independently written NumPy oracle checks DPO/ORPO selected IDs, full-window gradients, parameters, complete Adam state, invalid-window skip behavior, and partition invariance on public-family and independently seeded private windows. The submitted release and validation claims are cross-checked against fresh sandbox replays rather than trusted as self-report. A valid release_report.json, a valid validation_metrics.json, and a passing mutation-sensitive regression suite are mandatory pass gates: failure of any one caps the score below the 0.85 threshold and makes the release fail. Regression tests must pass on the submitted candidate and fail under controlled release-decision/optimizer-contract mutations; filenames, function names, comments, and keywords earn no credit. Public-family cases weigh 35% and private cases 65%; the normalized pass threshold is 0.85.","software_and_version":"Python 3.12.13, NumPy 2.3.5, Python 3.12 unittest","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"2d25c208-84c9-419b-8eb5-b55c4bfd3de4","filename":"input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":20697},{"file_id":"9e392dfc-5df0-4d8e-9404-87db31c09106","filename":"reference_output_and_evaluation_dependencies.zip","category":"reference_output","content_type":"application/zip","size_bytes":52509}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, coherent, professionally representative, objectively evaluable, and reachable on the declared CPU toolchain.\n\n**Completeness Check:**\n- Instruction: Complete — Defines the exact DPO and ORPO mathematics, full-window aggregation, curation, clipping, Adam behavior, required artifacts, and mandatory release gates.\n- Inputs: Complete — Supplies a runnable candidate, an 11-example qualification window, release configuration, baseline provenance, documentation, and pinned runtime metadata.\n- Toolchain: Complete — Python 3.12, NumPy 2.3.5, and standard-library unittest provide a legally obtainable, CPU-compatible execution path.\n- Reference output: Complete — Includes a qualified repository, mutation-sensitive contract tests, release report, validation metrics, evaluator cases, and an independent numerical oracle, separated from solver-visible inputs.\n- Rubric: Complete — Defines objective weights, visible/private weighting, numerical tolerance, the 0.85 threshold, the 0.84 mandatory-gate ceiling, and measurable pass/fail conditions.\n\n**File and Task Reachability:**\nAttachment classification: reachable. Task classification: reachable. I inspected `input_materials.zip` and `reference_output_and_evaluation_dependencies.zip`, including all 38 internal members; no content remained unreviewed. Both files match their manifested sizes, are readable deflated ZIP containers, pass integrity checks, and contain bounded regular files without traversal paths, links, devices, encryption, or unsafe expansion. Embedded member sizes and SHA-256 values are internally consistent. The content is predominantly ASCII or UTF-8 Python, Markdown, TOML, shell text, and valid JSON. Representative JSON inspection confirmed nested parameter trees, complete Adam state, public and private DPO/ORPO cases, nonfinite-row probes, report schemas, and weighted rubric fields.\n\nThe end-to-end path is complete: copy the candidate, audit or correct it, execute both objectives across all partition plans, add contract tests, and produce the two required JSON reports. Evaluation snapshots a fixed submission surface, removes privileged references before candidate execution, runs under resource limits, compares fresh results with an independent NumPy oracle, verifies partition and optimizer behavior, exercises controlled mutations, cross-checks report claims, and applies the mandatory-gate ceiling.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nThe submission already provides a rigorous numerical contract, realistic release workflow, secure evaluation boundary, private-case coverage, and mutation-sensitive mandatory gates. No changes are required.","reviewed_at":"2026-08-16T08:12:28.974961Z"},"claim":null}