{"claim":{"id":"db923d02-f7ab-495e-a73c-50896f39ca7b","submission_id":"3810cf34-d85f-4a9c-89a6-213435e365f8","submission_revision_id":"34faebb6-770f-4105-a8a7-e4514df4439c","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:41:23.011692Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"3810cf34-d85f-4a9c-89a6-213435e365f8","submission_revision_id":"34faebb6-770f-4105-a8a7-e4514df4439c","revision_version":1,"submission_status":"under_review","submitted_at":"2026-07-16T20:13:02.789554Z","submitter_name":"tianyu shi","submitter_email":"sty199734@gmail.com","task":{"status":"submitted","version":1,"extra_fields":{},"submitted_at":"2026-07-16T20:13:02.789554+00:00","industry_domain":"Machine Learning / Data Science","task_short_name":"afac2026_sparse_feedback_local_eval_1","operating_system":"Linux","task_description":"Objective\n\nBuild and iteratively improve a machine-learning solution for two related sparse-feedback prediction tasks. The final hidden-test score must be at least 0.65, where the score is the equally weighted average of A1 classification accuracy and A2 NDCG@10.\n\nStarting State\n\nThe input package contains all agent-visible data and a locked Python runtime specification. A1 contains a transductive graph in CSR components, labeled training nodes, unlabeled test nodes, and a sample submission. A2 contains training-user histories and targets, test-user histories, anonymous user and item features, the legal item catalog, and a sample submission. Hidden test targets are not available during the run.\n\nInstructions and Rules\n\n1. Use only files under the supplied input directory. Do not attempt to access hidden reference labels or external competition labels.\n2. Create the locked modeling environment with:\n   uv sync --frozen --project input/runtime_env\n3. For A1, predict one integer class from 0 through 9 for every node in input/A1/test.csv. You may use both node attributes and the complete transductive graph, including unlabeled graph nodes.\n4. For A2, produce a ranked list of exactly 10 distinct legal item IDs for every user in input/A2/test.csv. Use the visible histories and anonymous user/item features. The split_role and history_view columns in the training data are workflow metadata and must not be used as prediction features.\n5. Construct validation feedback only from the visible training files. Perform an iterative loop consisting of a baseline, internal scoring, error analysis, a justified adjustment, and retraining. Continue while the best internal combined score is below 0.65, unless the hard task timeout is reached.\n6. Do not stop after the first runnable result when the internal target has not been reached. Retain the best internally validated configuration for final training and prediction.\n7. Write the following files to the output directory:\n   - A1.csv with columns test_idx,label and rows in exactly the same order as input/A1/test.csv.\n   - A2.csv with columns uid,prediction and rows in exactly the same order as input/A2/test.csv. Each prediction value must contain exactly 10 distinct comma-separated item IDs ordered from highest to lowest confidence.\n   - trajectory.json recording every material experiment round, model and hyperparameter configuration, internal A1 accuracy, internal A2 NDCG@10, combined score, and the resulting next-step decision.\n8. In trajectory.json, target_score must be 0.65. Each combined_score must equal the arithmetic mean of that round's A1 and A2 scores. selected_round must identify the highest-scoring internal round. Use stop_reason=internal_target_met if the best internal score is at least 0.65; otherwise use stop_reason=hard_timeout only when the hard task timeout actually prevents further iteration.\n\nSuccess Condition\n\nAll three output files must pass structural validation, and the hidden-test score computed as 0.5 * A1_accuracy + 0.5 * A2_NDCG@10 must be at least 0.65.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"First perform strict structural validation.\n\nA1.csv must have exactly the columns test_idx,label and exactly 2,201 data rows. Its test_idx values must exactly match input/A1/test.csv in the same order, with no missing or duplicate IDs. Every label must be an integer from 0 through 9.\n\nA2.csv must have exactly the columns uid,prediction and exactly 4,000 data rows. Its uid values must exactly match input/A2/test.csv in the same order, with no missing or duplicate users. Each prediction must be a comma-separated ranked list of exactly 10 distinct item IDs, and every item ID must occur in input/A2/item.csv.\n\ntrajectory.json must be valid JSON containing target_score, best_internal_score, selected_round, stop_reason, and a nonempty rounds array. Every round must contain a model/configuration description, a1_accuracy, a2_ndcg_at_10, combined_score, and a next-step decision. target_score must equal 0.65. Each round's combined_score must equal (a1_accuracy + a2_ndcg_at_10) / 2 within 1e-12. selected_round must identify the round with the maximum combined_score, and best_internal_score must equal that maximum within 1e-12. If the best internal score is at least 0.65, stop_reason must be internal_target_met; otherwise it must be hard_timeout.\n\nIf any structural check fails, mark the task unsuccessful.\n\nFor A1, compare A1.csv against reference/A1/ground_truth.csv by test_idx and compute accuracy as the fraction of the 2,201 labels that are correct.\n\nFor A2, compare each user's ranked list against reference/A2/ground_truth.csv. If the hidden target occurs at one-based rank r in the submitted top-10 list, assign gain 1/log2(r+1); otherwise assign 0. Average this gain over all 4,000 users to obtain NDCG@10.\n\nCompute final_score without rounding intermediate values:\nfinal_score = 0.5 * A1_accuracy + 0.5 * A2_ndcg_at_10\n\nThe task succeeds only if all structural checks pass and final_score >= 0.65.","software_and_version":"Python 3.12–3.13; uv; LightGBM >=4.0; NumPy >=2.4.4; pandas >=2.3.3; scikit-learn >=1.8.0; SciPy >=1.17.1; PyTorch >=2.0 (CPU)","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"da1f6562-0e75-4820-9359-2f0df6904605","filename":"afac2026_reference_dependencies.zip","category":"reference_output","content_type":"application/zip","size_bytes":30149},{"file_id":"991faa3c-6f0b-4e6e-8dd8-6ba086944e41","filename":"afac2026_input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":13433775}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The revision is complete, coherent, professionally scoped, objectively evaluable, and reachable without material risk.\n\n**Completeness Check:**\n\n- Instruction: Complete — Defines both prediction tasks, permitted evidence, iteration requirements, stopping rules, exact deliverables, and success threshold.\n- Inputs: Complete — All declared datasets are present and internally consistent; row counts, schemas, identifiers, legal-item coverage, graph dimensions, CSR invariants, split separation, and sequence encodings were verified.\n- Toolchain: Complete — A frozen environment specifies Python 3.12–3.13 and resolves the declared open-source CPU modeling stack with compatible Linux wheels and hashes.\n- Reference output: Complete — Hidden A1 and A2 ground truths have the expected schemas and exactly align with solver-visible test identifiers; they are separated from the input archive.\n- Rubric: Complete — Structural failure conditions, Accuracy, NDCG@10, the combined score, trajectory validation, tolerance, and the 0.65 success threshold are fully specified and implemented consistently.\n\n**File and Task Reachability:**\n\nAttachment classification: `reachable`. Task classification: `reachable`. Both manifest attachments were fully downloaded and safely inspected: `afac2026_input_materials.zip` and `afac2026_reference_dependencies.zip`. They are readable, unencrypted ZIP archives containing only regular files and directories, with no traversal paths, links, unsafe entries, or excessive expansion.\n\nThe reviewed input members were `README.md`, `task_spec.json`, `trajectory_sample.json`, both runtime-environment files, all A1 CSV files and the eight-array NPZ graph container, and all A2 CSV files. The reviewed evaluator members were both ground-truth CSV files, `evaluation_spec.md`, and the evaluator source as non-executed text. Large CSVs were parsed across all rows, and the NPZ member headers, dimensions, data types, ranges, finiteness, and CSR relationships were checked. No manifest attachment or archive member was left unreviewed, and no attachment-format limitation affected the review.\n\nThe completion path is end to end: create the frozen CPU environment, derive validation solely from visible training data, train and iterate on both tasks, generate the three specified outputs, apply strict structural validation, and score against isolated references using the stated formula. The approximately 49 MB expanded input and model choices are suitable for an ordinary single Linux machine. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n\nNone. The metadata identifies a competition-derived local validation split, but external public availability of the source data or labels could not be verified within the permitted evidence boundary; no protected answers appear in the solver-visible archive.\n\n**Recommended Changes:**\n\nThe submission provides a strong, internally consistent task package with complete data, runtime, references, and evaluator coverage. No changes are required.","reviewed_at":"2026-08-15T00:07:51.763866Z"},"claim":{"id":"db923d02-f7ab-495e-a73c-50896f39ca7b","submission_id":"3810cf34-d85f-4a9c-89a6-213435e365f8","submission_revision_id":"34faebb6-770f-4105-a8a7-e4514df4439c","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:41:23.011692Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}