{"submission_id":"a1703700-4221-4feb-8afe-fd2422a33193","submission_revision_id":"a8a7e3bd-8e0e-4ebd-b751-6630055df5ad","revision_version":13,"submission_status":"submitted","submitted_at":"2026-08-06T09:32:27.756435Z","submitter_name":"Yuanze Lin","submitter_email":"lin357823511@gmail.com","task":{"status":"submitted","version":13,"extra_fields":{"ai_draft":{"source":"submissions_workbench","status":"ready","preview":{"domain":"Image Editing / Generative Vision Research","task_goal":"Fine-tune a mask-conditioned inpainting diffusion model, select best trial, and export an ONNX pipeline with replay parity.\n\nStarting state: A base pretrained inpainting UNet checkpoint, a labeled image/mask/edit-prompt dataset split into train/val/test, and a config describing conditioning schema are provided as raw working files.\n\nInstructions and rules: Fine-tune the inpainting model over at least 6 trials varying learning rate, guidance scale, and mask-dropout augmentation, selecting the best checkpoint by validation LPIPS/CLIP-consistency tradeoff on masked regions. Export the selected checkpoint to ONNX and verify that ONNX inference reproduces PyTorch outputs within a bounded pixel tolerance on held-out test triplets. Do not alter the conditioning schema or evaluation masks provided in the config.","short_name":"inpaint_finetune_export","input_files":"- images_masks_train/ (PNG folder, ~1200 image+mask pairs): Training images with corresponding binary masks and edit prompts used to fine-tune the inpainting model\n- images_masks_val_test.csv (CSV, ~5MB): Maps val/test image_id to mask filename, edit prompt text, and split label; defines the authoritative evaluation set\n- base_inpaint_unet.safetensors (SAFETENSORS, ~3.5GB): Pretrained UNet weights that serve as the fine-tuning starting point and conditioning architecture reference","output_files":"- key_metrics.tsv (TSV) [graded]: Per-trial validation LPIPS/CLIP-consistency scores, selected trial id, and ONNX-vs-PyTorch pixel parity statistics on the test split","operating_system":"Linux","software_licensing":"Free/Open Source","verification_method":"key_metrics.tsv: parse/schema pass; required fields = 100%; records > 0.\nkey_metrics.tsv: exact match for ids/labels; numeric delta <= 5% or domain tolerance; blocking mismatches = 0.\nCross-file check: outputs stay traceable to inputs (images_masks_train/, images_masks_val_test.csv); mismatch count = 0.\nOverall: machine-readable = true; constraint violations = 0; formats remain comparable (TSV; declared constraints).","evaluation_criterion":"Method: script + rubric review\nMetric: primary reference-output comparison over structure, key values, consistency, and evidence grounding\nThreshold: pass when the primary reference output has no critical mismatch and supporting artifacts show no blocking inconsistency","software_and_version":"PyTorch 2.2, Diffusers 0.27, ONNX Runtime 1.17","tools_software_details":"Software: PyTorch 2.2, Diffusers 0.27, ONNX Runtime 1.17\nRun multi-trial fine-tuning and select best checkpoint by validation metrics (Tools: PyTorch, Diffusers)\nExport selected checkpoint to ONNX and verify inference parity on test split (Tools: ONNX Runtime)"},"warnings":["estimated_expert_hours \"\" does not meet the ≥5 hour requirement"],"createdAt":1785667394494,"formPrefill":{"extra_fields":{},"industry_domain":"Image Editing / Generative Vision Research","task_short_name":"inpaint_finetune_export","operating_system":"Linux","task_description":"# Task Goal\n\nFine-tune a mask-conditioned inpainting diffusion model, select best trial, and export an ONNX pipeline with replay parity.\n\nStarting state: A base pretrained inpainting UNet checkpoint, a labeled image/mask/edit-prompt dataset split into train/val/test, and a config describing conditioning schema are provided as raw working files.\n\nInstructions and rules: Fine-tune the inpainting model over at least 6 trials varying learning rate, guidance scale, and mask-dropout augmentation, selecting the best checkpoint by validation LPIPS/CLIP-consistency tradeoff on masked regions. Export the selected checkpoint to ONNX and verify that ONNX inference reproduces PyTorch outputs within a bounded pixel tolerance on held-out test triplets. Do not alter the conditioning schema or evaluation masks provided in the config.\n\n# Input Files\n\n- images_masks_train/ (PNG folder, ~1200 image+mask pairs): Training images with corresponding binary masks and edit prompts used to fine-tune the inpainting model\n- images_masks_val_test.csv (CSV, ~5MB): Maps val/test image_id to mask filename, edit prompt text, and split label; defines the authoritative evaluation set\n- base_inpaint_unet.safetensors (SAFETENSORS, ~3.5GB): Pretrained UNet weights that serve as the fine-tuning starting point and conditioning architecture reference\n\n# Output Files\n\n- key_metrics.tsv (TSV) [graded]: Per-trial validation LPIPS/CLIP-consistency scores, selected trial id, and ONNX-vs-PyTorch pixel parity statistics on the test split\n\n# Tools / Software Details\n\nSoftware: PyTorch 2.2, Diffusers 0.27, ONNX Runtime 1.17\nRun multi-trial fine-tuning and select best checkpoint by validation metrics (Tools: PyTorch, Diffusers)\nExport selected checkpoint to ONNX and verify inference parity on test split (Tools: ONNX Runtime)\n\n# Evaluation Criterion\n\nMethod: script + rubric review\nMetric: primary reference-output comparison over structure, key values, consistency, and evidence grounding\nThreshold: pass when the primary reference output has no critical mismatch and supporting artifacts show no blocking inconsistency","software_licensing":"Free/Open Source","verification_method":"key_metrics.tsv: parse/schema pass; required fields = 100%; records > 0.\nkey_metrics.tsv: exact match for ids/labels; numeric delta <= 5% or domain tolerance; blocking mismatches = 0.\nCross-file check: outputs stay traceable to inputs (images_masks_train/, images_masks_val_test.csv); mismatch count = 0.\nOverall: machine-readable = true; constraint violations = 0; formats remain comparable (TSV; declared constraints).","software_and_version":"PyTorch 2.2, Diffusers 0.27, ONNX Runtime 1.17"},"draftSession":"f8ac77e4-de79-4954-bbd7-ca4cb89bf8a6"},"difficulty_self_test":{"model":"Claude Opus 4.8","harness":"Claude Code","estimated_score":"10–20%"}},"submitted_at":"2026-08-06T09:32:27.756435+00:00","industry_domain":"Image Editing / Generative Vision Research","task_short_name":"inpaint_finetune_export","operating_system":"Linux","task_description":"# Task Goal\n\nObjective: Fine-tune a mask-conditioned inpainting diffusion model over a pinned\ntrial grid, select the best checkpoint on a validation LPIPS/CLIP-consistency\ntradeoff, and export it to ONNX with a replay-parity proof.\n\n# What to do -- the whole task in six steps\n\n  1. Run EXACTLY the 8 trials pinned in task_config.json ->\n     trials.pinned_trial_grid (each fixes lr, mask-dropout, guidance, seed).\n  2. Select by argmin(val_lpips_masked - 0.25 * val_clip_consistency), ties\n     broken by ascending trial_id.\n  3. Load the selected trial's tensors onto a pristine backbone and export to\n     ONNX: opset 17, dynamic batch axis, external-data format.\n  4. Replay the 8-step DDIM loop over all 200 test samples twice -- once with\n     PyTorch, once with ONNX Runtime, BOTH ON CPU, from identical seeded noise,\n     recording the latent at every step.\n  5. Write key_metrics.tsv, test_parity.parquet and test_metrics.parquet.\n  6. Submit all eight trial checkpoints and the exported graph alongside them.\n\nProvisioning: one GPU with >= 12 GB VRAM (16 recommended), ~10 GB free disk.\nThe task makes no network call at runtime; the Python environment must be\npre-provisioned, or installed with network at setup time only -- exact pinned\nrecipe under Tools / Software Details.\n\nEverything below is detail on those six steps and the rules they must satisfy.\nThe same six steps appear in the shipped inputs as README.md -> \"What to do\".\n\n\"Fine-tune\" here is PARAMETER-EFFICIENT and the scope is PINNED, not a choice\nleft to the solver: exactly 56,252,416 of the 945,393,668 parameters train\n(6.0%) -- the cross-attention key/value projections, the 9-channel mask\nconditioning conv, and the text-context projection (the Custom Diffusion recipe,\nKumari et al. CVPR 2023, plus conv_in because that is where the mask enters).\nThe exact set is task_config.json -> training.trainable_scope. The grader\nenforces it: any tensor outside that set, or a differing parameter count, is\nrejected. Updating all 945.4M weights is a FAILING submission, not a more\nthorough one.\n\nStarting state: A RANDOMLY-INITIALISED 945.4M-parameter 9-channel inpainting\nUNet (base_inpaint_unet.safetensors, 3.52 GB - not a pretrained checkpoint);\n1200 PROCEDURALLY-RENDERED training triplets (image PNG + binary mask PNG + edit\nprompt) whose masks are exact rather than annotated; 400 held-out triplets split\n200 val / 200 test; precomputed latent caches so no training step decodes a PNG;\nfive frozen auxiliary models (latent encoder/decoder, perceptual net, prompt\nencoder, alignment proxy) that DEFINE the metrics; and task_config.json\nspecifying the immutable conditioning schema, the noise schedule, the pinned\ntrial grid, the selection rule, the export rules and the runtime budget.\n\nWhat this task does and does not measure, stated plainly: the backbone is\nrandomly initialised, so no amount of correct work will produce good inpainting,\nand \"fine-tune\" here means the MECHANICS of parameter-efficient adaptation\nrather than improving a pretrained model. The quality metrics exist to confirm\ntraining happened and moved in the right direction, not to rank aesthetic\nresults. The task is graded on pipeline correctness -- scope compliance,\nselection under a stated rule, external-data ONNX export, and bit-level\ncross-runtime parity. That is deliberate: it makes the task fully offline,\nreproducible on any machine, and free of pretrained-weight licensing, at the\ncost of the quality dimension.\n\nInstructions and rules: Run EXACTLY the 8 pinned trials in task_config.json ->\ntrials.pinned_trial_grid, which fix trial_id, learning rate, mask-dropout\nprobability, guidance scale and seed. Select the best trial by\nargmin(val_lpips_masked - 0.25 * val_clip_consistency), ties broken by ascending\ntrial_id. Load that trial's fine-tuned tensors onto a pristine backbone and\nexport to ONNX at opset 17 with a dynamic batch axis - the model is 3.52 GB, past\nthe 2 GB protobuf ceiling, so external-data format is required. Then replay the\n8-step DDIM inpainting loop twice over all 200 test samples, once with PyTorch\nand once with ONNX Runtime BOTH ON CPU, from identical seeded noise, recording\nthe latent at every step. Do not alter the conditioning schema or the evaluation\nmasks. Report per-trial metrics, per-test-sample quality and per-sample-per-step\nexport parity so results are machine-checkable, not narrative. Submit the\nfine-tuned tensors from all eight trials and the exported graph alongside the\ntables - the grader opens them, re-derives the reported numbers from them, and\nfails a submission whose tables are not backed by the artifacts they describe.\n\n# Input Files\n\nimages_masks_train/ (PNG dir, 132 MB): 1200 procedurally-rendered triplets -\n  256x256 RGB image, binary mask (1 = inpaint), and an edit prompt describing the\n  masked object's colour, material, shape and position. Renders, not photographs;\n  the mask is exact and the prompt is exact.\nimages_masks_val_test/ (PNG dir, 44 MB): the 400 held-out image+mask pairs.\nimages_masks_val_test.csv (CSV): THE AUTHORITATIVE EVALUATION TABLE - image_id,\n  image file, mask file, edit prompt, split label (val/test), per-sample\n  noise_seed, mask area fraction. Nothing re-derives the split from filenames.\nbase_inpaint_unet.safetensors (SAFETENSORS, 3.52 GB): randomly-initialised\n  945.4M-param 9-channel inpainting UNet with a calibrated output head. NOT a\n  pretrained checkpoint.\ntrain_latents.npz / val_test_latents.npz: precomputed clean and masked-image\n  latents - no PNG decoding required at train or eval time.\nlatent_encoder / latent_decoder / perceptual_net / prompt_encoder /\n  alignment_proxy .safetensors: the five frozen evaluation dependencies. The\n  metrics ship as weights, so two correct runs are comparable.\nmodeling_inpaint_unet.py, modeling_aux.py, dataset_utils.py: architecture, the\n  9-channel conditioning helper, the fine-tuning scope, metric functions, data\n  helpers.\nunet_config.json, task_config.json, README.md.\n\n# The conditioning schema (immutable)\n\nInpaintUNet.forward(x, timestep, context), where x is 9 channels:\n\n    channels 0:4   noisy latent          z_t\n    channel  4     binary mask           m      (1 = inpaint, 0 = keep)\n    channels 5:9   masked-image latent          E(x * (1 - m))\n\ncontext is (B, 16, 768) from the frozen prompt encoder; the all-PAD prompt is the\nnull embedding used as the unconditional branch of classifier-free guidance.\nRe-ordering the channels or flipping the mask polarity invalidates the calibrated\nconv_in and the exported graph.\n\n# Output Files\n\nkey_metrics.tsv (TSV) [graded]: one row per trial - trial_id, learning_rate,\n  mask_dropout, guidance_scale, seed, train_steps, batch_size, trainable_params,\n  final_train_loss, n_val_samples, val_lpips_masked, val_clip_consistency,\n  val_selection_score, weight_delta_all_trainable, checkpoint_sha256, selected,\n  onnx_opset, n_test_samples, n_parity_samples, n_denoise_steps,\n  onnx_parity_max_abs_diff, onnx_parity_mean_abs_diff,\n  onnx_parity_max_abs_pixel_diff, onnx_parity_pass, test_lpips_masked_mean,\n  test_clip_consistency_mean.\ntest_parity.parquet (PARQUET) [graded]: one row per (parity sample x denoising\n  step) - sample_id, step_index, max_abs_diff, mean_abs_diff, rel_l2,\n  parity_pass, plus PyTorch and ONNX latent summary statistics. 200 x 8 = 1600\n  rows.\ntest_metrics.parquet (PARQUET) [graded]: one row per test sample - sample_id,\n  split, edit_prompt, mask_area_frac, lpips_masked, clip_consistency. 200 rows.\ntrained_checkpoints/trial_01..08_finetuned.safetensors (SAFETENSORS, 215 MB\n  each, 1.7 GB total) [graded artifact]: the fine-tuned tensors from EVERY trial,\n  not just the selected one.\nselected_unet.onnx + selected_unet.onnx.data (ONNX, 0.9 MB graph + 3.52 GB\n  external weights) [graded artifact]: the exported graph.\nreference_baseline.json: untrained-backbone metrics, for the directional rule.\n\nBoth artifact groups are graded by EXECUTION, not inspection. The grader opens\nthem, rebuilds base+checkpoint itself, re-derives your reported numbers, and\nre-evaluates all eight checkpoints on the validation split. Tables that are not\nbacked by the artifacts they describe do not pass, however self-consistent they\nare, and `selected` has to be earned by the checkpoint you shipped in that slot\nrather than asserted. Full check list is in the evaluation criterion.\n\n# Tools / Software Details\n\nSoftware: PyTorch 2.2.2+cu121, Diffusers 0.27.2, ONNX Runtime 1.17.1, ONNX 1.16.0,\npyarrow 15.0.2, safetensors 0.4.3 (Python 3.10, Linux x86_64, CUDA 12.1)\n\nEnvironment provisioning -- to be unambiguous about \"offline\":\nThe TASK is offline. It downloads no dataset, no pretrained weights, no model\nhub artifact, and makes no network call at any point during training,\nevaluation, export or the parity replay. Everything it consumes ships in\ninput_materials.zip.\n\nThe ENVIRONMENT is a separate matter. The stack above must exist before the\nagent starts. If the evaluation image already provides it, no network is needed\nat all. If it does not, provisioning needs network ACCESS AT SETUP TIME ONLY,\nand this is the exact, pinned recipe (Python 3.10, Linux x86_64, CUDA 12.1):\n\n  pip install torch==2.2.2 torchvision==0.17.2 \\\n      --index-url https://download.pytorch.org/whl/cu121\n  pip install diffusers==0.27.2 transformers==4.39.3 accelerate==0.28.0 \\\n      safetensors==0.4.3 huggingface_hub==0.22.2 onnx==1.16.0 \\\n      onnxruntime==1.17.1 pyarrow==15.0.2 pandas==2.2.2 \"numpy<2\" \\\n      Pillow==10.3.0 scipy==1.13.1\n\nonnxruntime (CPU build) is the one that matters for the graded parity pass;\nonnxruntime-gpu is NOT required and must not replace it, because parity is\nspecified CPU-vs-CPU. No package is fetched from Hugging Face at runtime --\nhuggingface_hub is a transitive requirement of diffusers, not a download path.\n\nOperating System: Linux (Ubuntu 22.04, x86_64)\n\nGPU: minimum 12 GB VRAM, 16 GB recommended. Measured peak at batch 8: 6.72 GiB\nallocated / 7.58 GiB reserved (only 6% of parameters train, which is why a\n3.52 GB model does not need 48 GB). Tested on RTX 6000 Ada, CUDA 12.1, Ubuntu\n22.04.\n\nCPU: the ONNX parity pass must run on CPU for BOTH runtimes (see Evaluation\nCriterion). EXPECTED WALL-CLOCK for that pass, correctly configured, is ~26\nminutes for 200 samples x 8 steps through both runtimes (measured; it is also\ntask_config.json -> runtime_budget.cpu_parity_minutes). Runtime configuration\nfor both libraries is left to you and it is not a detail: on a many-core host\nthe gap between a well-chosen and a default setting is more than an order of\nmagnitude. If your pass is tracking toward hours rather than ~26 minutes, stop\nand fix the configuration rather than waiting it out.\n\nDisk: ~4 GB for inputs, ~4 GB scratch for the exported ONNX model, ~1.8 GB for\nthe eight trial checkpoints (225 MB each). Peak ~10 GB with all three resident.\nThe backbone is deliberately past the 2 GB protobuf ceiling, which is what makes\nexternal-data ONNX export mandatory rather than optional.\nNetwork: none AT RUNTIME -- no weights, datasets or hub artifacts are ever\nfetched. Environment provisioning is the only step that may need network, and\nonly if the evaluation image lacks the pinned stack; see above.\n\nMeasured rates on the reference run, for planning: training 0.160 s/step at\nbatch 8; validation 27 s for 200 samples at 8 DDIM steps with guidance. The CPU\nreplay rate depends heavily on how you configure the two runtimes, so no single\nnumber is quoted -- see the CPU note above. Total runtime is in the Expected\nCompletion Time field.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"AUDIENCE NOTE: this field is the GRADING RUBRIC -- it specifies what the\nevaluator's automated grader must check. It is not the agent's instructions;\nthose are the Task Description field, whose \"What to do\" section states the\nwhole task in six steps. The volume below is the cost of leaving no room for\nsubjective judgement in grading, and none of it is work the agent has to parse\nin order to do the task.\n\nMethod: automated script. grading/grade.py ships with the reference bundle.\nExit code 0 = pass, 1 = fail. No human judgement, no prose thresholds.\n\nTHE REFERENCE BUNDLE VERIFIES ITSELF. Unzip reference_output.zip and run its own\ngrader on its own artifacts -- no other setup, no reference flag, nothing from\nthe author's machine:\n\n  unzip reference_output.zip && cd reference_output\n  python grading/grade.py --submission . --inputs <inputs_dir> \\\n      --baseline reference_baseline.json\n\nMeasured on a fresh extraction: 68/68, with the parity replay reproducing at\n0.00e+00 (bit-exact) from the shipped checkpoint and the shipped ONNX graph.\nThat is the same code path a submission is graded through, so the bundle is not\nmerely inspectable -- it is executable evidence.\n\nRun:\n  python grading/grade.py --submission <agent_out> --inputs inputs/ \\\n      --baseline reference/reference_baseline.json --reference reference/\n\nMetric: 76 checks with --reference (68 without). PASS requires ALL of them.\n\nThe grader does NOT grade the submitted tables in isolation. It opens the\nsubmitted checkpoints and the submitted ONNX graph, and re-derives the reported\nnumbers from them by executing both runtimes itself. Roughly 8 minutes of that\ntotal is the replay; --recompute-n 0 skips it and --skip-artifacts drops the\nwhole tier, but neither is a real verdict and the reference is graded with both\nenabled.\n\nSCHEMA\n  1. key_metrics.tsv parses as TSV, has a header, and has EXACTLY 8 data rows\n     with 8 unique trial_ids. The grid is pinned at eight, so \">= 8\" was the\n     wrong shape: it admitted an invented ninth trial, and a duplicated trial_id\n     let eight rows describe seven real runs.\n  2. It contains all 26 required columns by exact name (list in Output Files).\n  3. test_parity.parquet contains sample_id (str), step_index (int),\n     max_abs_diff, mean_abs_diff, rel_l2 (float), parity_pass (bool), AND the\n     four per-step latent summaries (torch/onnx latent mean and std), rows > 0.\n  4. test_metrics.parquet contains sample_id (str), lpips_masked,\n     clip_consistency (float), rows > 0.\n\nSELF-CONSISTENCY\n  5. Exactly one row has selected=true.\n  6. That row carries every export/parity/test field NON-NA. Checked positively\n     and BEFORE any numeric parse, so a submission that leaves them blank gets a\n     readable FAIL rather than a grader traceback.\n  7. onnx_* fields are NA on all unselected trials.\n  8. The selected row is argmin(val_selection_score); ties by trial_id ascending.\n  9. All 8 trials from task_config.json -> trials.pinned_trial_grid are present\n     with EXACTLY the pinned learning_rate, mask_dropout, guidance_scale, seed.\n 10. learning_rate, mask_dropout and guidance_scale each take >= 3 distinct\n     values across the sweep.\n 11. Every trial reports train_steps >= 1000.\n 12. All 8 checkpoint_sha256 values are distinct.\n 13. All 8 weight_delta_all_trainable values are non-zero AND distinct.\n     <- anti-fabrication: one checkpoint copied into eight slots collides here,\n        and a trial that never ran reports a delta of exactly zero.\n\nCROSS-FILE JOIN\n 14. test_parity rows == n_parity_samples * n_denoise_steps == 200 * 8 == 1600.\n 15. test_metrics rows == n_test_samples == 200.\n 16. set(test_metrics.sample_id) == the 200 test image_ids in\n     images_masks_val_test.csv, exactly.\n 17. set(test_parity.sample_id) is a subset of those test ids, size n_parity.\n 18. step_index covers 0..7 completely for every sample_id.\n 19. ANCHOR 1: max(test_parity.max_abs_diff) == the selected row's\n     onnx_parity_max_abs_diff, within 1e-9.\n 20. ANCHOR 2: mean(test_metrics.lpips_masked) == the selected row's\n     test_lpips_masked_mean, within 1e-9.\n     <- THREE independently generated files must agree to nine decimal places.\n\nABSOLUTE NUMERIC THRESHOLDS\n 21. max(test_parity.max_abs_diff) < 5e-4, with BOTH runtimes run on CPU.\n 22. parity_pass true on >= 99% of the 1600 rows.\n 23. onnx_parity_max_abs_pixel_diff < 2/255 (one 8-bit level on [-1,1]).\n     <- this is the \"bounded pixel tolerance\" the task asks for.\n 24. No NaN or Inf in any numeric column of any of the three files.\n 25. >= 10% of the 1600 max_abs_diff values are distinct. A synthesised parquet\n     repeats a handful of values, or one per step; the reference measures 302\n     distinct (18.9%). The bar is low because max_abs_diff is a max over\n     float32-precision differences and genuinely collides.\n 26. val_lpips_masked is finite and > 0 on every trial row (it is a distance).\n\nDIRECTIONAL\n 27. Selected trial's val_lpips_masked < baseline's, per reference_baseline.json.\n 28. Selected trial's val_selection_score < baseline's.\n\nAGAINST THE REFERENCE BUNDLE (--reference)  <- the reference/rubric tie\n 29. Trial id sets match reference/key_metrics.tsv EXACTLY.\n 30. learning_rate, mask_dropout, guidance_scale, seed identical to the\n     reference on every trial row (exact, tolerance 1e-9).\n 31. onnx_opset, n_test_samples, n_parity_samples, n_denoise_steps identical.\n 32. Submission AND reference parity are both < 5e-4.\n\nREPORTED, NOT GATED\n  - val_clip_consistency vs baseline, per-step parity drift, and the LPIPS delta\n    against the reference are printed with an \"(informational)\" suffix. They must\n    be present and finite; their VALUE is never a pass condition.\n\nEXPECTED VALUE RANGES (so \"5% of what?\" has an answer):\nThese are TASK-LOCAL metrics computed with the frozen scorers shipped in the\ninputs. masked LPIPS is NOT VGG-LPIPS; CLIP-consistency is NOT OpenAI CLIP. Both\nare O(0.1-0.5), and a grader assuming published LPIPS ranges will misjudge them.\n  val_lpips_masked, untrained baseline    0.3314\n  val_lpips_masked, trained trials        0.1974 - 0.2944\n  val_clip_consistency, baseline          0.2464\n  val_clip_consistency, trials            0.2158 - 0.4458\n  val_selection_score, baseline           0.2698\n  val_selection_score, trials             0.0859 - 0.2332\n  test_lpips_masked_mean (selected)       0.1923\n  test_clip_consistency_mean (selected)   0.4082\n  onnx parity max_abs_diff (CPU)          7.629e-05   <- the graded number\n  onnx parity max_abs_pixel_diff (CPU)    6.455e-05   = 0.0082 of an 8-bit level\n  onnx parity max_abs_diff (CUDA)         1.611e-02   <- why CUDA is disallowed\nAny val_lpips_masked outside 0.05 - 1.0 indicates a broken pipeline.\n\nWHY QUALITY METRICS ARE NOT VALUE-MATCHED (this replaces \"numeric delta <= 5%\"):\nEight trials with independent seeds and hardware-dependent kernel selection do\nnot reproduce a reference LPIPS to any fixed tolerance, so a 5% band would fail\ncorrect work. Deterministic quantities -- parity, ids, hyperparameters, counts,\nthe two anchors -- are matched exactly; stochastic ones are graded on DIRECTION\nagainst the shipped untrained baseline. Note that the backbone ships randomly\ninitialised, so the directional checks are a sanity floor confirming training\nhappened at all, not a measure of inpainting quality. The difficulty of this\ntask is in the engineering pipeline -- scope compliance, export, cross-runtime\nparity -- and the grading is weighted accordingly.\n\nWHY CPU, AND WHERE 5e-4 COMES FROM (measured, both parquets ship in the bundle):\n  CPU  vs CPU  : 7.629e-05, 1600/1600 rows pass  = 0.0082 of an 8-bit level\n  CUDA vs CUDA : 1.611e-02,    0/256 rows pass   = 2.92 levels, visible\nORT's CUDA provider picks different kernels than PyTorch, so a CUDA comparison\nmeasures kernel choice, not export fidelity. 5e-4 sits 6.5x above the reference\n(headroom for a different BLAS or core count) and 32x below the CUDA gap.\n\nSUBMITTED ARTIFACTS -- CHECKPOINTS  (the grader opens the files)\n  A1. All 8 trial checkpoints are present under trained_checkpoints/.\n  A2. sha256 of each FILE equals that trial's checkpoint_sha256 column. Without\n      this the hash column is decoration -- 8 distinct hex strings are free.\n  A3. The 8 files are pairwise distinct on content.\n  A4. Every tensor name in every checkpoint lies inside the immutable trainable\n      scope from task_config (attn2.to_k, attn2.to_v, conv_in, context_proj),\n      and every key exists in the backbone.\n  A5. Summed parameter counts equal the reported trainable_params (56,252,416).\n  A6. ||W - W_base||_F recomputed against the pristine backbone is non-zero on\n      every trial, pairwise distinct, and matches weight_delta_all_trainable to\n      1e-6 relative.\n\nSUBMITTED ARTIFACTS -- ONNX GRAPH\n  A7.  selected_unet.onnx and selected_unet.onnx.data both present.\n  A8.  The graph parses (structure only; the 3.5 GB of weights are not loaded).\n  A9.  ai.onnx opset == 17, and equals the TSV's onnx_opset.\n  A10. Inputs are exactly [x, timestep, context], output [noise_pred].\n  A11. x is 4-D with 9 channels and context is [*, 16, 768] -- the conditioning\n       schema is immutable, and a \"simplified\" conditioning would otherwise sail\n       through a metrics-only grader.\n  A12. A symbolic batch axis on every input and output.\n  A13. No large tensor is inline, the protobuf is under the 2 GB ceiling, exactly\n       one external data file is referenced, and the highest offset+length\n       addressed by the graph fits inside the .data file actually shipped.\n\nRECOMPUTED FROM THE ARTIFACTS  (this is the anti-fabrication tier)\n  R1. ONNX Runtime reports providers == ['CPUExecutionProvider'] -- the backend\n      requirement is verified at runtime, not taken on trust.\n  R2. The grader rebuilds base+selected-checkpoint, replays a batch-aligned\n      window of the parity split through BOTH runtimes on CPU, and requires the\n      submitted per-(sample,step) max_abs_diff to match to 1e-6 relative. The\n      reference reproduces at 0.00e+00 -- bit-exact. The window must be\n      batch-aligned: MKL varies reduction order with batch shape, so replaying\n      2 samples that originally ran 8-at-a-time disagrees ~30% on identical\n      weights.\n  R3. That independently measured max|diff| is itself under 5e-4.\n  R4. The grader re-scores those samples with the frozen scorers and requires\n      the submitted lpips_masked / clip_consistency to match (reference: 0.00e+00).\n  R5. val_selection_score is RECOMPUTED as lpips - 0.25*clip on every row, and\n      the selected trial must be the argmin of the recomputed scores -- the\n      selection column is never taken on trust.\n  R6. WHICH window is replayed is the evaluator's choice (--recompute-offset,\n      default mid-split). Re-running at a different offset re-measures a\n      different slice of the submitted parquet, so a submission that replayed\n      one window honestly and invented the rest does not survive a second run.\n      test_grader.py exercises the same fabrication at two offsets.\n\nPER-TRIAL VALIDATION REBIND  (binds every checkpoint to its own row)\n  V1. All 8 submitted checkpoints cover an identical tensor-name set, so one\n      model instance can be reused across them without residue.\n  V2. Each of the 8 is loaded onto the pristine backbone and evaluated on the\n      FULL authoritative validation split (200 samples) at THAT trial's pinned\n      guidance_scale. Not a subset: the reported means are over all 200, so a\n      subset mean would not be comparable.\n  V3. Every trial's val_lpips_masked and val_clip_consistency must match what\n      its own checkpoint produces, within 2% relative.\n  V4. The selected trial must be the argmin of the RECOMPUTED selection scores,\n      not of the reported ones.\n\n  Why 2%: it absorbs cross-device kernel differences (the grader may not run on\n  the hardware that trained the sweep) while staying far tighter than what it\n  has to discriminate. Measured on the reference, the winning trial leads the\n  runner-up by 0.0494 on a score of 0.0859 -- a margin ~29x the tolerance band,\n  so selection stays decidable with enormous headroom.\n\n  This tier runs on CUDA when available (~4 min for all 8 trials); CPU-only is\n  ~1 h. Only the EXPORT PARITY claim requires both runtimes on CPU -- quality\n  scoring carries no such constraint.\n\nWHAT THESE TIERS EXIST TO STOP:\nCross-file anchors prove the three files were generated together, not that\nanything was computed. Two fabrication shapes survive anchors alone: tables with\nno artifacts behind them at all, and -- subtler -- eight genuinely trained\ncheckpoints reported with a validation table that does not describe them. Four\ncorruptions in test_grader.py are built to be exactly those submissions\n(`fabricated_parity`, `wrong_ckpt_for_selected`, `checkpoint_trial_swap`,\n`val_metrics_fabricated`); each passes every structural check and fails only on\nthe tier that re-derives the number from the artifact.\n\nThreshold: exit code 0. The reference bundle scores 76/76 with --reference.\n\nTIERED DIAGNOSTIC SCORING (--report tiered)\nPass/fail is unchanged -- a submission passes only by satisfying every check.\n--report tiered adds a breakdown so a partial result is legible:\n\n  Schema & parse 8 | Self-consistency 15 | Cross-file join 7 | Thresholds 5\n  Directional 3 | Vs reference 8 | Artifacts 17* | Recomputed 8* | Val rebind 5*\n\n  * = INTEGRITY tier. If any check in a starred tier fails, the score is 0 --\n  not the structural percentage the submission would otherwise have earned. A\n  forged table set that satisfies every schema, join and threshold check scores\n  0.0%; an honest run that missed a threshold keeps its partial credit.\n  Incompleteness is graded, fabrication is not. If an integrity tier is switched\n  off, the score is WITHHELD rather than reported.\n\ntest_grader.py asserts this, not just exit codes: the clean bundle must score\n100.0% with integrity VERIFIED, and all 18 fabrication corruptions must score\nexactly 0.0. Measured partial credit on honest defects: wrong argmin 90.0%, two\nselected rows 88.9%, dropped latent column 83.3%, deleted column 50.0%.\n\ngrading/test_grader.py additionally confirms 35 targeted corruptions are each\nrejected, and that none crashes the grader. They span: schema violations, broken\ncross-file anchors, blown thresholds, off-grid or duplicated hyperparameters,\ntampered/duplicated/missing checkpoints, out-of-scope trained tensors, a graph\nshipped without its external weights or re-stamped to opset 14, export fields\nclaimed by trials that never earned them, the parity fabrication replayed at two\ndifferent offsets, a checkpoint-to-trial swap with hashes and deltas swapped to\nmatch, and invented validation metrics behind otherwise genuine artifacts. The\nfull list, with each corruption's construction and the check it trips, is in\ngrading/test_grader.py -- it is executable, so it does not need restating here.","software_and_version":"PyTorch 2.2.2+cu121, Diffusers 0.27.2, ONNX Runtime 1.17.1, ONNX 1.16.0, pyarrow 15.0.2, safetensors 0.4.3 (Python 3.10, Linux x86_64, CUDA 12.1)","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"9cf9d2bf-926e-4f69-812e-d9c3374bbbf6","filename":"input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":4039368379},{"file_id":"7f3073d6-bfe7-47ff-90c2-7fe99c0e9501","filename":"reference_output.zip","category":"reference_output","content_type":"application/zip","size_bytes":5122826104}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The submission is complete, technically coherent, safely reachable, and supported by strong artifact-level verification, with only minor documentation drift.\n\n**Completeness Check:**\n- Instruction: Complete — Defines the exact trial grid, training scope, selection formula, export constraints, CPU parity procedure, and deliverables.\n- Inputs: Complete — Dataset counts, split table, latent caches, model architecture, configuration, and frozen model weights are present and structurally consistent.\n- Toolchain: Complete — Provides a pinned, open-source Linux/Python/CUDA environment, offline runtime requirements, hardware sizing, and installation recipe.\n- Reference output: Complete — Includes eight distinct checkpoint artifacts, complete metric tables, an externally stored ONNX model, baseline data, provenance summaries, and replay evidence.\n- Rubric: Complete — Covers schemas, joins, thresholds, checkpoint scope and hashes, ONNX structure, independent parity replay, validation rebinding, integrity failures, and partial diagnostic scoring.\n\n**File and Task Reachability:**\nThe attachment set is `reachable`, and the end-to-end task is `reachable`. Both declared attachments were inspected and match their ZIP/ZIP64 formats and manifest sizes. Their directories contain no encryption, traversal paths, absolute paths, duplicate names, symbolic links, device entries, or extreme expansion.\n\nThe input archive contains 3,221 members, including 3,200 PNGs. Inspected content included `README.md`, `task_config.json`, `unet_config.json`, the 400-row evaluation CSV, the 1,200-row training JSONL, all three supplied Python modules as static text, a representative matching RGB image and binary mask, both NPZ directories and all ten NPY headers, and headers from the backbone and five auxiliary Safetensors files. The CSV has 200 validation and 200 test rows with unique IDs. The caches contain the expected float16 `(N,4,32,32)` latents and uint8 `(N,256,256)` masks. The backbone header reports 945,393,668 float32 parameters.\n\nThe reference archive contains 46 members. Inspected content included the TSV, all three Parquet tables, baseline and trial JSON files, checkpoint fingerprints, ONNX metadata, CPU/CUDA parity summaries, relevant logs, grading modules, and solution modules as static text. All eight checkpoint headers expose the same 60 in-scope tensors totaling 56,252,416 parameters. The graded Parquets contain 200 metric rows and 1,600 CPU parity rows with the declared schemas. The ONNX graph directly confirms opset 17, symbolic batch axes, required input/output shapes, 479 external initializers, and exact accounting against the 3.780 GB external-data member.\n\nThe remaining individual image bodies, complete numerical tensor payloads, complete external-weight body, and ancillary authoring scripts were not exhaustively decoded. Uploaded programs were not executed; therefore, self-grader pass claims were assessed through static code, schema, graph, header, and log inspection rather than rerunning the submitted grader. This is a review-safety limitation, not an attachment defect.\n\nThe completion path is viable on one suitable machine: train the eight pinned trials on one GPU, evaluate and select the measured argmin, rebuild from a pristine backbone, export external-data ONNX, replay PyTorch and ONNX Runtime on CPU, emit the tables and checkpoints, then let the evaluator reopen all artifacts, replay selected samples, and rebind every validation row to its checkpoint.\n\nLicense keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nNo material changes are required. As optional documentation cleanup, update the input `README.md` to list all 26 TSV fields and all ten parity columns, and replace the retained older 45-check run log with output from the current 68/76-check grader.","reviewed_at":"2026-09-15T10:20:58.784514Z"},"claim":null}