{"submission_id":"4247d266-c595-4eaa-8954-c58ea8a04f53","submission_revision_id":"29d06879-2789-433d-bc8a-a4cdbcc70c33","revision_version":7,"submission_status":"submitted","submitted_at":"2026-08-15T16:52:48.911696Z","submitter_name":"Yuanze Lin","submitter_email":"lin357823511@gmail.com","task":{"status":"submitted","version":7,"extra_fields":{"ai_draft":{"source":"submissions_workbench","status":"ready","preview":{"domain":"Retail Supply Chain","task_goal":"Train and calibrate a scikit-learn model predicting which SKUs risk stockout before next replenishment cycle.\n\nStarting state: Analyst has raw POS transactions, DC inventory snapshots, and supplier lead-time extracts pulled from Snowflake into flat files, with no engineered features or labels yet.\n\nInstructions and rules: Engineer inventory-turnover and lead-time features per SKU-DC-week, define a forward-looking stockout label using the supplied replenishment cycle window, and train a calibrated classifier that outputs a risk score usable for prioritizing replenishment review. The model must avoid using post-cutoff data (no label leakage) and must report calibrated probabilities, not raw scores.","short_name":"sku_stockout_propensity","input_files":"- pos_transactions.csv (CSV, ~150MB): Daily SKU-level POS sales quantities by DC/store, used to compute demand velocity and turnover features\n- dc_inventory_snapshot.csv (CSV, ~40MB): Weekly on-hand and on-order inventory quantities per SKU-DC, used to build days-of-supply features and derive stockout labels\n- supplier_lead_time.csv (CSV, ~5MB): Supplier-level historical lead time and replenishment cycle length per SKU, used to define the forward-looking prediction horizon and lead-time-risk features","output_files":"- sku_risk_scores.csv (CSV) [graded]: SKU-DC-week level calibrated stockout probability scores with prediction cutoff date and feature snapshot id, the primary scoring artifact\n- model.pkl (PKL) [graded]: Trained and calibrated scikit-learn pipeline (feature transformer + classifier) used to regenerate scores deterministically","operating_system":"Linux","software_licensing":"Free/Open Source","verification_method":"sku_risk_scores.csv: parse/schema pass; required fields = 100%; records > 0.\nsku_risk_scores.csv: exact match for ids/labels; numeric delta <= 5% or domain tolerance; blocking mismatches = 0.\nmodel.pkl: cross-file totals and joins reconcile with sku_risk_scores.csv; mismatch count = 0.\nOverall: machine-readable = true; constraint violations = 0; formats remain comparable (CSV, PKL; declared constraints).","evaluation_criterion":"Method: script + rubric review\nMetric: primary reference-output comparison over structure, key values, consistency, and evidence grounding\nThreshold: pass when the primary reference output has no critical mismatch and supporting artifacts show no blocking inconsistency","software_and_version":"Python 3.11, scikit-learn 1.4, Snowflake (SQL), JupyterLab","tools_software_details":"Software: Python 3.11, scikit-learn 1.4, Snowflake (SQL), JupyterLab\nEngineer SKU-DC-week features and forward stockout labels (Tools: Snowflake, Python 3.11)\nTrain, calibrate, and validate propensity model with time-based split (Tools: scikit-learn 1.4, JupyterLab)"},"warnings":["estimated_expert_hours \"\" does not meet the ≥5 hour requirement"],"createdAt":1785679146116,"formPrefill":{"extra_fields":{},"industry_domain":"Retail Supply Chain","task_short_name":"sku_stockout_propensity","operating_system":"Linux","task_description":"# Task Goal\n\nTrain and calibrate a scikit-learn model predicting which SKUs risk stockout before next replenishment cycle.\n\nStarting state: Analyst has raw POS transactions, DC inventory snapshots, and supplier lead-time extracts pulled from Snowflake into flat files, with no engineered features or labels yet.\n\nInstructions and rules: Engineer inventory-turnover and lead-time features per SKU-DC-week, define a forward-looking stockout label using the supplied replenishment cycle window, and train a calibrated classifier that outputs a risk score usable for prioritizing replenishment review. The model must avoid using post-cutoff data (no label leakage) and must report calibrated probabilities, not raw scores.\n\n# Input Files\n\n- pos_transactions.csv (CSV, ~150MB): Daily SKU-level POS sales quantities by DC/store, used to compute demand velocity and turnover features\n- dc_inventory_snapshot.csv (CSV, ~40MB): Weekly on-hand and on-order inventory quantities per SKU-DC, used to build days-of-supply features and derive stockout labels\n- supplier_lead_time.csv (CSV, ~5MB): Supplier-level historical lead time and replenishment cycle length per SKU, used to define the forward-looking prediction horizon and lead-time-risk features\n\n# Output Files\n\n- sku_risk_scores.csv (CSV) [graded]: SKU-DC-week level calibrated stockout probability scores with prediction cutoff date and feature snapshot id, the primary scoring artifact\n- model.pkl (PKL) [graded]: Trained and calibrated scikit-learn pipeline (feature transformer + classifier) used to regenerate scores deterministically\n\n# Tools / Software Details\n\nSoftware: Python 3.11, scikit-learn 1.4, Snowflake (SQL), JupyterLab\nEngineer SKU-DC-week features and forward stockout labels (Tools: Snowflake, Python 3.11)\nTrain, calibrate, and validate propensity model with time-based split (Tools: scikit-learn 1.4, JupyterLab)\n\n# Evaluation Criterion\n\nMethod: script + rubric review\nMetric: primary reference-output comparison over structure, key values, consistency, and evidence grounding\nThreshold: pass when the primary reference output has no critical mismatch and supporting artifacts show no blocking inconsistency","software_licensing":"Free/Open Source","verification_method":"sku_risk_scores.csv: parse/schema pass; required fields = 100%; records > 0.\nsku_risk_scores.csv: exact match for ids/labels; numeric delta <= 5% or domain tolerance; blocking mismatches = 0.\nmodel.pkl: cross-file totals and joins reconcile with sku_risk_scores.csv; mismatch count = 0.\nOverall: machine-readable = true; constraint violations = 0; formats remain comparable (CSV, PKL; declared constraints).","software_and_version":"Python 3.11, scikit-learn 1.4, Snowflake (SQL), JupyterLab"},"draftSession":"e1f12b1e-e972-4a98-abb7-58047698596a"},"difficulty_self_test":{"model":"Claude Opus 4.8","harness":"Claude Code","estimated_score":"40–60%"}},"submitted_at":"2026-08-15T16:52:48.911696+00:00","industry_domain":"Retail Supply Chain","task_short_name":"sku_stockout_propensity","operating_system":"Linux","task_description":"# Task Goal\n\nBuild a calibrated SKU-DC-week stockout propensity model: engineer inventory-turnover\nand lead-time-risk features from raw retail feeds, define a forward-looking stockout\nlabel over each SKU's own replenishment cycle, and train a scikit-learn pipeline whose\noutput is a probability a replenishment planner can threshold directly.\n\n## Starting state\n\nYou have four files. Two fiscal years of a six-DC network: daily POS sales, weekly DC\ninventory positions, and the purchase-order history that sets each SKU's replenishment\ncycle. There are no engineered features, no labels, and no train/test split materialised\nanywhere.\n\n**Data provenance — read this before you read anything else.** These files are\n**synthetic**: deterministically simulated by `10_make_inputs.py` (seed 20260802), which\nships with the reference bundle. No real retailer, distribution centre, supplier,\npurchase order or point-of-sale record is involved, and no customer or commercial data\nof any kind. Demand, replenishment, lead times and short-shipment come from an\norder-up-to inventory simulation. The scenario is written as though an analyst had\npulled these extracts out of Snowflake into flat files, because that is the working\nsituation being modelled — but that is the *framing*, not a claim about where the bytes\ncame from. The same disclosure is in `schema.json` under `provenance`, and the seeded\ngenerator makes every row reproducible. Nothing about the modelling task depends on the\ndata being real; the latent structure below is what you are being asked to recover.\n\nThree properties of the data drive the work, and all three are documented in\n`schema.json`:\n\n1. **POS units are sales, not demand.** When a SKU-DC is out of stock the sale does not\n   happen and no row is written. Demand velocity computed straight off\n   `pos_transactions.csv` is therefore biased *downward* for exactly the SKU-DC-weeks\n   that ran out. `dc_inventory_snapshot.stockout_days` tells you which observations are\n   censored.\n\n2. **The prediction horizon is not constant.** `replenishment_cycle_days` is a per-SKU\n   property in {7, 14, 21, 28}, split 30.0 / 33.7 / 21.7 / 14.6 percent of scorable rows.\n   A 7-day-cycle SKU is being asked a one-week question and a 28-day-cycle SKU a\n   four-week one. Any single fixed horizon is the wrong horizon for at least 66% of\n   rows. Note the label is a threshold on a *sum*, so it is more forgiving than that\n   suggests: a fixed 2-week horizon still agrees with the correct label on 96.5% of\n   scorable test rows (1-week 95.9%, 4-week 90.5%). It is not a shortcut -- 3.5% of\n   78,864 rows is ~2,700 wrong labels, and the label column is checked with a required\n   mismatch count of zero -- but do not expect the error to be visually obvious.\n\n3. **Supplier behaviour is latent.** Suppliers pass through multi-week windows where\n   realised lead times run well above contract, and some short-ship — delivering less\n   than the PO quantity, with the balance cancelled rather than backordered. Nothing\n   flags either condition. Only the PO history reveals them, and only if it is trended\n   and differenced against nominal rather than averaged flat.\n\n## Instructions and rules\n\n1. **Engineer features at SKU-DC-week grain.** Inventory turnover, days of supply\n   against demand velocity, coverage against lead time, and supplier lead-time risk are\n   all worth building, and the reference uses all four. They are SUGGESTIONS, not a\n   checklist: grading is method-agnostic and scores the delivered probabilities, so any\n   feature set that meets the accuracy and calibration bars passes, whatever it is built\n   from. Be aware the suggestions are not individually load-bearing — dropping all 16\n   lead-time and coverage features from the reference panel costs only 0.0008 ROC-AUC,\n   because the remaining features recover most of that signal. Multi-window velocity\n   (the ratio of a short window to a long one) is what separates a SKU whose demand is\n   accelerating from one that is merely large.\n\n2. **Define the forward label exactly as specified below.** This is a specification, not\n   a modelling choice, because the grader recomputes it:\n\n   > For a cutoff week `w`, `horizon_weeks = replenishment_cycle_days // 7`.\n   > `stockout_label = 1` if the sum of `stockout_days` over fiscal weeks\n   > `w+1 … w+horizon_weeks` is **>= 3**, else 0. The cutoff week itself is not part of\n   > the window. Three or more out-of-stock days in the forward window is a *material*\n   > stockout; a one- or two-day blip is not actionable and is deliberately labelled 0.\n\n3. **Score exactly the scorable population.** A `(sku, dc, w)` row is scorable iff all\n   three hold:\n   - `w >= 14` — thirteen weeks of trailing history must exist;\n   - `w + horizon_weeks <= 104` — the entire forward window is observed. A row whose\n     window runs past the end of the data has an unknowable label and must **not** be\n     scored;\n   - `on_hand_units > 0` at the cutoff. This is the review-queue population: the model\n     ranks SKU-DCs that are still in stock. One already sitting at zero is an expedite,\n     not a forecast.\n\n   Deliver every scorable test-split row and no others. The row count is graded with a\n   required difference of zero.\n\n4. **Respect the cutoff.** Every feature for a row with cutoff date `D` must be\n   computable from `pos_transactions` rows with `txn_date <= D`, `dc_inventory_snapshot`\n   rows with `snapshot_date <= D`, and `supplier_lead_time` rows with `receipt_date <= D`.\n   No record dated after `D` may enter a feature — not directly, and not through a\n   rolling window, a group statistic, or a transformer fitted on the full panel. Test\n   labels are for final reporting only: not for fitting, not for model selection, not\n   for threshold selection, not for fitting the calibrator.\n\n   One trap is worth naming because the reference author fell into it. Fiscal week `w`\n   *ends* on `2024-01-07 + 7*(w-1)`, so if you map an arbitrary date to a week by\n   flooring `(date - week_end_of_week_1) / 7` you will assign Monday-to-Saturday\n   **after** a cutoff to the week that just ended, and every rolling window will\n   quietly contain six post-cutoff days. Anchor on the week's FIRST day instead:\n   `week = (date - 2024-01-01).days // 7 + 1`. Snapshot dates are week-ending Sundays\n   so both forms agree on them; `txn_date` and `receipt_date` are arbitrary and they\n   do not. Assert it on boundary dates before you trust any window.\n\n5. **Split on time, never at random.** Train = fiscal weeks 14–78. Validation = weeks\n   79–90. Test = weeks 91–104, trimmed per SKU by its own horizon — so in practice the\n   last scored cutoff is week 103 for a 7-day-cycle SKU and week 100 for a 28-day one,\n   and 13 distinct cutoff dates appear in the delivered file, not 14. A random split\n   scatters one SKU-DC's autocorrelated weeks across train and test and inflates every\n   number you report.\n\n6. **Report calibrated probabilities, not raw scores.** The deliverable is a probability\n   a planner thresholds directly, so it is graded on calibration as well as ranking.\n   Note the base rate drifts across the split — validation sits in a quieter trading\n   period than test — so a calibrator fitted on validation is being asked to extrapolate.\n   Choose the calibration approach on evidence and fit it on the validation split.\n\n7. **Make the scores regenerable.** `model.pkl` must reproduce `sku_risk_scores.csv`\n   from the feature matrix you ship, deterministically.\n\n# Input Files\n\n- **pos_transactions.csv** (CSV, 129 MB): 3,092,581 rows, one per POS transaction LINE.\n  Columns: `txn_date`, `sku_id`, `dc_id`, `store_id`, `units_sold`, `net_sales_amt`,\n  `promo_flag`. A given (`txn_date`, `sku_id`, `dc_id`, `store_id`) can appear on more\n  than one line -- 87,734 rows (2.84%) share a key with another row and 53,552 are\n  byte-identical to another row. That is a till feed, not corruption: **aggregate with a\n  SUM, do not de-duplicate.** Dropping the repeats discards 114,850 of 8,360,703 units\n  and biases every velocity and turnover feature downward. Source for demand velocity,\n  turnover, and promotional history. Read the censoring note in `schema.json` before\n  computing velocity.\n- **dc_inventory_snapshot.csv** (CSV, 30 MB): 745,784 rows, a complete SKU-DC x fiscal\n  week rectangle (7,171 pairs x 104 weeks). Columns: `snapshot_date` (week-ending\n  Sunday), `sku_id`, `dc_id`, `on_hand_units`, `on_order_units`, `in_transit_units`,\n  `safety_stock_units`, `reorder_point_units`, `stockout_days`. The position is as of the\n  end of the snapshot day and immediately *before* that week's replenishment run.\n  `in_transit_units` is scheduled off the *contracted* lead time — the planning system's\n  expectation, not a promise, and it does not know about slippage or short-shipment. It\n  is therefore **not** guaranteed to be a subset of `on_order_units`: in 35,942 rows\n  (4.82%) it exceeds it, and in 26,974 of those `on_order_units` is 0 while\n  `in_transit_units` is positive. That plan-versus-reality gap is signal; do not clip it\n  away. Source for days-of-supply features and for the label.\n- **supplier_lead_time.csv** (CSV, 7.5 MB): 120,220 purchase orders at SKU level.\n  Columns: `po_id`, `sku_id`, `supplier_id`, `po_date`, `receipt_date`, `ordered_qty`,\n  `received_qty`, `nominal_lead_time_days`, `actual_lead_time_days`,\n  `replenishment_cycle_days`. Sets the per-SKU prediction horizon and carries the\n  lead-time and fill-rate risk signal.\n- **schema.json** (JSON, 7.1 kB): a `provenance` block (synthetic origin, generator,\n  seed), then field definitions in exact column order, the fiscal calendar, the\n  censoring note, the label definition, the scorable-row rule, the leakage rule, the\n  split, and the required `feature_snapshot_id` format. Read it first.\n\n# Output Files\n\n- **sku_risk_scores.csv** (CSV) [graded]: one row per scorable test-split SKU-DC-week.\n  Required columns, exactly these names:\n\n  | column | definition |\n  |---|---|\n  | `sku_id` | string, as in the inputs |\n  | `dc_id` | string, as in the inputs |\n  | `fiscal_week_index` | integer, the 1-indexed CUTOFF week `w` on the fiscal calendar in `schema.json` (week 1 ends 2024-01-07). Not an index within the test split. |\n  | `prediction_cutoff_date` | the week-ending date of `fiscal_week_index`, i.e. `2024-01-07 + 7*(w-1)`. Write it `YYYY-MM-DD`. |\n  | `horizon_days` | integer, that SKU's `replenishment_cycle_days` (7/14/21/28) |\n  | `feature_snapshot_id` | string, `FS-YYYYMMDD` of `prediction_cutoff_date` (e.g. `FS-20250928`) |\n  | `stockout_risk_score` | float in [0, 1], the CALIBRATED probability |\n  | `stockout_label` | integer 0/1, the label derived by the rule above |\n\n  Additional columns are permitted and ignored by the grader, with one exception you\n  may want to use: if you emit `bin_equal_width` and `bin_equal_frequency` (integer\n  0-9), the grader uses them to reconcile your reliability table. Otherwise it derives\n  bin membership from that table's own `bin_lower`/`bin_upper` edges. Ship one or the\n  other or the reliability reconciliation cannot run.\n\n  Row order is not graded. **Write `stockout_risk_score` at full float64 precision**\n  (pandas' default `to_csv` repr round-trips; do not round to a fixed number of\n  decimals) — two blocking checks compare your CSV against numbers recomputed\n  elsewhere, and rounding is the easiest way to fail them for no real reason.\n\n  Key verification targets: row count difference from the canonical scorable set = 0\n  [industry standard]; ROC-AUC not more than 0.025 below the expert reference of\n  0.88632, i.e. at least 0.86132 [typical range — not normative].\n\n- **model.pkl** (PKL) [graded]: the trained, calibrated scikit-learn estimator —\n  feature transformer and classifier together, exposing `predict_proba`. It must\n  regenerate the delivered scores from the shipped feature matrix. Two shapes are\n  accepted:\n    * the fitted estimator pickled directly (e.g. a `Pipeline`, or a\n      `CalibratedClassifierCV` wrapping one); or\n    * a dict containing it under any one of the keys `model`, `pipeline`,\n      `estimator`, `clf`, `calibrated_model`, optionally alongside a\n      `feature_columns` list giving the column order `predict_proba` expects.\n  If your feature matrix's column order is not otherwise recoverable, use the dict\n  form so the grader knows which columns to pass. Key verification targets: Brier\n  score within 0.008 of 0.048533 and ECE (10 equal-width bins) within 0.03 of\n  0.008298 [typical range — not normative].\n\n- **model.skops** (SKOPS) [strongly recommended]: the same fitted estimator saved with\n  `skops.io.dump`. A pickle is executable code, so the grader will not deserialise one\n  whose static opcode scan finds a global outside its allowlist (roots `sklearn`,\n  `numpy`, `scipy`, `pandas`, `collections`, `copyreg`, `_codecs`, plus a small set of\n  builtins) — a submission that ships only such a pickle fails the reconciliation\n  check it cannot run. skops has no such problem: it reconstructs the estimator without\n  pickle machinery, and the grader prefers it whenever present.\n  `skops.io.dump(model, \"model.skops\")` is one line and removes the question.\n\n## Required supporting artifacts\n\n- **features_test.parquet**: the exact engineered feature matrix the delivered scores\n  were produced from, row-aligned to `sku_risk_scores.csv` (same order, same length).\n  This is what makes \"regenerate deterministically\" checkable, and its absence is a\n  blocking failure. **Ship it as Parquet, not CSV.** A gradient-boosted tree's bin\n  thresholds sit exactly on repeated feature values, so decimal rounding flips boundary\n  rows: during authoring, writing this matrix as CSV at 10 significant digits changed\n  41% of the predictions, by up to 0.106, with a completely correct model. float64\n  round-trip is the only way the reconciliation means anything.\n\n- **reliability_table.tsv**: calibration bins. Graded in full when present, and a\n  malformed or non-reconciling table is a blocking failure, so the conventions are\n  specified rather than left to taste. Tab-separated, one header row, 20 data rows:\n\n  * `scheme` — exactly the literals `equal_width` and `equal_frequency`, 10 rows each\n  * `bin_index` — integers 0..9 within each scheme, no gaps. **Empty bins must appear\n    as explicit zero-count rows**, not be omitted\n  * `bin_lower`, `bin_upper` — bin edges. For `equal_width`, bins tile **[0, 1]** in\n    ten 0.1-wide steps regardless of your score range. For `equal_frequency`, use score\n    quantiles. Intervals are `[lower, upper)` with the last bin closed at its upper edge\n  * `n`, `n_pos`, `n_neg` — row counts in the bin; `n_pos + n_neg == n`\n  * `mean_predicted` — mean `stockout_risk_score` in the bin; write `0` for an empty bin\n  * `empirical_rate` — `n_pos / n`; write `0` for an empty bin\n  * `bin_weight` — `n / total_rows`; the ten weights sum to 1\n  * `calibration_gap`, `ece_contribution` — optional; checked only if present\n\n- **metrics.json**: headline metrics computed from the delivered rows. Optional — its\n  absence is not penalised. If present it must parse, and if it contains a ROC-AUC\n  under any of `headline.roc_auc`, `test.roc_auc`, `metrics.roc_auc`, `roc_auc`, or\n  `test.test_roc_auc`, that value must match the ROC-AUC recomputed from your CSV to\n  within 1e-6. If the grader finds no recognisable ROC-AUC field it skips the check\n  rather than failing you.\n\n- **run_report.md**: short human-readable summary. Informational, not graded.\n\n# Tools / Software Details\n\nSoftware: Python 3.11, scikit-learn 1.4.2, pandas 2.2.2, numpy 1.26.4, pyarrow 16.1.0,\nskops 0.14.0.\n\n- Engineer SKU-DC-week features and the forward stockout label (pandas / numpy)\n- Train, calibrate, and validate the propensity model on a time-based split\n  (scikit-learn, JupyterLab)\n- Export the fitted pipeline (pickle, and skops for a deserialisation-safe copy)\n\nAll components are open source: scikit-learn BSD-3, pandas BSD-3, numpy BSD-3,\npyarrow Apache-2.0, skops MIT, Python PSF. No commercial licence, account, token or\ncredential is needed at any point. In particular there is no Snowflake step to run:\nthe extraction exists only in the scenario framing, and the flat files described above\nare what you actually start from.\n\n# Environment and Resource Requirements\n\n**No GPU is required, and none is used.** scikit-learn's `HistGradientBoostingClassifier`\nis CPU-only; the pipeline is CPU-bound end to end. All figures below are measured from\nthe reference run on one Linux node (Ubuntu, Linux 6.x x86_64, Python 3.11).\n\n| stage | wall clock | peak RSS |\n|---|---|---|\n| generate/read the input files | ~2 min | ~6 GB |\n| build the 576,637 x 51 feature panel | ~17 s (16 threads) | ~4 GB |\n| baseline ladder + 4-config sweep + foil | ~90 s | ~3 GB |\n| calibration study and final fit | ~15 s | ~3 GB |\n| assemble deliverables and verify | ~20 s | ~2 GB |\n\nRecommended: 8+ CPU threads and 16 GB RAM. It runs on 4 threads in roughly three times\nthe wall clock. Disk: ~600 MB for inputs, panel, and outputs, plus ~4 GB if the conda\nenvironment is built from scratch.\n\nPut the workspace on a data volume rather than a home directory — the intermediate\nfeature panel is ~300 MB and the POS file is 129 MB.\n\n# Expected Completion Time\n\n**14 hours** (range 10–18) for a competent ML engineer comfortable with pandas and\nscikit-learn. Breakdown, so the figure is auditable:\n\n| hours | activity |\n|---|---|\n| 1.5 | read `schema.json`, profile the three feeds, confirm the panel is a complete rectangle |\n| 2.0 | label construction: per-SKU horizons, the >= 3 day rule, the three scorability conditions |\n| 3.5 | feature engineering at panel scale — multi-window velocity, turnover, coverage, censoring flags |\n| 2.0 | lead-time and supplier features: trend against nominal, fill rate from ordered vs received |\n| 1.5 | leakage audit — every window re-checked to end at the cutoff, transformers fitted on train only |\n| 1.5 | time-split training and a small sweep, plus baselines that make the graph of value measurable |\n| 1.0 | calibration study across validation, with the base-rate drift into test understood |\n| 1.0 | assemble deliverables so `model.pkl`, the feature matrix and the CSV reconcile exactly |\n| 0.5 | metrics, reliability table, report |\n\nThe label and scorability rules are ~2 of the 14 hours and are the part most easily\nunderestimated: an off-by-one in the forward window, a fixed horizon, or forgetting the\n`on_hand_units > 0` condition each produce a plausible-looking file that fails the\nrow-count check outright.\n\n# Evaluation Criterion\n\nMethod: script + rubric review\nMetric: primary reference-output comparison over structure, key values, consistency, and\nevidence grounding\nThreshold: pass when the primary reference output has no critical mismatch and supporting\nartifacts show no blocking inconsistency","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"All checks below are machine-checkable. Every threshold stated as a tolerance is a\nnumeric comparison against the reference value shipped in reference_output.zip.\n\"Blocking\" means the submission fails.\n\nThe grader that implements sections A-G is shipped as\nreference_output.zip/scripts/60_grade_submission.py. It rebuilds the ground truth\nfrom the INPUT files and recomputes every metric from the submitted CSV rather than\ntrusting the submission's own metrics.json. Run it through\nscripts/65_sandbox_grade.sh, which grades inside a disposable filesystem and\nprocess sandbox — see section E for why that matters, and note that a submitted\npickle is REFUSED outright unless the grader can prove at load time that it is\ninside one.\n\nEvery rule below is implemented. The grader emits 51 checks against the reference\nand all 51 pass. Where a rule says BLOCKING the grader fails the submission, and\nwhere it says WARNING the grader warns — the two are kept in step deliberately, so\nthis document can be read as the specification of the code rather than an\naspiration for it.\n\nHow the numbered rules map to emitted checks, so the counts reconcile on inspection:\n\n  section   rules   emitted   note\n  A         5       6         two separate file-presence checks\n  B         7       9         plus a join check and an extra-key diagnosis\n  C         3       3\n  D         4       18        D3 and D4 have lettered sub-items, one check each,\n                              run per scheme\n  E         10      9         E4 is the sandbox WRAPPER (65_sandbox_grade.sh), not\n                              a check inside the grader; E3b contributes the two\n                              both-artifacts checks and fires only when a\n                              submission ships model.pkl AND model.skops\n  F         5       3         F1 is B3 and F4 is C2; F2, F3 and F5 are separate,\n                              and F5 is advisory-only — it can WARN, never FAIL\n  G         4       3         G4 is a grading POLICY, not an assertion about a\n                              submission, so it deliberately has no check\n\n  Emitted total on the reference: 6+9+3+18+9+3+3 = 51.\n\n────────────────────────────────────────────────────────────────────────────\nA. sku_risk_scores.csv — parse and schema\n────────────────────────────────────────────────────────────────────────────\nA1. File parses as CSV with pandas. Blocking on failure.\nA2. Required columns present, 100% of them:\n      sku_id, dc_id, fiscal_week_index, prediction_cutoff_date, horizon_days,\n      feature_snapshot_id, stockout_risk_score, stockout_label\n    Additional columns are permitted and ignored.\nA3. Dtypes coerce cleanly: fiscal_week_index / horizon_days / stockout_label to\n    integer, stockout_risk_score to float, prediction_cutoff_date to a date.\nA4. Record count > 0.\nA5. A model artifact is present: model.pkl OR model.skops. **model.skops is an\n    accepted replacement for model.pkl** — ship either, or both. If you ship BOTH,\n    both must be real: E2b requires the pickle to load under the same safe policy\n    even though the grader prefers the skops copy, and E2c requires the two to be\n    the same estimator. A placeholder named model.pkl beside a working model.skops\n    is a FAIL, not a pass.\n\n────────────────────────────────────────────────────────────────────────────\nB. sku_risk_scores.csv — exact-match identity (mismatches must be 0)\n────────────────────────────────────────────────────────────────────────────\nThe canonical scorable set is rebuilt from dc_inventory_snapshot.csv and\nsupplier_lead_time.csv by the rule in schema.json. A (sku, dc, w) row is scorable\niff w >= 14 AND w + (replenishment_cycle_days // 7) <= 104 AND on_hand_units > 0\nat the cutoff. The test split is cutoff weeks 91-104, trimmed per SKU by horizon.\n\nB1. Row count == 78,864. Difference must be exactly 0. Blocking.\nB2. (sku_id, dc_id, fiscal_week_index) is unique. Blocking.\nB3. The delivered key set equals the canonical scorable test set — 0 extra keys and\n    0 missing keys. Blocking. This is the single check most submissions get wrong,\n    so the grader also diagnoses WHY an extra key is extra: a train/val week, a week\n    below the warm-up, an incomplete forward window, or a row that was already at\n    zero on hand.\nB4. prediction_cutoff_date == the week-ending date of fiscal_week_index\n    (2024-01-07 + 7*(w-1)). 0 mismatches. Blocking.\nB5. horizon_days == that SKU's replenishment_cycle_days. 0 mismatches. Blocking.\n    This is what catches a solver who applied one fixed horizon to every SKU.\nB6. feature_snapshot_id == \"FS-\" + prediction_cutoff_date formatted YYYYMMDD, and\n    it is a 1:1 function of the cutoff date. 0 mismatches.\nB7. stockout_label == the label derived by the rule in schema.json:\n    sum(stockout_days) over weeks w+1 .. w+horizon_weeks >= 3. 0 mismatches.\n    Blocking. This is an integrity check on the submitted file, not a scoring check\n    — a submission that alters the ground truth is invalid. Reference positives:\n    5,582 of 78,864 (7.0780%).\n\n────────────────────────────────────────────────────────────────────────────\nC. sku_risk_scores.csv — discrimination\n────────────────────────────────────────────────────────────────────────────\nC1. stockout_risk_score is finite and within [0, 1] for every row. Blocking.\nC2. ROC-AUC computed from (stockout_label, stockout_risk_score).\n    Reference = 0.88632. The comparison is ONE-SIDED WITH AN UPPER REVIEW BAND:\n\n      AUC >= 0.86132  (reference - 0.025)   -> pass\n      AUC <  0.86132                        -> fail\n      AUC >  0.91632  (reference + 0.030)   -> do not auto-fail;\n                                               route to human leakage review\n\n    The one-sided form is deliberate. A symmetric band would auto-fail a correct\n    submission for being better than the reference, which is not a defect. The\n    reference is a competent expert solution, not a ceiling.\n\n    The upper band matters more than usual here, and the reason is worth stating\n    plainly. An earlier revision of this submission shipped a reference whose\n    feature builder mapped dates to fiscal weeks by flooring from the week-ENDING\n    Sunday. That put Monday-to-Saturday AFTER each cutoff into the week that had\n    just ended, so every POS and purchase-order window silently absorbed six\n    post-cutoff days. It scored 0.94957. Removing the leak -- a one-line change of\n    anchor, from the week end to the week's first day -- dropped the reference to\n    0.88632. The leak was worth 0.063 AUC. Any submission far above the current\n    reference should be assumed to have made the same class of mistake until a\n    human has checked, which is exactly what the upper band routes.\n\n    Where the floor comes from -- a measured ladder of feature sets, every rung\n    trained with the SAME HistGradientBoosting configuration on the SAME split, so\n    the only thing varying is which features exist. Test ROC-AUC, all leakage-free:\n\n      service history alone (5 features)                     0.57531\n      position only: on hand, on order, in transit,\n        safety stock, reorder point (6)                      0.81701\n      + multi-window velocity, days of supply, turnover (11) 0.83819\n      + lead-time risk and coverage gaps (16)                0.86795\n      + service history (19)                                 0.86906\n      all reference features (50)                            0.88613\n      deployed reference (sweep-selected config, calibrated) 0.88632\n\n    Reported separately because they use a different estimator -- single-feature\n    logistic regressions, for scale: on-hand units alone 0.68422, days of supply\n    alone 0.71164, the six raw inventory columns 0.78657.\n\n    0.86132 sits between the 11-feature rung (0.838, fails) and the 16-feature rung\n    (0.868, passes): it separates a shallow attempt -- inventory columns plus a\n    velocity average -- from a genuinely engineered panel, which is the intended\n    discrimination. It does NOT single out any one feature family as mandatory. The\n    ladder is nested, so each rung's gain is the gain of ADDING it to what came\n    before, not proof that it is required. Measured the other way round, dropping\n    all 16 lead-time and coverage features from the full panel scores 0.88533\n    (b3_no_lead_time_features in metrics.json) -- still comfortably over the floor,\n    because the remaining 35 features recover nearly all of that signal. So\n    lead-time work is one sufficient route over the floor, not a necessary one, and\n    the rubric grades the delivered probabilities rather than the route. See F3.\n    Note also the first rung -- recent stockout history ALONE is worth almost\n    nothing (0.575). This is not an autocorrelation task.\n\nC3. PR-AUC is recomputed and reported for context but is NOT gated.\n    Reference 0.47998 at a 7.08% base rate.\n\n────────────────────────────────────────────────────────────────────────────\nD. Calibration\n────────────────────────────────────────────────────────────────────────────\nD1. Brier score from the delivered rows, ONE-SIDED:\n      Brier_submission <= 0.056533   (reference 0.048533 + 0.008)\n    Only the upper bound blocks. A submission scoring BELOW 0.040533 is not failed\n    for being better than the reference -- it is passed through to human leakage\n    review as a WARN, exactly the way C2 treats an unusually high ROC-AUC. (The\n    previous revision used |Brier - 0.048533| <= 0.008, which auto-rejected a\n    legitimately better-calibrated model. That was a false-fail and is fixed.)\n    For scale: a constant predictor at the base rate scores 0.06577, so the usable\n    band is roughly 0.0485-0.0658 and this bound spans most of it.\nD2. ECE with 10 equal-width bins over [0, 1], weighted by bin population, ONE-SIDED:\n      ECE_submission <= 0.038298     (reference 0.008298 + 0.030)\n    One-sided for the same reason, and here it is also the only coherent form: ECE\n    is non-negative, so the old symmetric rule's lower bound of -0.021702 was\n    unreachable and the check was already upper-bound-only in effect. Now it says so.\n    What this catches, measured not assumed: an identical model trained with\n    class_weight='balanced' and reported straight off the head scores ROC-AUC\n    0.88660 — statistically indistinguishable from the reference — but a mean\n    predicted probability of 0.25398 against an actual base rate of 0.07078, for an\n    ECE of 0.18320. That is 22x the reference and 6.1x outside this tolerance.\n    Class-rebalancing is routine at a 7% positive rate and it destroys the\n    probability scale; repairing it is what \"report calibrated probabilities, not\n    raw scores\" asks for.\n\n    What it does NOT catch, deliberately: a submission trained with an unweighted\n    objective whose raw head is already near-calibrated. The check grades the\n    calibration of the delivered probabilities, not the presence of a calibration\n    step. A solver who arrives at well-calibrated output by another route has\n    satisfied the requirement.\n\nD3. reliability_table.tsv. Conventions are specified in the task statement\n    (scheme literals equal_width / equal_frequency; equal-width bins tile [0,1] in\n    ten 0.1 steps; empty bins present as explicit zero rows with mean_predicted 0;\n    bin_weight = n / total; intervals [lower, upper) with the last closed).\n    ABSENT is a non-blocking warning — D1/D2 already grade\n    the delivered probabilities. PRESENT means graded in full, and a malformed or\n    non-reconciling table is BLOCKING, not a warning: a shipped artifact that does\n    not hold up is a defect, not a missing optional extra. Checks, in order:\n      a. parses as TSV with a header\n      b. required columns present: scheme, bin_index, n, n_pos, mean_predicted,\n         empirical_rate\n      c. both \"equal_width\" and \"equal_frequency\" schemes present\n      d. each scheme has exactly 10 bins, bin_index 0..9, no gaps. Empty bins must\n         appear as explicit zero-count rows, not be omitted.\nD4. Cross-file reconciliation, per scheme, every mismatch count must be 0:\n      a. sum(n) == submitted row count\n      b. sum(n_pos) == sum(stockout_label)\n      c. sum(n_neg) == row count - sum(stockout_label), when n_neg is present\n      d. sum(bin_weight) == 1 within 1e-9, when bin_weight is present\n      e. a per-bin group-by of the submitted CSV reproduces each bin row's n, n_pos\n         and mean_predicted (mean_predicted at rtol 1e-6)\n    For (e) the grader needs to know which row went in which bin. It takes the\n    submission's own bin_equal_width / bin_equal_frequency columns when they exist,\n    otherwise it re-derives membership from the table's bin_lower / bin_upper edges.\n    If neither is available the table is unverifiable and (e) FAILS — ship one or the\n    other.\n\n────────────────────────────────────────────────────────────────────────────\nE. The model — safe deserialisation, then deterministic regeneration (blocking)\n────────────────────────────────────────────────────────────────────────────\nA submitted model file is UNTRUSTED INPUT. `pickle.load` executes whatever the\nGLOBAL / STACK_GLOBAL / REDUCE opcodes tell it to; a submission can therefore run\narbitrary code inside the grader simply by being loaded. The grader never does that\nblindly. Five independent controls, in order:\n\nE1. **Prefer model.skops.** If the submission ships one, it is loaded with\n    `skops.io` — no pickle machinery at any point. Every type it declares is\n    checked: on the exact allowlist, or resolvable inside sklearn/numpy/scipy to a\n    CLASS (not a callable) whose name is not a loader. The reference declares two\n    such types, both on the exact list. Its absence is a non-blocking WARNING, but\n    shipping it is the difference between a reconciliation that always runs and one\n    that may be refused.\nE2. **Static opcode scan of model.pkl.** `scripts/64_scan_pickle.py` walks the\n    pickle with `pickletools` and resolves every GLOBAL / STACK_GLOBAL operand,\n    tracking the memo so module names emitted once and later referenced by BINGET\n    still resolve exactly. Nothing is imported and nothing is executed. It reports\n    which globals the file would import and which fall outside the allowlist.\nE3. **Refuse, or restrict — against an EXACT allowlist.** If the scan finds any\n    off-allowlist global, or leaves any operand unresolved, the grader REFUSES to\n    deserialise and the load fails — it does not guess. If the scan is clean,\n    loading goes through a `RestrictedUnpickler` whose `find_class` re-checks the\n    same set at the one gate every callable must pass before REDUCE can invoke it.\n    There is no `--allow-pickle` override; the flag has been removed.\n\n    The allowlist is an exact set of 98 (module, name) pairs — types and vetted\n    reconstruction helpers — not a list of trusted package roots. That is a\n    correction, and the reason matters. The previous revision trusted any callable\n    whose module ROOT was sklearn / numpy / scipy / pandas, which is not a control\n    at all: `pandas.io.pickle.read_pickle` sits under an allowed root, so a REDUCE\n    calling it passed both the scan and `find_class`, and it then opened a SECOND\n    pickle with the stock unrestricted unpickler. Measured on the authoring host: a\n    two-stage pickle built exactly that way scanned CLEAN under the old rule and\n    executed its payload on load; under the exact set the scan reports\n    `pandas.io.pickle.read_pickle` as off-allowlist, the load is refused, and the\n    payload never runs. The set also self-audits on every import — an entry whose\n    name contains a file, name-resolution or process marker (`load`, `open`,\n    `read`, `exec`, `import`, `system`, ...) raises at import time rather than\n    quietly widening the gate.\n\n    Consequence a solver should know: an unusual-but-honest estimator can be\n    off-list and its pickle refused. That costs nothing if you also ship\n    model.skops, which is why E1 recommends it. The refusal message names the exact\n    pairs to vet and add.\nE3b. **Both artifacts shipped means both are graded.** The loader returns on the\n    first success, so when a valid model.skops is present the pickle would never be\n    opened at all. That made \"model.pkl present\" satisfiable by an empty file. Two\n    checks close it: E2b independently loads model.pkl under the same scan +\n    restricted-unpickler policy, and E2c requires model.pkl and model.skops to\n    produce identical predictions on the shipped feature matrix (max |delta| <=\n    1e-9). Both are proven to bite -- an empty model.pkl beside a valid\n    model.skops FAILs E2b (50 checks, 48 pass, 2 fail), and a model.pkl whose Platt\n    calibrator was perturbed by x1.35 FAILs E2c at max |delta| 7.319e-02 (51\n    checks, 49 pass, 2 fail). If you ship only one artifact, neither check applies\n    and nothing is lost.\n\nE4. **Attested sandbox — required for every pickle load.** `65_sandbox_grade.sh` is\n    not advice any more. Before the grader hands a single byte to the pickle\n    machinery it runs `common.attest_sandbox()`, which TESTS the isolation live, on\n    that host, in that process, by attempting the three things hostile code would:\n\n      leg 1  open TCP to 1.1.1.1:443 and resolve a hostname — both must fail\n      leg 2  create a file in each tree it is grading (its own code directory,\n             inputs, submission, reference, /usr) — all must fail\n      leg 3  read a canary the wrapper wrote OUTSIDE the jail, plus the host's\n             own dotfiles — none must be readable\n      leg 4  count visible PIDs — a PID namespace, not the host's process table\n\n    If any leg fails the pickle is REFUSED. Not warned about: refused. This closes\n    the previous revision's real gap — it offered `unshare -rn`, a NETWORK namespace\n    only, which left the whole host filesystem readable and writable to anything a\n    pickle chose to run.\n\n    The wrapper's third branch builds a jail that passes: a tmpfs root, selective\n    read-only rbinds of only /usr /lib /lib64 /bin /sbin /etc plus the conda env,\n    the scripts, the inputs, the submission and the reference, a fresh /proc, a\n    tmpfs HOME, then `pivot_root`, all inside `unshare --user --map-root-user\n    --mount --net --pid`. No root, no docker, no setuid helper.\n\n    Measured on the authoring host, same reference bundle, same command:\n      inside the jail   network ok (no egress, no DNS); 6 write paths all reject\n                        a create; 5 read canaries all invisible; 2 PIDs visible;\n                        attestation PASSED; pickle opened; 51 checks, 51 pass\n      bare login shell  TCP connected AND DNS resolved; scripts/ and /home/<user>\n                        writable; 1432 PIDs; attestation FAILED; pickle refused;\n                        50 checks, 49 pass, 1 warn (the reconciliation ran off\n                        model.skops, so the submission still PASSes)\n    A pickle-ONLY submission on that bare shell cannot be graded at all: it FAILs\n    with \"this is an ENVIRONMENT failure, not a submission defect\" and the command\n    to re-run. Inside the jail the same submission grades 49 checks, 48 pass,\n    1 warn, PASS.\n\n    Disclosure on what was actually exercised: the authoring host has neither\n    docker nor bwrap, so ONLY the userns-jail branch has been run. The docker and\n    bwrap branches are provided for hosts that have them and are untested here;\n    both are written to satisfy the same attestation, and since attestation is what\n    gates the load, a branch that silently under-isolates fails closed rather than\n    passing. The docker branch additionally assumes $PYTHON resolves to an\n    interpreter that works inside the chosen image, which is not true if you pass a\n    host conda env into python:3.11-slim. The script header says so too. There is\n    no FORCE_UNSANDBOXED override any more.\n\n    Degradation was measured, not assumed. Reference bundle, four configurations:\n      jail, skops library + model.skops present   51 checks, 51 pass          PASS\n      jail, model.skops absent (pickle path)      49 checks, 48 pass, 1 warn  PASS\n      jail, skops library not installed           49 checks, 49 pass          PASS\n      no jail, both artifacts present             50 checks, 49 pass, 1 warn  PASS\n    The count moves because E2b/E2c only exist when BOTH artifacts are gradable;\n    the dropped checks are skipped, never silently passed, and the score\n    reconciliation runs in all four cases.\n\n    This was tested adversarially, not assumed. A pickle whose `__reduce__` returns\n    `(os.system, ('touch /tmp/PWNED_BY_PICKLE',))` is caught by the scan as\n    `posix.system`, refused before deserialisation, graded FAIL — and the payload\n    never executes. The two-stage `read_pickle` bypass in E3 is caught the same way.\n    The legitimate blind-trial pickle, by contrast, scans clean at 18 globals, all\n    on the exact list, and loads through the restricted unpickler without incident.\n\nThen the substance:\n\nE5. The loaded estimator exposes predict_proba. Blocking.\n    Both shapes are accepted: a bare estimator, or a dict carrying the estimator\n    under any of model / pipeline / estimator / clf / calibrated_model plus an\n    optional feature_columns list.\nE6. It bundles a feature transformer with the classifier (a Pipeline, or a\n    calibrated wrapper around one). Checked, non-blocking — a submission that\n    reproduces its scores exactly has satisfied the intent regardless of shape.\nE7. A feature matrix is shipped (features_test.parquet preferred, features_test.csv\n    accepted), row-aligned to the score file. Absence is BLOCKING: without it\n    \"regenerate scores deterministically\" cannot be verified at all.\nE8. predict_proba on that matrix reproduces stockout_risk_score. Two tiers:\n\n      max |delta| <= 1e-9                                    -> pass, exact\n      else: mean |delta| <= 5e-3 AND spearman rho >= 0.995   -> pass, with a note\n      else                                                    -> fail\n\n    The tolerant tier exists for a measured reason. A gradient-boosted tree's bin\n    thresholds sit exactly on repeated feature values — on-hand units and 4-week\n    velocity are integer-derived, so thousands of rows share a value that IS a\n    quantile threshold. Writing the feature matrix as CSV at 10 significant digits\n    perturbs those values by ~5e-10 relative, which moves them across the split:\n    during authoring that changed 41% of predictions, by up to 0.106, with a model\n    that was entirely correct. Bit-exactness is a property of the serialisation\n    format, not of the model, so it must not be the whole test. The reference ships\n    Parquet and reconciles at max |delta| = 0.0 through both model.pkl and\n    model.skops.\n\nE9. metrics.json. ABSENT is fine. PRESENT is graded: it must parse, and if it carries\n    a ROC-AUC under any of headline.roc_auc / test.roc_auc / metrics.roc_auc /\n    roc_auc / test.test_roc_auc, that value must reproduce from the submitted CSV\n    within 1e-6. If no recognisable ROC-AUC field is found the check is SKIPPED, not\n    failed — the grader does not guess at key names. An unreadable file, or a\n    recognisable value that disagrees, is BLOCKING: either means the reported numbers\n    were not computed from the delivered rows.\n\n────────────────────────────────────────────────────────────────────────────\nF. Leakage guards\n────────────────────────────────────────────────────────────────────────────\nF1. No delivered row lies outside the canonical scorable set. This is B3 and it is\n    the primary leakage guard: a solver who scored weeks whose forward window runs\n    past the end of the data has necessarily invented labels.\nF2. Feature-name screen on the shipped feature matrix: no column name contains\n    future, next_week, next_cycle, forward, fwd_, lead_label, target, label,\n    y_true, post_cutoff, actual_stockout, or outcome. Warn, not fail — a name is\n    weak evidence — but it routes to human review.\nF3. Oracle screen: the grader computes single-feature ROC-AUC for every shipped\n    feature. If any lands above 0.97 or below 0.03, the label has almost certainly\n    entered a feature; warn and route to human review. For calibration, the\n    strongest single feature in the reference is on_hand_vs_13w_avg at 0.18459\n    (i.e. 0.815 inverted), comfortably inside the band, and the strongest RAW INPUT\n    column is on_hand_units at 0.29125. The inputs themselves contain no oracle:\n    that is verified by 50_verify.py, which is also shipped.\nF4. The C2 upper review band. See section C.\nF5. ADVISORY, never blocking: does the shipped feature matrix contain a lead-time /\n    supplier-risk feature family (columns naming lead_time, coverage, fill_rate,\n    supplier, cycle, late, short_ship, po_)? REPORTED FOR A HUMAN, and it cannot\n    fail a submission. Rationale, measured: the b3_no_lead_time_features baseline in\n    metrics.json drops all 16 such features and still scores 0.88533, versus 0.88613\n    with them and a 0.86132 floor. Requiring them would therefore fail a solver\n    whose model is 0.0008 from the reference, contradicting G4. It is reported\n    because it is useful to see which route a solver took -- blind trial 2 built the\n    family under entirely different names (dos_over_cycle, dos_minus_cycle,\n    cycle_days), which is exactly why a name-based check must never block.\n\n────────────────────────────────────────────────────────────────────────────\nG. Overall\n────────────────────────────────────────────────────────────────────────────\nG1. machine-readable = true for both graded files: sku_risk_scores.csv parsed, and\n    the model loaded under the section-E policy. Emitted as a real check rather\n    than left as prose.\nG2. Constraint violations = 0 — the grader counts its own blocking failures so far\n    and asserts the total is zero.\nG3. Formats remain comparable: sku_risk_scores.csv is CSV with a header carrying at\n    least the required columns, and the model artifact loads under scikit-learn\n    1.4.2 (from model.skops when shipped, otherwise the scanned pickle).\nG4. Method-agnostic. This is a grading POLICY, not something to assert about a\n    submission, so it has no corresponding check — and F5 is deliberately advisory\n    rather than blocking so that no *stated* feature suggestion contradicts it.\n    The reference selects Platt over\n    isotonic by a\n    one-standard-error rule and the two are statistically tied here (out-of-fold\n    Brier 0.038941 vs 0.038996, SE 0.000231). Grading must not require a particular\n    calibrator, estimator family, or feature list — only that the delivered\n    probabilities meet the stated tolerances and that the shipped model regenerates\n    them.\n\n────────────────────────────────────────────────────────────────────────────\nReference values (from reference_output.zip/metrics.json)\n────────────────────────────────────────────────────────────────────────────\n  scored rows                          78,864\n  positives                            5,582  (7.0780%)\n  distinct SKU-DC pairs                7,171\n  weekly cutoffs                       13\n  horizon mix (7/14/21/28 days)        25,503 / 26,944 / 16,181 / 10,236\n  ROC-AUC (calibrated)                 0.88632\n  PR-AUC                               0.47998\n  Brier                                0.048533\n  ECE_10bin equal width                0.008298\n  ECE_10bin equal frequency            0.008305\n  mean predicted vs actual             0.06248 vs 0.07078\n  ROC-AUC, class-balanced foil         0.88660  (ECE 0.18320)\n  ROC-AUC, inventory columns only      0.78657\n  ROC-AUC, days-of-supply alone        0.71164\n  operating point @ recall 0.60        review 8,792 rows (11.15%), precision 0.381\n  calibrator selection (out-of-fold Brier, one-SE rule)\n                                       sigmoid 0.038941, isotonic 0.038996,\n                                       SE 0.000231 -> sigmoid\n  model regeneration max delta         0.0 (pkl and skops both)\n\n────────────────────────────────────────────────────────────────────────────\nH. Artifact provenance\n────────────────────────────────────────────────────────────────────────────\n**The data is synthetic.** All three CSVs and schema.json are deterministically\nsimulated by scripts/10_make_inputs.py at seed 20260802, which ships inside\nreference_output.zip. No real retailer, distribution centre, supplier, purchase\norder or point-of-sale record is involved, and no customer or commercial data of\nany kind. The task statement's framing of the files as Snowflake extracts describes\nthe working scenario a solver is asked to model — it is not a claim about the origin\nof the bytes. The same disclosure is machine-readable in schema.json under\n`provenance`, and 50_verify.py asserts that block is present and says so.\n\n**Checksums travel inside the archives.** Each zip carries its own\nMANIFEST.sha256 covering every other member of that archive, so provenance is\nverifiable from the two attachments alone, with no external file required:\n\n    cd inputs           && sha256sum -c MANIFEST.sha256\n    cd reference_output && grep -v '^#' MANIFEST.sha256 | sha256sum -c\n\nThe archive-level hashes, for confirming the attachments themselves:\n\n  input_materials.zip    29,202,954 bytes\n    5748feeeaee2f64c3cc879e928ef9f01af0c63b1c673b2cc9284341b3b1e1042\n  reference_output.zip   12,130,504 bytes\n    aaba706f936ef514c1573f73e5b024460bb06facfbc5997135da5733d383ba30\n\nThese two values are reproducible, not incidental: 70_package.sh normalises member\ntimestamps and entry order, so repackaging the same payload yields a byte-identical\nzip. Verified by packaging twice and comparing hashes. The same values are repeated\nin upload/CHECKSUMS.txt, a convenience copy for the submitter — the authoritative\nin-band record is the MANIFEST.sha256 inside each archive.\n\ninput_materials.zip contains exactly the four files a solver starts from\n(pos_transactions.csv, dc_inventory_snapshot.csv, supplier_lead_time.csv,\nschema.json) plus its manifest — no labels, no engineered features, no reference\noutputs.\n\nreference_output.zip contains the graded deliverables, the supporting artifacts,\nits manifest, and the full reproduction pipeline under scripts/: 00_setup_env.sh,\n10_make_inputs.py, 20_train_calibrate.py, 25_diagnose.py, 40_make_outputs.py,\n50_verify.py, 60_grade_submission.py, 64_scan_pickle.py, 65_sandbox_grade.sh,\n70_package.sh, common.py, run_all.sh. 50_verify.py self-checks the reference bundle\n(50 checks, or 48 on a host without the skops library, which skips the two\nskops-specific ones rather than failing them; including an oracle screen on the\nraw input columns, the provenance\nblock, and a static scan of the reference's own pickle); 60_grade_submission.py\napplies sections A-G to an arbitrary submission (51 checks on the reference).\nBoth were re-run against the packaged copy after unzipping both archives into a\nclean directory — in-archive MANIFEST.sha256 verified for every member,\n50_verify.py 50/50, and 60_grade_submission.py 51/51 through\n65_sandbox_grade.sh, i.e. inside the attested jail (mount + PID + network\nnamespaces, read-only binds, pivot_root), not merely a network namespace.\n\nReproducibility, measured on the current (leakage-free) build by rerunning the whole\npipeline in a clean directory at OMP_NUM_THREADS=16 against a reference produced at 24:\n\n  pos_transactions.csv, dc_inventory_snapshot.csv, supplier_lead_time.csv,\n  schema.json                          bit-identical (SHA-256 match)\n  delivered key set, row order, labels identical\n  stockout_risk_score                  max |delta| 0.0 — bit-identical\n  ROC-AUC                              0.886315040 both\n\nSo `bash scripts/run_all.sh` reproduces the reference exactly on this build. Note that\nis not guaranteed in general: HistGradientBoosting's parallel histogram build has\nthread-count-dependent float reduction order, and an earlier build of this submission\ndid drift by up to 4.9e-09 across the same thread-count change (ROC-AUC still identical\nto 9 decimals, 0 rows differing by more than 1e-6). Treat exact reproduction as a\nproperty of this build rather than a promise, do not compare model.pkl or model.skops\nby checksum, and note that E8 grades regeneration numerically rather than by hash for\nexactly this reason. The MANIFEST.sha256 inside each archive is a provenance record for\nTHAT packaged copy.\n\n────────────────────────────────────────────────────────────────────────────\nI. Difficulty calibration — why the thresholds sit where they do\n────────────────────────────────────────────────────────────────────────────\nTHIS SECTION WAS REWRITTEN AFTER A LEAK WAS FOUND IN THE REFERENCE. Read the\ncorrection first, because it inverts the earlier conclusion.\n\nAn admin review found that the reference's feature builder mapped dates to fiscal\nweeks by flooring from the week-ENDING Sunday, which put Monday-to-Saturday after\neach cutoff into the week that had just ended. Every POS and purchase-order window\ntherefore absorbed six post-cutoff days. Fixing the anchor dropped the reference\nfrom ROC-AUC 0.94957 to 0.88632.\n\nThe blind trials did NOT have that bug. Their own numbers never moved. So the\n\"0.065 gap between the reference and a strong agent\" reported in the previous\nrevision was not a difficulty signal at all — it was the size of the leak. Corrected:\n\n  ROC-AUC        blind trial 2   blind trial 1   leak-free reference\n                 0.88446         0.88483         0.88632\n\nTwo independent agents landed within 0.002 of a clean expert reference. Re-graded\nagainst it, trial 2 passes 51 of 51 checks, VERDICT PASS:\n\n  metric              trial 2     reference    delta        result\n  rows / positives    78,864      78,864       0            pass\n                      5,582       5,582\n  ROC-AUC             0.88446     0.88632      -0.00185     pass\n  Brier               0.048259    0.048533     -0.000274    pass\n  ECE_10bin           0.003292    0.008298     -0.005006    pass (beats reference)\n  reliability table   full pass   --                        pass\n  model.skops         shipped     shipped                   pass\n  regeneration        2.2e-16     0.0                       pass\n\nConsequences for grading, all changed from the previous revision:\n\n1. The self-test band is 40-60%, not 10-20%. A strong agent SOLVES this task in\n   ~35 minutes of wall clock. The earlier 10-20% figure was an artefact of grading\n   honest submissions against a contaminated reference and must not be reinstated.\n\n2. C2's floor still discriminates, and its placement is unchanged in spirit. At\n   reference - 0.025 = 0.86132 it sits between the 11-feature rung (0.838) and the\n   16-feature rung (0.868). Engineering the lead-time and coverage-gap features is\n   still the difference between failing and passing; a position-plus-velocity\n   attempt still fails. What changed is that a GOOD attempt now passes, which is\n   the correct behaviour and was previously being suppressed.\n\n3. What actually discriminates is narrower than claimed before. B1/B3/B7 (key set,\n   labels, horizons) were passed by both trials. D2 was passed by both, and both\n   beat the reference on it. D3/E were passed by trial 2. C2 is the only check that\n   separates a shallow attempt from a real one, and the ladder is what justifies\n   where it sits. Everything else is an integrity floor, not a discriminator — keep\n   them blocking, but do not expect them to rank submissions.\n\n4. The upper review band is now the load-bearing leakage control, because the exact\n   mistake it guards against is one this submission itself made. A submission\n   materially above 0.88632 should be checked for post-cutoff data in a feature\n   window before it is believed. scripts/50_verify.py now ships four cutoff-boundary\n   checks that assert the date-to-week mapping directly and end to end; they fail\n   loudly on the old anchor and pass on the corrected one.\n\nFor transparency: 01_TASK_DESCRIPTION.txt contains no reference-solution summary.\nThe label rule, the scorability rule and the output conventions are stated exactly,\nbecause the grader recomputes them and an ambiguous spec would make the row-count\ncheck a coin flip; but no calibration method, estimator, or feature list is\nsuggested. The Environment section does quote the panel size (576,637 x 51), and\nboth trials used it to confirm their scorability filter — that makes B1/B3 easier\nand is part of why they do not discriminate.","software_and_version":"\tPython 3.11, scikit-learn 1.4.2, pandas 2.2.2, skops 0.14, JupyterLab","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"b916d6ce-559f-4195-84fb-08836e0e21d1","filename":"reference_output.zip","category":"reference_output","content_type":"application/zip","size_bytes":12130504},{"file_id":"aa42db3b-3f83-4b22-86fc-05ec02d75473","filename":"selftest_evidence.txt","category":"difficulty_self_test_evidence","content_type":"text/plain","size_bytes":6133},{"file_id":"230c519a-3b20-4ac4-877c-7d270911cfc0","filename":"input_materials.zip","category":"input_materials","content_type":"application/zip","size_bytes":29202954}],"deep_review":{"overall_verdict":"Strong Accept","review_text":"**Overall Verdict:** Strong Accept — The submission is complete, internally consistent, safely reviewable, professionally representative, and objectively evaluable end to end.\n\n**Completeness Check:**\n- Instruction: Complete — The modeling goal, temporal boundaries, label construction, scorable population, split, calibration requirement, and output contract are precise.\n- Inputs: Complete — All three synthetic feeds and the schema are readable, checksum-valid, correctly separated from reference materials, and consistent with the stated dimensions and semantics.\n- Toolchain: Complete — The specified open-source Python stack supports the full CPU-only workflow on the stated single-machine resources.\n- Reference output: Complete — Scores, both model formats, the 78,864-row feature matrix, calibration table, metrics, report, and reproduction sources are present and mutually consistent.\n- Rubric: Complete — The grader defines objective structural, identity, performance, calibration, leakage, security, and deterministic-regeneration checks with explicit thresholds and failure severity.\n\n**File and Task Reachability:**\nAttachment classification: reachable. Task classification: reachable. I inspected `input_materials.zip`, `reference_output.zip`, and `selftest_evidence.txt`. Both ZIP files are valid, unencrypted containers containing only regular files with safe paths; their archive hashes match the submitted values, every compressed member passes integrity testing, and every in-band manifest checksum validates.\n\nThe input archive contains only the three source CSVs, `schema.json`, and its manifest. Full profiles confirmed 3,092,581 POS records, 745,784 inventory records forming 7,171 complete 104-week SKU-DC panels, and 120,220 purchase orders, with declared schemas, plausible ranges, and no blank fields. An independent reconstruction produced exactly 78,864 test rows and 5,582 positives, with zero key, date, horizon, snapshot-ID, or label mismatches against the reference scores.\n\nThe reference archive contains 20 members. Its score metrics independently reproduce exactly: ROC-AUC 0.886315, PR-AUC 0.479982, Brier 0.048533, and equal-width ECE 0.008298. The Parquet footer confirms 78,864 rows, 51 feature columns, and zero nulls. The SKOPS schema identifies the expected calibrated scikit-learn pipeline and scientific-library state types. Uploaded programs were inspected statically but not executed, and model artifacts were not deserialized; numerical model arrays were reviewed structurally rather than value by value. No relevant attachment was left unreviewed at the member-inventory level.\n\nThe task has a coherent path from raw feeds through leakage-controlled feature engineering, temporal training and calibration, score generation, and automated evaluation. The stated CPU, memory, disk, and runtime requirements are feasible on one suitable Linux machine. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\nNone\n\n**Recommended Changes:**\nThe submission already provides complete inputs, reproducible reference artifacts, strong safety controls, and a discriminating machine-checkable rubric. No changes are required.","reviewed_at":"2026-09-15T10:05:35.783681Z"},"claim":null}