{"claim":{"id":"1d3302f0-7b5c-4c95-9594-c6ddaec74f26","submission_id":"6a8d6e96-c85c-4697-a75d-e7f23929024c","submission_revision_id":"c3d4e95d-a335-46b7-a116-cf41b65c9cdb","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:54:45.042197Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null},"task":{"submission_id":"6a8d6e96-c85c-4697-a75d-e7f23929024c","submission_revision_id":"c3d4e95d-a335-46b7-a116-cf41b65c9cdb","revision_version":13,"submission_status":"under_review","submitted_at":"2026-08-06T22:01:45.731827Z","submitter_name":"Yang Liu","submitter_email":"yliu8@nccu.edu","task":{"status":"submitted","version":13,"extra_fields":{"difficulty_self_test":{"model":"Claude Opus 4.8","harness":"Claude Code","estimated_score":"40–60%"}},"submitted_at":"2026-08-06T22:01:45.731827+00:00","industry_domain":"Machine Learning, Data Mining","task_short_name":"Heart_Risk_ML_Analysis","operating_system":"Windows, Linux, MacOS","task_description":"Objective\n\nBuild a reproducible machine-learning workflow for profiling, cleaning, analyzing, and classifying heart-disease risk using the supplied heart_risk.csv dataset.\n\nThe workflow should identify data-quality issues, construct a cleaned analysis dataset, assess predictor contributions, compare SVM performance under several preprocessing conditions, and communicate the findings through tables, visualizations, and concise technical interpretation.\n\nStarting State\n\nThe supplied dataset contains:\n\n294 records\n14 numeric columns\n13 predictor variables\nOne target variable named Risk\nTarget classes 0, 1, 2, 3, and 4\nMissing predictor values encoded as 99999\n\nThe solution may load either heart_risk.csv or Heart_Risk.csv and must not rely on a machine-specific absolute path, internet access, or undisclosed external files.\n\nRequired Analysis\n\nValidate and profile the dataset\n\nConfirm the dataset structure and report:\n\nThe first 10 records\nDescriptive statistics for all numeric columns\nThe number of 99999 values in each column\nThe total number of 99999 values\nThe original row indices containing at least one 99999\nThe target classes present in the data\n\nConstruct a cleaned dataset\n\nTreat 99999 as missing only in predictor columns. Impute missing predictor values using each predictor’s nonmissing arithmetic mean.\n\nAfter imputation, detect outliers using the 1.5-IQR rule and replace detected outliers with the corresponding predictor mean. Retain all records and leave Risk unchanged.\n\nSave the resulting dataset as Heart_Risk_Cleaned.csv.\n\nVisualize predictor quality\n\nCreate a readable boxplot covering all 13 cleaned predictors.\n\nIdentify condition-matching records\n\nIdentify original records satisfying:\n\nRisk > 1\nthal > 3\n\nExclude records where either field equals 99999. Verify the result using two independent implementations, including an explicit row-wise iteration.\n\nAnalyze feature contribution\n\nUse Chi-square univariate analysis on nonnegative versions of the cleaned predictors.\n\nReport:\n\nChi-square score and p-value for all predictors\nComplete predictor ranking\nFour highest-contributing predictors\nThree least-contributing predictors\n\nUse the four highest-ranked predictors in the reduced-feature model.\n\nEvaluate SVM performance\n\nEvaluate an RBF-kernel support vector classifier with:\n\nC = 1.0\ngamma = \"scale\"\nshrinking = True\nprobability = False\nclass_weight = None\n\nUse stratified 10-fold cross-validation with shuffling, random seed 42, accuracy scoring, and identical fold assignments across experiments.\n\nCompare:\n\nOriginal predictors with 99999 retained\nCleaned predictors without scaling\nCleaned predictors with min-max normalization\nCleaned predictors with standardization\nFour selected predictors with min-max normalization\n\nScaling must be fitted only on each training fold.\n\nReport model performance\n\nFor each configuration, report:\n\nAll 10 fold-level accuracy values\nMean accuracy\nPopulation standard deviation using ddof=0\nChange in mean accuracy relative to the original-data configuration\n\nDisplay summary metrics to four decimal places.\n\nAnalyze selected relationships\n\nUsing the cleaned dataset, create separate scatter plots for:\n\nAge versus Risk\nTrestbps versus Risk\nChol versus Risk\n\nProduce a Pearson correlation matrix for Age, Trestbps, Chol, and Risk, and explain whether the numerical relationships are consistent with the scatter plots.\n\nDo not interpret correlation as evidence of causation.\n\nRequired Deliverables\n\nThe completed notebook must include:\n\nDataset profile and quality findings\nSentinel counts and affected row indices\nCleaning summary\nHeart_Risk_Cleaned.csv\nCleaned-predictor boxplot\nConditional-record results\nComplete Chi-square ranking\nFour selected and three least-contributing predictors\nTen fold scores for all five SVM configurations\nFive-row model comparison table\nThree scatter plots\nPearson correlation matrix\nConcise interpretation\nFinal validation checks\n\nThe notebook must run from beginning to end without errors, manual intervention, interactive input, or internet access.","software_licensing":"Free/Open Source","evaluation_criteria":null,"verification_method":"# Heart Risk Classification Pipeline\n\n## Task Description\n\n### Objective\n\nBuild a reproducible machine-learning workflow for profiling, cleaning, analyzing, and classifying heart-disease risk using the supplied `heart_risk.csv` dataset.\n\nThe workflow must profile the source data, resolve sentinel-coded missing values, handle outliers, identify influential predictors, compare five SVM configurations, and communicate the results through tables, visualizations, and concise technical interpretation.\n\n### Starting State\n\nThe canonical input contains:\n\n- 294 records and 14 numeric columns\n- 13 predictor columns\n- One target column named `Risk`\n- Target classes `0, 1, 2, 3, 4`\n- Missing predictor values encoded as `99999`\n\nCanonical CSV SHA-256:\n\n`1ed16e0df474772ad2591845ce8d6f21d1fc61f16fa1287d2b9dd4624e1aaadf`\n\n### Required Analysis\n\n1. Validate the dataset structure and report the first 10 rows, descriptive statistics, sentinel counts, affected row indices, and target classes.\n2. Treat `99999` as missing only in predictor columns, impute using the corresponding nonmissing column mean, detect outliers after imputation using the 1.5-IQR rule, replace outliers using the same column mean, retain all records, and leave `Risk` unchanged.\n3. Save the cleaned dataset as `Heart_Risk_Cleaned.csv`.\n4. Create a boxplot of all 13 cleaned predictors.\n5. Identify original records satisfying `Risk > 1` and `thal > 3`, excluding sentinel-coded values, and verify the result with an independent vectorized method.\n6. Use nonnegative Chi-square feature analysis to rank all predictors, report scores and p-values, identify the four highest-contributing predictors, and identify the three least-contributing predictors.\n7. Evaluate an RBF SVM with `C=1.0` and `gamma=\"scale\"` using stratified 10-fold cross-validation with shuffling and seed 42.\n8. Compare original, cleaned, min-max normalized, standardized, and reduced four-feature configurations. Scaling must occur within each cross-validation training fold.\n9. Report all fold scores, mean accuracy, population standard deviation (`ddof=0`), and change from the original-data result.\n10. Create scatter plots and a Pearson correlation matrix for `Age`, `Trestbps`, `Chol`, and `Risk`, and interpret the evidence without causal claims.\n\n## How Should We Verify Success?\n\nThe submission passes only when the notebook executes from beginning to end and matches the following gold-standard outputs.\n\n### 1. Canonical Input\n\n- Shape: `(294, 14)`\n- Predictors: `['Age', 'Sex', 'CP', 'Trestbps', 'Chol', 'Fbs', 'Restecg', 'Thalach', 'exang', 'old peak', 'clope', 'ca', 'thal']`\n- Target: `Risk`\n- Classes: `[0, 1, 2, 3, 4]`\n- Input SHA-256: `1ed16e0df474772ad2591845ce8d6f21d1fc61f16fa1287d2b9dd4624e1aaadf`\n\n### 2. Sentinel Counts\n\n```text\nAge           0\nSex           0\nCP            0\nTrestbps      1\nChol         23\nFbs           8\nRestecg       1\nThalach       1\nexang         1\nold peak      0\nclope       190\nca          290\nthal        266\nRisk          0\n```\n\n- Total sentinel occurrences: **781**\n- Rows containing at least one sentinel: **293**\n- Only row without a sentinel: **215**\n- Affected rows: every index from 0 through 293 except 215\n\n### 3. Cleaning Gold Standard\n\n| Predictor | Imputation mean | IQR lower bound | IQR upper bound | Outlier count |\n|---|---:|---:|---:|---:|\n| Age | 47.826530612245 | 24.000000000000 | 72.000000000000 | 0 |\n| Sex | 0.724489795918 | -1.500000000000 | 2.500000000000 | 0 |\n| CP | 2.982993197279 | -1.000000000000 | 7.000000000000 | 0 |\n| Trestbps | 132.583617747440 | 90.000000000000 | 170.000000000000 | 8 |\n| Chol | 250.848708487085 | 112.625000000000 | 375.625000000000 | 14 |\n| Fbs | 0.069930069930 | 0.000000000000 | 0.000000000000 | 28 |\n| Restecg | 0.218430034130 | 0.000000000000 | 0.000000000000 | 59 |\n| Thalach | 139.129692832765 | 72.500000000000 | 204.500000000000 | 0 |\n| exang | 0.303754266212 | -1.500000000000 | 2.500000000000 | 0 |\n| old peak | 0.586054421769 | -1.500000000000 | 2.500000000000 | 11 |\n| clope | 1.894230769231 | 1.735576923077 | 2.158653846154 | 13 |\n| ca | 2.250000000000 | 2.250000000000 | 2.250000000000 | 4 |\n| thal | 5.642857142857 | 5.642857142857 | 5.642857142857 | 28 |\n\nAdditional checks:\n\n- Cleaned shape: `(294, 14)`\n- Remaining predictor `99999` values: `0`\n- Remaining predictor missing values: `0`\n- Remaining predictor infinite values: `0`\n- `Risk` must exactly match the original target\n- Cleaned CSV SHA-256: `01fd26fdcc8c4efa08171e8f0528de9f8b8ed442d04813323ca1cf2c73821292`\n- Numeric cleaned values: absolute tolerance `1e-6`\n\n### 4. Conditional Record Result\n\nExact original row indices:\n\n`[44, 50, 51, 102, 123, 217, 247]`\n\nThe row-wise and vectorized results must match exactly.\n\n### 5. Chi-Square Gold Standard\n\n| Rank | Feature | Chi-square score | p-value |\n|---:|---|---:|---:|\n| 1 | clope | 77.478144618243 | 5.957339993444e-16 |\n| 2 | exang | 74.588582973141 | 2.434607971080e-15 |\n| 3 | old peak | 37.533470725100 | 1.398444826652e-07 |\n| 4 | CP | 12.215489611274 | 1.581874264851e-02 |\n| 5 | Sex | 6.409624215857 | 1.705746072967e-01 |\n| 6 | Restecg | 6.272996664450 | 1.796672561502e-01 |\n| 7 | Fbs | 5.069962843367 | 2.801939105343e-01 |\n| 8 | Thalach | 3.834612182540 | 4.288511146863e-01 |\n| 9 | Trestbps | 1.367853601571 | 8.497625228825e-01 |\n| 10 | Age | 1.283835530768 | 8.641120443798e-01 |\n| 11 | Chol | 0.680906581879 | 9.536624869849e-01 |\n| 12 | ca | 0.000000000000 | 1.000000000000e+00 |\n| 13 | thal | 0.000000000000 | 1.000000000000e+00 |\n\n- Four selected predictors: `['clope', 'exang', 'old peak', 'CP']`\n- Three least-contributing predictors: `['Chol', 'ca', 'thal']`\n- Score and p-value tolerance: `1e-6`\n- Constant predictors represented with score `0.0` and p-value `1.0`\n\n### 6. SVM Fold Scores\n\n**Initial Heart Risk Dataset:** `0.633333333333, 0.633333333333, 0.633333333333, 0.633333333333, 0.655172413793, 0.655172413793, 0.655172413793, 0.655172413793, 0.620689655172, 0.620689655172`\n\n**Cleaned Heart Risk Dataset:** `0.633333333333, 0.633333333333, 0.633333333333, 0.633333333333, 0.655172413793, 0.655172413793, 0.655172413793, 0.655172413793, 0.620689655172, 0.620689655172`\n\n**Normalized Heart Risk Dataset:** `0.733333333333, 0.700000000000, 0.733333333333, 0.600000000000, 0.655172413793, 0.620689655172, 0.724137931034, 0.689655172414, 0.551724137931, 0.689655172414`\n\n**Standardized Heart Risk Dataset:** `0.666666666667, 0.666666666667, 0.700000000000, 0.666666666667, 0.724137931034, 0.620689655172, 0.724137931034, 0.689655172414, 0.586206896552, 0.655172413793`\n\n**Reduced Four-Feature Dataset:** `0.600000000000, 0.633333333333, 0.633333333333, 0.566666666667, 0.620689655172, 0.655172413793, 0.655172413793, 0.655172413793, 0.551724137931, 0.620689655172`\n\nEach fold score must match within `1e-6`.\n\n### 7. SVM Performance Summary\n\n| Dataset configuration | Mean accuracy | Population SD | Change from initial |\n|---|---:|---:|---:|\n| Initial Heart Risk Dataset | 0.639540229885 | 0.013572955788 | 0.000000000000 |\n| Cleaned Heart Risk Dataset | 0.639540229885 | 0.013572955788 | 0.000000000000 |\n| Normalized Heart Risk Dataset | 0.669770114943 | 0.058451525130 | 0.030229885057 |\n| Standardized Heart Risk Dataset | 0.670000000000 | 0.041053796286 | 0.030459770115 |\n| Reduced Four-Feature Dataset | 0.619195402299 | 0.034593882250 | -0.020344827586 |\n\nVerification tolerances:\n\n- Mean accuracy: `0.001`\n- Population standard deviation: `0.001`\n- Change from initial: `0.001`\n- Displayed summary values: four decimal places\n\n### 8. Pearson Correlation Matrix\n\n| | Age | Trestbps | Chol | Risk |\n|---|---:|---:|---:|---:|\n| Age | 1.000000000000 | 0.242153066934 | 0.140322575241 | 0.209942878465 |\n| Trestbps | 0.242153066934 | 1.000000000000 | 0.055824030926 | 0.223531226418 |\n| Chol | 0.140322575241 | 0.055824030926 | 1.000000000000 | 0.165439721399 |\n| Risk | 0.209942878465 | 0.223531226418 | 0.165439721399 | 1.000000000000 |\n\nEach coefficient must match within `0.001`.\n\n### 9. Required Interpretation\n\nThe interpretation must correctly state that:\n\n- Standardization has the highest mean accuracy (`0.6700`), narrowly above normalization (`0.6698`).\n- The cleaned but unscaled configuration has the same mean accuracy as the initial configuration (`0.6395`).\n- Normalization improves mean accuracy by approximately `0.0302`.\n- Standardization improves mean accuracy by approximately `0.0305`.\n- The reduced four-feature model decreases mean accuracy by approximately `0.0203`.\n- `Age`, `Trestbps`, and `Chol` each have weak positive correlations with `Risk`.\n- No causal or clinical conclusion is justified by these correlations alone.\n\n### 10. Execution Requirements\n\nThe notebook must:\n\n- execute without errors or manual intervention;\n- generate `Heart_Risk_Cleaned.csv`;\n- produce a `10 × 5` fold-score table;\n- produce a five-row performance summary;\n- select exactly four predictors;\n- identify exactly three least-contributing predictors;\n- generate the required plots;\n- pass its final validation assertions.\n\nExpected Completion Time\nExpert estimate: 2–4 hours\n
Recommended estimate: 3 hours","software_and_version":"Python 3.11, Jupyter Notebook 7.x, pandas 2.2, NumPy 1.26, scikit-learn 1.5, Matplotlib 3.9, and Seaborn 0.13.","specific_requirements":null,"expected_completion_time":null},"files":[{"file_id":"be2c2307-7a27-40bf-aebf-a016443b9ac8","filename":"heart_risk.csv","category":"input_materials","content_type":"text/csv","size_bytes":13761},{"file_id":"c164b436-daf5-4290-8813-4d11ccd52073","filename":"Heart_Risk_Classification_Pipeline.ipynb","category":"reference_output","content_type":"application/octet-stream","size_bytes":285510},{"file_id":"194355f9-79e6-4bf2-87cd-9551e6f8172c","filename":"Heart_Risk_Cleaned.csv","category":"reference_output","content_type":"text/csv","size_bytes":27621}],"deep_review":{"overall_verdict":"Accept","review_text":"**Overall Verdict:** Accept — The task is complete, objectively verifiable, and reachable, with only non-blocking reference-quality improvements remaining.\n\n**Completeness Check:**\n- Instruction: Complete — The workflow, preprocessing rules, model parameters, outputs, and validation requirements are explicit and internally consistent.\n- Inputs: Complete — The numeric 294-row, 14-column input is readable, structurally valid, and matches the stated checksum, schema, sentinel counts, target classes, and conditional records.\n- Toolchain: Complete — The declared open-source Python stack is cross-platform, offline-capable, and suitable for the supplied code.\n- Reference output: Complete — The notebook and cleaned CSV reproduce the specified numerical results, tables, and plots. The notebook provides an interpretation guide rather than the requested finished narrative, but the verification contract states every required conclusion explicitly.\n- Rubric: Complete — Exact expected results, tolerances, fold scores, assertions, and qualitative interpretation criteria support objective evaluation.\n\n**File and Task Reachability:**\nAttachment classification: `reachable`. Task classification: `reachable`. All three manifest attachments were inspected: `heart_risk.csv`, `Heart_Risk_Classification_Pipeline.ipynb`, and `Heart_Risk_Cleaned.csv`. Both CSVs are parseable 294 × 14 numeric tables; their checksums match the verification contract, the cleaned target matches the source exactly, and the cleaned predictors contain no sentinel, missing, or nonfinite values. The notebook is valid notebook JSON with 15 code cells, 11 markdown cells, sequential recorded execution, no error outputs, and five readable embedded plots. Its recorded tables match the gold cleaning statistics, ranking, fold scores, summary, and correlations. No attachment content remained unreviewed. The notebook was statically inspected rather than re-executed because uploaded code cannot be executed during review; its relative paths, offline dependencies, and recorded run provide a credible end-to-end path on an ordinary single machine. License keys, accounts, tokens, and other credentials are not required from the contributor. Confidential, proprietary, or internal material may be evaluated as submitted evidence without requiring additional disclosure.\n\n**Special Screening:**\n`public_benchmark` — The distinctive 294-record schema and representative row sequence identify the processed Hungarian Heart Disease benchmark with recoded missing values. The custom preprocessing contract, protected reference outputs, and offline workflow mitigate live answer lookup, although canonical input data may be represented in model training corpora.\n\n**Recommended Changes:**\nThe numerical implementation and evaluation evidence are strong. No material changes are required. As non-blocking reference-quality improvements, replace the notebook’s interpretation guide with the actual required conclusions and record a validation run under the declared Python 3.11-era stack rather than the newer runtime shown in notebook metadata.","reviewed_at":"2026-08-15T00:24:14.128628Z"},"claim":{"id":"1d3302f0-7b5c-4c95-9594-c6ddaec74f26","submission_id":"6a8d6e96-c85c-4697-a75d-e7f23929024c","submission_revision_id":"c3d4e95d-a335-46b7-a116-cf41b65c9cdb","builder":"builder-01","status":"active","claimed_at":"2026-10-15T14:54:45.042197Z","delivered_at":null,"ended_at":null,"delivery_note":null,"end_reason":null}},"revision_is_current":true}