Predicting Health Insurance

Dataset profile & anomaly cleaning

The dataset contains 72,458 records across 15 features with 32,260 missing cells (2.97% of all values). The target column health_ins comprises 65,553 insured (90.5%) and 6,905 uninsured (9.5%) customers. An unlabelled holdout set of 804 rows was provided for evaluation.

Exploratory analysis identified two structural anomalies and several redundant features:

Age distribution placeholders. age spans values from 0 to 120. A histogram reveals a sharp spike at 0, zero observations between 1 and 20, a continuous distribution from 21 to 99, and isolated spikes at 100, 110, 114, and 120. Records at age 0 and above 100 show adult characteristics, averaging 36,700 in income and 1.95 vehicles. 43 records at age 0 and 167 records above age 90 are flagged as employed. Values outside [21, 99] represent placeholder entries and were converted to NaN for imputation.

Redundant features. code_column is perfectly correlated with state_of_res (Cramér's V = 1.00) as a duplicate categorical index and was removed. rooms exhibits a uniform distribution across its six discrete levels (~12,000 observations each) with negligible correlation to the target (Cramér's V = 0.0084).

Non-random missingness in employment

Missing values occur in two distinct structural blocks. is_employed accounts for 25,515 missing entries (35.2% of rows), while 6,745 missing cells are distributed across household variables (housing_type, num_vehicles, gas_usage, recent_move_b), representing ~1,686 records missing all household attributes simultaneously.

The 25,515 missing employment values represent Not Missing at Random (NMAR) data. Naively imputing these gaps as False would inflate the recorded unemployment rate from 3.2% to 38.4%, distorting a feature where insured rates differ significantly (90.7% for employed vs 75.2% for unemployed). Age profiling indicates that missing employment status concentrates among students (ages 21–25) and retirees (ages 66–69).

To preserve this signal, is_employed was encoded as a three-state categorical variable (employed, unemployed, unrecorded) before one-hot expansion. Missing entries in recent_move_b were restored to NaN to be handled by KNN imputation alongside the household block.

Feature processing pipeline

The feature engineering pipeline transforms raw input columns into a 19-dimensional continuous matrix through eight sequential operations:

1. Feature selection: Remove custid, index, and redundant code_column.
2. Bounds filtering: Set age outside [21, 99] to NaN.
3. Scaling: Scale age to [0, 10] and encoded state_of_res to [0, 1] via min-max scaling.
4. Log transformation: Apply log1p(x - min + 1) to right-skewed variables (income, gas_usage).
5. Encoding: Label-encode binary/categorical fields (sex, is_employed, state_of_res, health_ins).
6. One-hot expansion: One-hot encode marital_status, housing_type, and is_employed.
7. Missing restoration: Revert recent_move_b missing category to NaN.
8. Imputation: Fit a 5-nearest-neighbor distance-weighted KNNImputer over all continuous and indicator columns.

Log transformations mitigate extreme right skew in financial features (income ranges from -6,900 to 1,257,000; gas_usage reaches 570 against a median of 10), preventing distance calculations in KNN imputation from being dominated by extreme values. Although EDA showed rooms had no target correlation, it was retained in the pipeline output.

Baseline performance on imbalanced data

Initial classifiers were trained on preprocessed features using a 70/30 train/test split (50,720 training rows, 21,738 test rows) without resampling:

A dummy baseline predicting the majority class ("insured") for every observation achieves 0.904 accuracy and 0.950 F1 score. All three tuned classifiers match this baseline accuracy while scoring lower on F1.

Evaluating Random Forest performance across tree counts demonstrates the pathology of class imbalance: while accuracy and F1 remain flat near 0.90, specificity drops from 0.16 at 10 trees to 0.07 at 500 trees. Additional capacity increases model bias toward the majority class.

Resampling strategy & dataset complexity

Six resampling methods from imbalanced-learn were applied to balance the training set:

To avoid selecting a resampler based on downstream model artifacts, candidate datasets were evaluated using feature-based complexity measures from problexity: F1 (maximum Fisher's discriminant ratio), F2 (volume of overlap region), F3 (maximum feature efficiency), and F4 (collective feature efficiency).

Five resamplers produced nearly identical complexity metrics (F2 = 0.514). SVM-SMOTE recorded F2 = 0.000, indicating zero feature overlap between classes. Because SVM-SMOTE generates synthetic samples along decision boundary support vectors, it produces an artificially separable feature space. The 131,106-row SVM-SMOTE dataset was selected for model benchmarking.

Classifier benchmark on balanced data

Eight models were trained on the SVM-SMOTE balanced dataset. Tree models and KNN were evaluated via parameter sweeps, while Logistic Regression, SVM, XGBoost, and MLP underwent 5-fold GridSearchCV optimized for F1 score:

On balanced data, metric scores align closely within each model. KNN is the exception, yielding high specificity (0.963) alongside lower recall (0.818) due to dense synthetic minority clusters influencing distance voting.

Random Forest (100 trees) achieved the highest overall performance (0.935 accuracy, 0.936 F1, 0.930 specificity). XGBoost placed second (0.914 accuracy), constrained by a grid search cap of max_depth=7 whereas single unpruned decision trees in Random Forest reached depths of 40. Linear and kernel methods trailed ensemble models by 12 to 17 percentage points.

Refitting the 100-tree Random Forest on all 131,106 balanced observations and evaluating on the 804 holdout records produced the final submission.