Medicare Provider Cost Analysis

Devlog

Today closed out the entire pipeline, from training the last two models to deploying a production web app. The session started with the LSTM and OOP models still untrained (code was written on April 7 but GPU wasn't available), and ended with a six-page Next.js app live on Vercel backed by pre-computed predictions in Supabase.

The first challenge was getting PyTorch with CUDA working. The Windows Anaconda environment had torch CPU-only, and WSL Ubuntu had no torch at all despite having the 5070 Ti visible via nvidia-smi. I copied the repo into WSL, created a venv, installed PyTorch with the cu128 index, and ran both the sequence creation (05_lstm_sequences_local.py) and training (train_lstm_local.py) from there. The LSTM hit R-squared of 0.886 on the 2022-2023 validation set with MAE of $8.84, early stopping at epoch 19 out of 100. Two bugs surfaced in the visualization code: the label_encoders.json stores lists (not dicts), and parquet deserializes the years column as numpy arrays (not Python lists), breaking `.index()` calls. Both were straightforward fixes.

OOP quantile regression ran next in WSL with the same venv. The P50 model scored R-squared of 0.40 with exact 50% coverage, and the P90 achieved 90% coverage. These are on synthetic data, so the R-squared matters less than the calibration.

Phase 6 was the big pivot. The original plan was Streamlit, but I switched to Next.js with Material UI and Supabase. The key architectural decision was pre-computing all predictions rather than running models at request time. I aggregated the 103M gold rows into 32,818 Stage 1 lookup groups (by specialty, bucket, state, place of service), the 10.3M synthetic OOP rows into 23,424 Stage 2 groups, and uploaded the 62,703 LSTM forecast rows directly. Total database: about 119K rows across 7 tables, well within Supabase free tier. The Cost Estimator page does two Supabase queries (Stage 1 + Stage 2) and displays Medicare allowed amounts alongside patient OOP ranges with P10/P50/P90 breakdowns.

The honest friction point was bulk data upload. The Supabase MCP tool couldn't handle 2000-row INSERT statements, and two background agents both failed to push the data through. The fix was simple: use the Python supabase client with the service role key directly. Took about 5 minutes once I had the right key.

What's next: Connect the Vercel project to the GitHub repo for auto-deploy on push, then shift focus to other projects.

Changelog

Added

• Add Next.js 16 web app in `web/` with Material UI, Recharts, and Supabase client
• Add 6 pages: Dashboard, Cost Estimator, Forecast Explorer, Model Comparison, Data Explorer, About
• Add Supabase schema with 7 tables: lookup_labels, stage1_allowed_amounts, stage2_oop_estimates, lstm_forecasts, state_summary, model_metrics, feature_importances
• Add `web/scripts/upload_all.py` for pre-computing aggregations and uploading to Supabase
• Add `.claude/launch.json` for dev server preview
• Add synthetic data disclaimer on Cost Estimator OOP results

Changed

• Update `.gitignore` with web/node_modules, web/.next, web/.env.local
• Update PROGRESS.md with Phase 3 training results (R-squared=0.886), Phase 5 training results, and full Phase 6 milestones
• Update Phase 3 status from "code complete" to complete (LSTM trained in WSL with CUDA)
• Update Phase 5 status from "code complete" to complete (OOP quantile regression trained)

Fixed

• Fix `train_lstm_local.py` label_encoders handling (list-based, not dict-based)
• Fix `train_lstm_local.py` numpy array `.index()` call replaced with `np.where()`
• Fix `.env` Windows carriage return causing Databricks MLflow auth failure in WSL

Devlog

Three phases landed in a single session today, covering the LSTM forecasting architecture, MCBS beneficiary data integration, and the Stage 2 out-of-pocket prediction model. The two-stage pipeline vision is now fully wired: Stage 1 predicts what Medicare allows for a service, Stage 2 predicts what the patient pays out of pocket.

Phase 3 was the LSTM time-series forecaster. The model takes year-ordered sequences of allowed amounts grouped by specialty, HCPCS bucket, and state, then forecasts 2024-2026 using autoregressive rollout with MC Dropout for confidence bounds. The architecture uses static embeddings for the three group keys concatenated with the target value at each timestep, feeding into a 2-layer LSTM with a linear head. Teacher forcing during training, temporal split (train on 2013-2021, validate on 2022-2023). Training is deferred until GPU is free since other models are running on the 5070 Ti.

Phase 4, the MCBS integration, is where the real surprises hit. I built the download script, Bronze ingest, and Silver cleaning pipeline expecting to join survey demographics with cost supplement spending data at the beneficiary level. Three constraints in the actual MCBS Public Use File forced a redesign. First, the PUF has no Census region (suppressed for privacy). Second, the Survey File and Cost Supplement use completely different PUF_IDs with zero overlap, so they cannot be joined. Third, the Survey File ships as three seasonal CSV rounds per year with entirely different columns that need merging. The Cost Supplement turned out to be self-contained with its own demographics, which simplified things once I stopped trying to force the survey join.

The solution was a dual-track architecture. Track A works with the real PUF data at a national aggregate level, which is honest but limited. Track B generates synthetic per-service OOP records with Census region by sampling beneficiary demographics from real MCBS distributions and modulating OOP amounts based on dual eligibility, supplemental insurance, and chronic condition count. The synthetic data is clearly labeled, and anyone with access to the actual MCBS Limited Data Set (which costs $600 per module and requires a Data Use Agreement) can drop in their real data and run the same pipeline unchanged.

Phase 5 trained the Stage 2 quantile regression on the synthetic data: three separate XGBoost boosters for P10, P50, and P90. The P50 model hit MAE of $9.78 with exact 50.0% coverage, and the P90 model achieved 90.0% coverage. The key design choice was making the Stage 1 target (allowed amount) the primary input feature for Stage 2, which is what connects the two stages into a coherent prediction pipeline.

What's next: LSTM training once GPU is available, then Phase 6 (Streamlit portfolio app combining both stages plus LSTM forecast display with uncertainty bands).

Changelog

Added

• Add train_lstm_local.py: PyTorch LSTM with static embeddings, temporal split, MC Dropout autoregressive forecast
• Add pull_mcbs_data.py: downloads MCBS Survey File (2015-2022) and Cost Supplement (2018-2023) from CMS
• Add 06_mcbs_bronze_local.py: handles CMS directory structure, merges 3 seasonal survey rounds on PUF_ID
• Add 07_mcbs_silver_local.py: Cost Supplement cleaning with real CMS column names (PAMTOOP, CSP_AGE, etc.)
• Add 08_mcbs_crosswalk_local.py with dual-mode flag (--mode puf for national, --mode synthetic for regional)
• Add generate_synthetic_mcbs.py: produces 10.3M synthetic per-service OOP records with Census region
• Add train_oop_local.py: Stage 2 XGBoost quantile regression (P10/P50/P90) on 12 features
• Add synthetic_metadata.json with provenance and drop-in replacement instructions

Changed

• Update compare_models_local.py with LSTM in Stage 1 table and separate Stage 2 OOP section
• Update CLAUDE.md with LSTM commands, MCBS pipeline, dual-track architecture, torch dependency
• Update PROGRESS.md with Phases 3-5 milestones and PUF constraints documentation

Devlog

With the infrastructure in place, I spent the next five days running the full pipeline end-to-end and training models at national scale. The most consequential change happened before any model training: I swapped the target variable.

The original target was `Avg_Mdcr_Pymt_Amt` (what Medicare actually pays). But the project's two-stage vision is to first predict what Medicare allows and then, in Stage 2, predict patient out-of-pocket costs. The allowed amount is the right Stage 1 target because it represents Medicare's determination of the service value before provider-specific adjustments. This required updating the IQR outlier bounds in the silver layer, the feature engineering in gold, and all three training scripts.

The bigger lesson was catching data leakage. The initial feature set included `Avg_Mdcr_Pymt_Amt` and `Avg_Mdcr_Stdzd_Amt` as features, both of which are derived from the target. With those columns in, the models hit R-squared of 0.9996, which looked impressive but was meaningless. Removing the leaky features and their ratio derivatives dropped performance to realistic levels and forced me to engineer genuinely predictive features.

The final feature set has 10 columns: three label-encoded categoricals (provider type, state, HCPCS code), a coarse clinical bucket (mapping CPT code ranges to 6 categories like surgery, radiology, lab), a facility/office flag, HCC risk scores from a separate CMS provider-level dataset (joined on NPI and year, with median imputation for missing values), log-transformed service and beneficiary counts, submitted charge amount, and a services-per-beneficiary ratio.

Training at 103 million rows required a regional batch approach. I split the data into five Census regions and trained incrementally: XGBoost continues its booster across regions (125 rounds each), Random Forest uses sklearn's warm_start (125 trees per region, 625 total). This keeps memory manageable while still training on the full national dataset. XGBoost auto-detects CUDA for GPU acceleration when available.

Random Forest came out as the best performer on the test set. The GLM diverged and needs hyperparameter work. I also prepared the LSTM sequence data: 23,672 provider-service-state groups with year-ordered target vectors, ready for Phase 3.

What's next: Phase 3 LSTM time-series forecasting using the prepared sequences, training on 2013-2021 and validating against 2022-2023, with projections forward to 2024-2026.

Changelog

Added

• Add pull_provider_data.py for CMS "by Provider" dataset (HCC risk scores, 11 years)
• Add 5 new features: hcpcs_bucket, place_of_srvc_flag, Bene_Avg_Risk_Scre, log_srvcs, log_benes
• Add regional batch training by Census region (Northeast, South, Midwest, West, Territories)
• Add CUDA auto-detection for XGBoost GPU acceleration
• Add cuML auto-detection for Random Forest full mode
• Add LSTM sequence preparation script (23,672 groups, 10,540 with complete 11-year coverage)
• Add model comparison harness with paired t-tests and feature importance extraction
• Add label_encoders.json for cross-script encoding consistency
• Add PROGRESS.md documenting Phase 2 completion and results

Changed

• Update target variable from Avg_Mdcr_Pymt_Amt (payment) to Avg_Mdcr_Alowd_Amt (allowed amount)
• Update silver layer IQR outlier bounds for new target
• Update gold layer to per-state parquets instead of monolithic file
• Update all training scripts for spec-aligned 10-feature set

Removed

• Remove leaky features: Avg_Mdcr_Pymt_Amt, Avg_Mdcr_Stdzd_Amt, pymt_to_charge_ratio, stdz_to_pymt_ratio

Devlog

I set up the full data infrastructure for the Medicare provider cost analysis project in a single day. This is an H515 class project, but the scope is real: predicting Medicare allowed amounts per service using CMS Physician & Practitioners data spanning 2013 to 2023, roughly 103 million rows at national scale.

The first task was building the data pull. The CMS Data API serves the "by Provider & Service" dataset through paginated REST endpoints, one UUID per year. I wrote a client that fetches in 5,000-row batches, handles rate limiting, and writes raw CSVs to disk. Then a partitioning script splits each year's CSV by state and provider type, which is important because the full national file is too large to process in memory on a single machine. During partitioning, I inject the year column by extracting it from the filename via regex. This turned out to be a critical decision: the year column is not in the raw CMS data, and without it the LSTM time-series work I have planned for Phase 3 would require re-processing everything.

The pipeline follows a medallion architecture with dual execution modes. The Databricks path (PySpark + Delta Lake on S3) is the production target, and I stubbed out all four notebooks: bronze ingest, silver cleaning, gold feature engineering, and EDA. The local path (pandas + PyArrow) mirrors the same logic for development on my machine. Both paths share the same logical transformations so results stay consistent.

I also stubbed out three model training scripts (GLM, Random Forest, XGBoost) and a comparison harness, all wired to log experiments to MLflow on Databricks. The initial target variable was set to `Avg_Mdcr_Pymt_Amt` (Medicare payment amount), though I already suspected this might need to change once I dug deeper into the distinction between what Medicare allows versus what it pays.

What's next: Running the local pipeline end-to-end (bronze through gold), engineering the feature set, and training the first models at national scale.

Changelog

Added

• Add CMS Data API client for "by Provider & Service" dataset (2013-2023, 11 years)
• Add partition script splitting raw CSVs by state and provider type with year column injection
• Add CSV-to-parquet conversion script for columnar storage
• Add Databricks notebook stubs: bronze ingest, silver clean, gold features, EDA
• Add local pipeline equivalents for pandas/PyArrow development
• Add model training stubs: GLM (SGDRegressor), Random Forest, XGBoost
• Add model comparison harness with MLflow logging to Databricks workspace
• Add project proposal document
• Add README with pipeline architecture overview
• Add .gitignore excluding data/, partitioned_data/, local_pipeline/

Devlog

I moved model training from my local WSL2 environment (24 GB RAM, 30% data sampling) to Google Colab Pro A100 GPUs and trained on the full 126.8 million rows for the first time. The V1 models had been bottlenecked by memory: XGBoost at R2=0.833 and Random Forest at R2=0.884, both on only 30% of the data with regional batching. V2 removes all those compromises.

The results were dramatic for individual models. LightGBM hit R2=0.9575 (MAE $6.73), XGBoost reached R2=0.9452 (MAE $7.73), and CatBoost landed at R2=0.9070 (MAE $10.88) after needing 3,000 iterations to converge on the GPU. The jump from V1 to V2 came almost entirely from using all the data, not from model architecture changes. The same XGBoost hyperparameters that gave 0.833 on 30% gave 0.945 on 100%.

The most interesting finding was the no-charge ablation. I expected removing `Avg_Sbmtd_Chrg` (which carried 61.8% feature importance in V1) to crater R2 by 0.30 or more. Instead, the drop was only 0.015 across all three models. LightGBM without the charge feature still scored R2=0.9428, better than any V1 model with it. The `hcpcs_target_enc` (procedure-level mean encoding) and `HCPCS_Cd_idx` together capture the same pricing signal through a different path.

The ensemble was the toughest result to accept. After 13.3 hours of 5-fold cross-validation on the A100 (15 full training runs), the stacked ensemble lifted R2 by only +0.0004 over LightGBM alone. The Ridge meta-learner assigned weight 1.120 to LightGBM and negative weights to both XGBoost (-0.097) and CatBoost (-0.023), effectively reducing the ensemble to LightGBM with minor corrections. The three boosters make correlated errors on the same hard cases (rare procedures, unusual provider patterns), so stacking cannot average away what is structurally the same blind spot. My compute estimates for this notebook were also significantly off: I had budgeted 12-16 CU but burned approximately 149 CU. CatBoost at 3,000 GPU iterations on 100M+ rows took roughly 100 minutes per fold, a 3.5x underestimate.

What's next: Stage 2 OOP improvements (CatBoost monotonic constraints, conformalized quantile regression, zero-inflated modeling) and TFT forecasting with external economic covariates. These use much smaller datasets and will not repeat the compute surprise.

Changelog

Added

• Add `colab/V2_01_stage1_full_training.ipynb` for full-data XGBoost, CatBoost, LightGBM training
• Add `colab/V2_02_stage1_ablation.ipynb` for no-charge feature ablation (12 features)
• Add `colab/V2_03_stage1_ensemble.ipynb` for 5-fold OOF stacking with Ridge meta-learner
• Add `V2_MODEL_SPEC.md` with full implementation spec for 8 Colab notebooks
• Add `pull_external_covariates.py` to download Medicare CF, Medical CPI (BLS API), sequestration rates
• Add `modeling/train_catboost_local.py` with native categorical handling via ordered target statistics
• Add `modeling/train_lgbm_local.py` with GOSS and leaf-wise growth

Changed

• Update CatBoost GPU config: `bootstrap_type='MVS'` (required for subsample on GPU), removed `rsm` (unsupported for regression on GPU), increased to 3000 iterations
• Update LightGBM to CPU mode with `num_threads=-1` (GPU mode fails with bin size > 255 on high-cardinality categoricals)
• Update notebook memory pattern: save train/test splits to Colab local SSD, each model cell reloads independently to keep peak RAM under 25 GB

Fixed

• Fix CatBoost Pool creation: use DataFrame input instead of numpy float64 arrays (CatBoost rejects float arrays when `cat_features` is specified)
• Fix `gc` import missing from V2_01 cell 2, causing NameError on parquet concat cleanup
• Fix Colab Drive mount: `force_remount=True` and `fusermount -u` fallback for stale ephemeral mounts

Devlog

I redesigned the AllowanceMap frontend to match a new design system and bring the UI up to date with V2 model results. The existing pages were functional but had a mix of static PNG charts baked from Python EDA scripts and V1 model references (Random Forest as the best model at R2=0.884). The new design system, documented in `design-docs/design-system-v1.html` and `design-docs/color-exploration.html`, codifies the Slate Teal (#0F6E8C) primary and Warm Teal-Green (#15755D) secondary palette, Inter + IBM Plex Mono typography pairing, and a complete token set for spacing, shadows, and radii.

The process started with three self-contained HTML mockups (one per page) that I built to preview the layout and visual design before touching any React code. Raj reviewed these, flagged two things: no em dashes anywhere, and all charts should be rendered client-side with Recharts rather than serving static PNGs. The static image approach gave zero interactivity and required regenerating plots whenever the data changed.

The Forecast page was the heaviest lift. It went from a single ComposedChart and three Image tags to a full dashboard: three summary stat cards (last observed, 2026 P50, projected CAGR), a restyled confidence-band chart, a year-by-year detail table with YoY change, and three entirely new Recharts visualizations in the "Overall Trends" section. The multi-line specialty trends chart fetches forecast data for 12 specialties in parallel from the Railway API and renders them as a LineChart. The 2026 distribution chart uses ComposedChart with Bar and ErrorBar for a box-plot effect by HCPCS bucket. The growth chart is a horizontal BarChart sorted by 3-year CAGR.

The About page needed both a content update and chart replacement. The Models tab now shows LightGBM (no-charge, R2=0.943) as the production model with a PRODUCTION badge, CatBoost monotonic as second, and seven methodology accordions. The Data tab replaces three static PNGs with rendered mini-histograms for feature distributions, a custom SVG correlation heatmap, and a horizontal bar chart for top provider types. I created `v2-model-data.ts` as a constants file with all V2 results, since the Supabase `model_metrics` and `feature_importances` tables may not be populated yet.

One friction point was MUI v9's stricter TypeScript enforcement. Props like `fontSize`, `fontWeight`, and `display` that worked as direct Typography attributes in v6 now throw type errors; they all need to go in `sx`. I hit this across all three page files and had to fix roughly a dozen instances after the initial build failed.

What's next: Deploy to Vercel, verify the full end-to-end flow with the Railway API, and optionally upload V2 model metrics to Supabase so the About page can pull live data instead of hardcoded constants.

Changelog

Added

• Add `web/design-docs/` with color exploration, design system v1, and three HTML page mockups
• Add `web/src/lib/v2-model-data.ts` with V2 model metrics, feature importances, methodology text, correlation matrix, feature histograms, provider type counts, and specialty color palette
• Add 3 summary stat cards to Forecast page (observed value, P50 forecast, projected CAGR)
• Add forecast detail table with P10/P50/P90 columns and YoY change percentage
• Add multi-line specialty trends chart (12 specialties, fetched in parallel from API)
• Add 2026 forecast distribution chart (ComposedChart with ErrorBar per HCPCS bucket)
• Add top growth specialties horizontal bar chart (3-year CAGR, sorted descending)
• Add custom SVG correlation heatmap component to About > Data tab
• Add rendered mini-histogram grid (6 features) to About > Data tab
• Add rendered provider types horizontal bar chart to About > Data tab

Changed

• Update Estimator page R2 reference from 0.958 to 0.943 (V2 LightGBM no-charge)
• Update About > Overview tab with V2 model names (LightGBM, CatBoost monotonic) and tech stack (Railway)
• Update About > Models tab with V2 model comparison table and PRODUCTION badge on LightGBM
• Update About > Models tab methodology accordions to 7 entries (added LightGBM, CatBoost monotonic, CatBoost Quantile OOP)
• Update About > Models tab feature importance chart label from Random Forest to LightGBM (no-charge)
• Restyle Forecast chart with Slate Teal palette, dashed P10/P90 bounds, and FORECAST reference label
• Redirect `/explorer` and `/models` routes to `/about` (content merged into tabs)

Fixed

• Fix MUI v9 Typography type errors (move fontSize, fontWeight, display to sx prop)
• Fix growth percentage sign display (negative values no longer show `+-`)
• Fix YoY change color (negative changes now render in red, not green)
• Fix specialty index mapping for trend chart (aligned with actual lookup_labels table)

Removed

• Remove all static PNG image references from Forecast and About pages
• Remove full Explorer page implementation (replaced with redirect)
• Remove full Models page implementation (replaced with redirect)

Devlog

I replaced the pre-computed Supabase lookup approach with a real-time prediction API. The original frontend queried ~33K pre-aggregated rows from Supabase tables: a user picked a specialty, state, and service category, and the app returned the group-level mean. That worked for a demo but meant the predictions were static averages, not actual model outputs. Today I built a FastAPI backend that runs the trained LightGBM and CatBoost models at inference time, deployed to Railway.

The core challenge was replicating the gold-layer feature engineering pipeline in a standalone service. The LightGBM model expects 12 features (no-charge variant) in exact order, with label encoding from `label_encoders.json`, HCPCS target encoding from a 7,636-code lookup, and derived interaction features like `specialty_bucket = provider_type_idx * 6 + hcpcs_bucket`. I traced every feature through `train_lgbm_local.py` and `03_gold_features_local.py` to get the ordering and transforms right. The `log1p` target transform on Stage 1 output (requiring `expm1` inversion) was the kind of thing that would have silently produced garbage if missed.

I opted for the no-charge model (`lgbm_v2_no_charge.txt`, R2=0.943) over the full model (R2=0.958) because users won't know the submitted charge amount when they're looking up costs. The V2 ablation study showed the charge feature mostly duplicated information already captured by HCPCS target encoding. For Stage 2 OOP, the CatBoost monotonic quantile models from V2_04 slot in cleanly: they accept a `Pool` with categorical declarations and return P10/P50/P90 directly, with built-in monotonicity constraints on dual eligibility and supplemental insurance.

The model artifacts (two LightGBM .txt files at 18 MB each, three CatBoost .cbm files totaling 8 MB) are bundled directly in the repo. Railway hobby tier handles it comfortably: the full app uses under 500 MB RAM, and single predictions return in under 10 ms. The Supabase connection stays for reference data (forecasts, state summaries, model metrics) that doesn't need real-time inference.

What's next: Wire the Vercel frontend to the Railway URL via `NEXT_PUBLIC_API_URL`, verify the full estimator flow end-to-end, and redesign the frontend to take advantage of the richer prediction inputs the API now supports.

Changelog

Added

• Add `api/` subdirectory with full FastAPI backend (config, routers, schemas, services, models)
• Add `api/models/loader.py` for startup model loading (LightGBM + CatBoost into `app.state`)
• Add `api/services/prediction.py` with Stage 1 (13/12 feature) and Stage 2 (12 feature) inference
• Add `api/services/supabase.py` for server-side proxy to forecast, label, and metrics tables
• Add `api/Dockerfile` for Railway deployment (python:3.12-slim, libgomp1 for LightGBM OpenMP)
• Add `scripts/export_models.py` for MLflow artifact download or manual placement
• Add `api/models/artifacts/` with bundled LightGBM and CatBoost model files (~26 MB total)

Changed

• Update `web/src/lib/queries.ts` to call Railway API via `fetch()` instead of Supabase directly
• Update `web/src/app/page.tsx` to use `POST /predict/full` for real-time two-stage predictions
• Update `web/src/lib/types.ts` with `Stage1Prediction`, `Stage2Prediction`, `FullPredictionResponse`
• Update `web/src/app/estimator/page.tsx` to redirect to home page (was a duplicate)
• Update `modeling/train_lgbm_local.py` and `train_oop_local.py` to save artifacts to `api/models/artifacts/`
• Update `api/models/loader.py` to prefer no-charge LightGBM and auto-detect feature set from model

Devlog

I closed two gaps in the Provider Anomaly Investigation Agent today. The first was an analyst review surface for the Claude-generated briefs sitting in `local_pipeline/anomaly/briefs/`. They were great artifacts but lived on disk with no way to triage them. The second was the OUT_OF_SPECIALTY rule, which had been stubbed to always return NOT EVALUABLE because no specialty-HCPCS scope table existed yet. Both are the kind of finish-line work that makes the difference between a research prototype and something a stakeholder can actually click through.

For the scopes, I wrote a one-time scan of all 54 state silver parquets (4.1 GB, 158 seconds) that collapses every (specialty, HCPCS, NPI) triple and computes two in-scope criteria per code: coverage (billed by at least 1% of the specialty's providers) and volume (within the cumulative 99% of the specialty's total services). A code clears scope if it satisfies either. The output is 86,924 (specialty, HCPCS) rows across 130 specialties, of which 27% land in-scope. The retriever loads this as a `{specialty: set_of_codes}` dict and the rule now reports a concrete out-of-specialty percentage per NPI, triggering above 20%.

The surprise here was the optometry case. The original spec imagined the rule would automatically flag optometrists billing cataract surgery (66984), because a clinical reading says that's out of scope. The data disagrees: 28% of optometrists in the dataset bill 66984, so it's overwhelmingly in-scope by population behavior. That NPI got flagged as CRITICAL for other reasons (extreme procedure concentration plus high intensity), not for specialty mismatch. The lesson is that "what this specialty is allowed to bill" and "what this specialty actually bills in CMS data" are two different populations, and a peer-relative rule can only learn from the second.

For the review UI, I built a static-JSON pipeline: a Node script copies briefs from the pipeline output into `web/public/data/investigations/` and generates an `index.json` with a parsed rule summary per brief (handling both em-dash and colon bullet formats Claude emits). The Next.js `/investigations` route is a client-rendered sortable table; `/investigations/[id]` is a detail view with Approve, Escalate, and Dismiss buttons that persist to localStorage along with free-text analyst notes. This keeps the whole surface deployable on Vercel without a backend. MUI v9 bit me twice: the Stack component dropped `alignItems` and `justifyContent` as direct props (must go through `sx`), and `TableRow component={Link}` produces invalid HTML, so I switched to `onClick` with `useRouter().push`.

What's next: Re-run `anomaly/agent.py --top-n 100 --live` with caching enabled so the existing briefs get regenerated with OUT_OF_SPECIALTY actually evaluated, then use the new UI to walk through the top providers and see which flags the rule actually catches at scale.

Changelog

Added

• Add `anomaly/rules/specialty_scopes.py` to scan silver and produce per-specialty HCPCS whitelists with coverage and volume thresholds
• Add `local_pipeline/anomaly/specialty_scopes.parquet` (86,924 rows, 130 specialties, 27% in-scope) and matching `specialty_scopes_summary.json`
• Add `web/src/app/investigations/page.tsx` with sortable, filterable list of Claude-generated briefs
• Add `web/src/app/investigations/[id]/page.tsx` detail view with analyst decision buttons, notes, and rendered brief sections
• Add `web/src/lib/investigations.ts` with client-side fetch helpers and shared types
• Add `web/scripts/sync-briefs.mjs` (`npm run sync-briefs`) to copy briefs into `web/public/data/investigations/` with a parsed rule summary
• Add `Investigations` entry to the navbar via `web/src/lib/constants.ts`

Changed

• Update `anomaly/retrieve_context.py` to load the scope table and populate `out_of_specialty_pct` plus `out_of_specialty_codes` per NPI-year
• Update `anomaly/check_rules.py` OUT_OF_SPECIALTY rule from stubbed NOT EVALUABLE to a real check that triggers above 20% out-of-scope services
• Update `web/package.json` with a `sync-briefs` script

Fixed

• Fix MUI v9 Stack prop regression (move `alignItems` and `justifyContent` into `sx`)
• Fix invalid HTML in list table (switch from `TableRow component={Link}` to `onClick` with `useRouter().push`)
• Fix MUI v9 icon rename (`CheckCircleOutline` is now `CheckCircleOutlineOutlined`)

Devlog

I closed out Phase 9D today by regenerating the 10 investigation briefs against the now-live OUT_OF_SPECIALTY rule and adding an NPI redaction toggle. Both were finish-line items that turn the prior research-grade pipeline into something that can actually be demoed to a stakeholder without compromising provider identifiability.

The brief regeneration surfaced an interesting interaction with Anthropic's prompt cache. My first regen pass failed on the 6th of 10 calls with a 529 overloaded response, which wiped out the savings the cache was supposed to provide. I added an exponential-backoff retry for 429 and 529 status codes, then padded the system prompt from about 350 tokens to 2584 with substantive Medicare interpretation material: HCPCS bucket reference, specialty-by-specialty billing norms, and rule-specific weighting heuristics. The padding was not filler. It crossed Sonnet 4.6's 2048-token cache minimum AND it materially improved brief quality: the regenerated Internal Medicine brief correctly flags a J7170 hemophilia A biologic billing pattern as out-of-specialty with 99.7% concentration, and the Multispecialty Clinic brief dropped from MEDIUM to LOW once the model had explicit context about umbrella-specialty label drift. Post-padding, cache_read hit 2578 on every call after the first. Total spend for 10 briefs came to $0.42.

The NPI redaction question came up when I was writing the progress report. I had been casually naming NPIs in the devlog until I realized these are real Medicare providers in real CMS data, not synthetic. NPPES makes them public, but that does not make it appropriate to show them in a portfolio demo or public screenshot. I built two independent layers: a runtime toggle in both the list and detail headers that masks to format 1033****74 and persists to localStorage, and a build-time --mask-npis flag on sync-briefs.mjs that rewrites the deployed JSONs so devtools can't recover the raw IDs. Setting NEXT_PUBLIC_REDACT_NPIS=1 at build time locks the toggle on for public deploys.

One finding worth noting: the spec originally imagined OUT_OF_SPECIALTY would automatically catch the optometrist-doing-cataract-surgery case. It does not, because the rule is peer-relative. Roughly 28% of optometrists in the CMS data actually bill HCPCS 66984, reflecting real-world co-management arrangements and scope-of-practice expansions in some states. That NPI still shows CRITICAL, but via HIGH_INTENSITY and PROCEDURE_CONCENTRATION, not out-of-specialty. "What a specialty is allowed to bill" and "what a specialty actually bills in CMS data" are different populations, and a peer-relative rule can only see the second.

What's next: Phase 9 is closed. The immediate follow-ups are optional: scale to --top-n 100 to validate calibration at larger N (estimated under $2 with caching), or pivot entirely to the original stakeholder demo now that the UI is deployable on Vercel.

Changelog

Added

• Add retry-with-backoff for 429 and 529 responses in anomaly/generate_brief.py
• Add substantial interpretation material to SYSTEM_PROMPT: HCPCS bucket reference, specialty norms, per-rule weighting heuristics (prompt tokens: 350 to 2584, crosses Sonnet 4.6 cache threshold)
• Add runtime NPI redaction toggle to /investigations list and detail pages (format 1033****74, localStorage persisted, cross-tab synced)
• Add --mask-npis build flag (and npm run sync-briefs:masked) to scripts/sync-briefs.mjs for public deploys
• Add NEXT_PUBLIC_REDACT_NPIS env var support that locks the redaction toggle on
• Add phase-9d-complete git tag on the merge commit

Changed

• Regenerate all 10 top-severity investigation briefs with OUT_OF_SPECIALTY now evaluable (2 of 10 now trigger the rule; the Multispecialty Clinic brief correctly recalibrated from MEDIUM to LOW)
• Track web/public/data/ (previously ignored via top-level data/ rule) so Vercel can serve /investigations as static JSON
• Deploy /investigations route via PRs #3 and #4 merged to main

Fixed

• Fix cache-miss-on-every-call issue by padding SYSTEM_PROMPT above 2048 tokens
• Fix transient API failures cascading into full-batch aborts via retry logic