Cellar Index
System design

Architecture

Two paths share one feature pipeline. Each quarter an offline refresh turns public data into a database and three trained models; on every page view the model service reads that database and runs the active model live.

Cellar Index system architectureOffline, each quarter: public data sources are parsed and linked into PostgreSQL, three models are trained with a shared feature package, stored in a versioned model registry, and their results loaded back into the database. Online, on every request: the browser talks to the Next.js frontend, which calls the FastAPI model service; the service reads PostgreSQL and runs the active model in memory for live and what-if predictions.OFFLINE · QUARTERLY REFRESH (EventBridge Scheduler → refresh-aws.sh)ONLINE · EVERY REQUESTSourcesIngestion (backend/ingest)DatabaseTraining (ml/)Model registryPLCB price lists40 quarterly PDFs, 2016–2026NASA POWERdaily weather, 49 regionsUSDA crush reportsCalifornia supply, 2009–2025Parse & classifyPDF text → wine, region,brand, grapeLink vintagescodes + names → 31k winesGrowing-season featuresweather anomalies, crop supplyPostgreSQLwinesprice_observationsregionsmodel_runsforecastsvintage_outlookfair_pricesExporttraining CSV + context tablesTrain on SageMakerprice-change · next-vintage ·fair-price (Processing job)wineprice packageshared features & modelsLoad resultsforecasts, outlooks,fair prices, metricsModel registryversioned artifactsin S3one active versionweather & supply tablesexportsaveload forecasts & metrics into the databaseBrowserwine lists, charts,what-if panelNext.js frontendserver-rendered pages/api/predict proxy forwhat-if requestsFastAPI model servicedata endpoints, OpenAPI docslive & what-if predictionsexplanations (Bedrock)Active modelheld in memory,hot-swapped when a newversion is activatedHTTPSJSONSQL readsload artifactpredictsame feature code(parity-tested)
Data flowShared code Core services

Offline: the quarterly refresh

  1. Download the state's quarterly price-list PDFs, NASA POWER weather and USDA crush reports.
  2. Parse the PDFs, identify wines, and extract region, brand, grape and classification from names.
  3. Link the same wine across quarters and name changes; each vintage is its own wine.
  4. Export training data with context tables and train three models using the shared wineprice package.
  5. Evaluate on held-out time periods, save a versioned artifact, and load its results into the database.

Online: every request

  1. The browser loads pages rendered by Next.js.
  2. Next.js calls the FastAPI model service for wines, prices, forecasts and model metrics.
  3. The service reads PostgreSQL and keeps the active model in memory.
  4. What-if requests re-run the model live on the wine's history with the changed price or promotion.
  5. A parity test checks that live predictions match the batch forecasts exactly.

How it maps to AWS

What runs where on AWS, chosen to keep running costs low (about $17 a month).

ComponentStatusHow it runs
Website + model serviceLiveOne EC2 t4g.small (Ubuntu, arm64): nginx routes / to Next.js and /api/v1 to FastAPI
HTTPS and cachingLiveCloudFront in front; the server only accepts traffic from CloudFront, and has no SSH
DatabaseLivePostgreSQL on the same instance, restored from a dump at each deploy; nightly backups to S3
InfrastructureLiveAWS CDK (Python): every deploy rebuilds the server from source, reproducibly
Cost guardrailLiveAWS Budgets alerts at $5 and $20 a month
Quarterly trainingLiveEventBridge Scheduler -> SSM Run Command on the server -> SageMaker Processing job (ml.t3.xlarge) trains all three models
Model registryLiveTrained models are published to S3 (registry/<version>/) and hot-swapped by the API
Plain-English explanationsLiveAmazon Bedrock (Amazon Nova Lite), generated per wine on request, cached and rate-limited

Details on the models themselves are on How it works.