MLA 014 Machine Learning Hosting and Serverless Deployment

Builders can scale ML from simple API calls to full MLOps pipelines using SST on AWS, utilizing Aurora pgvector for search and Spot instances for 90 percent cost savings. External platforms like Modal or GCP Cloud Run provide superior serverless GPU options for real-time inference when AWS native limits are reached.

Links

• Notes and resources at ocdevel.com/mlg/mla-14
• Try a walking desk - stay healthy & sharp while you learn & code
• Generate a podcast - use my voice to listen to any AI generated content you want

Core Infrastructure

SST uses Pulumi to bridge high-level web components (API, Database) with low-level AWS resources (SageMaker, GPU clusters). The framework enables infrastructure-as-code in TypeScript, allowing developers to manage entire ML lifecycles within a single configuration.

Level 1-2: Foundational Models and Edge Inference

• AWS Bedrock: Managed gateway for models including Claude 4.5, Llama 4, and Nova. It provides IAM security, VPC isolation, and integrated billing.
• Knowledge Bases: Automates RAG pipelines by chunking S3 documents and storing embeddings in Aurora pgvector.
• Cloudflare Workers AI: Runs open-source models (Llama, Mistral, Flux) on edge GPUs. Pricing uses "Neurons" units, measuring compute per request rather than tokens.

Level 3-4: Cost-Effective CPU and Batch Processing

• Lambda Inference: Use ONNX-formatted models on AWS Lambda with SnapStart to minimize costs and 16-second cold starts.
• Vector Search: The SST Vector component manages semantic search within existing Aurora PostgreSQL databases using pgvector, matching dedicated database performance.
• SST Task: Runs Fargate containers for CPU-bound ETL and data preprocessing.
• AWS Batch: Orchestrates GPU training on EC2. Using Spot instances reduces costs by 60 to 90 percent, with checkpointing protecting against instance reclamation.

Level 5: Real-Time GPU Inference

• AWS Options: SageMaker Real-Time endpoints support scale-to-zero since late 2024. SageMaker Async handles large payloads via S3 queues.
• External Alternatives:
  • GCP Cloud Run: Offers serverless L4 and Blackwell GPUs with per-second billing.
  • Modal: Python-native serverless GPU platform with 2 to 4 second cold starts.
  • Groq: Uses LPU hardware for LLM inference, reaching 1300 tokens per second.
  • RunPod: Provides the lowest raw GPU pricing and FlashBoot for fast starts.

Level 6-7: MLOps and Mature Production

• SageMaker Platform: Includes Studio for IDE work, JumpStart for one-click model deployment, and Model Registry for version tracking.
• Monitoring: Use Arize Phoenix or Evidently AI to detect data and concept drift. Log all predictions to S3 for weekly distribution analysis.
• Hardware Optimization: AWS Inferentia and Trainium chips offer 70 percent lower inference costs compared to GPUs. Transition becomes viable when monthly GPU spend exceeds 10,000 dollars.
• Self-Hosting: API calls are cheaper until volume reaches 30 million tokens daily. For self-hosting, use vLLM for high-throughput PagedAttention.