From on-prem GPUs to elastic AWS training

A Retail business working with Cloud Combinator on AWS. The client is anonymised at their request.

The challenge had three focus areas, each about turning a fixed on-premises setup into a scalable, well-governed AWS training pipeline.

We delivered the migration as a sequence of milestones, each with a clear deliverable and acceptance point, so the client could see progress and validate results at every stage.

All three focus areas were delivered and accepted. The client's demand-forecasting training and tuning now run on Amazon SageMaker, reproduced faithfully from their existing repository, on infrastructure that is entirely defined as code and monitored end to end.

• Elastic GPU compute — The client moved from two fixed RTX 3090 cards to on-demand SageMaker GPU tiers, with a multi-GPU ml.g5.12xlarge (4x NVIDIA A10G, 96 GB VRAM) for hyperparameter tuning and single-GPU ml.g5.xlarge for standard runs.
• Faithful reproduction — Their custom forecasting repository and full Python dependency stack, including the private statsforecast dependency, run inside SageMaker, with tuning results validated as comparable to the on-premises environment.
• Repeatable infrastructure — The entire environment, SageMaker Studio, S3, VPC, IAM and CloudWatch, deploys from a single CloudFormation template with least-privilege access throughout.
• Full observability — CloudWatch dashboards and alarms give the client a live view of training runtime, cost and job failures across the pipeline.
• Handover — The client received a runbook, reference architecture and a knowledge-transfer session, and signed off that they can operate, maintain and scale the pipeline independently.

With training no longer bound to the hardware on the desk, the client can scale experiments up and down as their models demand, and the GPU spend on SageMaker also builds toward the NVIDIA Inception Programme credits the team is pursuing. The pipeline is no longer capped by fixed on-premises GPUs, but set up to support whatever the client takes on next.