ML Engineer Notes: ship the model that fits

Module 1: Collect and store data

Domain 1 starts with where the bytes are. Amazon S3 is the usual landing zone for training files. The exam guide also lists Amazon EBS, Amazon EFS, Amazon RDS, Amazon DynamoDB, and Amazon OpenSearch Service as sources you may have to extract from. Pick the store for cost, performance, structure, and compliance. A hot training set does not belong in a cold archive you must restore before every job.

Streaming ingest uses Amazon Kinesis, Apache Flink, or Apache Kafka. Use a replayable stream when a later job must read the same hour again. Merge sources with code, AWS Glue, or Apache Spark when two systems describe the same entity with different keys. Store the result in a format that matches the read: Apache Parquet or ORC when later jobs need columns, CSV or JSON when the file is only landing.

Vector workloads are new in MLA-C02. OpenSearch Service, Amazon RDS with pgvector, and Amazon S3 are the guide’s examples of scalable vector storage. Text, images, and audio are all in-scope data types. SageMaker Feature Store is where shared features live so training and live inference read the same definition.

Module 2: Quality, labels, and bias

Validate quality with DataBrew or AWS Glue Data Quality before a bad partition reaches training. Label and annotate when the raw text has no target. Split, shuffle, and augment to reduce bias, and check metrics across numeric columns, text, and images. Class imbalance makes accuracy look strong while the rare class is ignored. Resample or augment. Do not duplicate only the majority class.

Clean the table: treat outliers that are errors, impute missing values with a fill that is not a real category, and deduplicate retries. For a fine-tune, validate that each prompt has an acceptable response and screen the pair for unsafe content. A feature that exists only in the training file will be missing at inference. Drop it now or the served model will not match the trained one.

Module 3: Choose the approach

Evaluate foundation models in Amazon Bedrock against the task, then the latency and the price. Identify a fine-tuning strategy only when prompting is not enough and you have the pairs. Compare a custom trainer, a managed AI service, a pre-trained model, and a foundation model. Retrieval is an architecture choice: use it when the answer must come from your documents.

Amazon Textract reads documents. Amazon Transcribe turns speech into text. Amazon Comprehend analyzes text you already have. Amazon Rekognition labels images. Amazon Polly speaks. Amazon Translate changes language. Do not train a model for a problem one of those APIs already solves, unless the stem requires your weights or your network boundary.

The target candidate does not design the company’s AI strategy. The question wants the implementation that meets the constraint, not a program charter.

Module 4: Train, fine-tune, and customize

Use a SageMaker AI built-in algorithm when it covers the task. Use script mode with a supported framework when you must bring the training loop. Automatic model tuning searches hyperparameters. Early stopping ends a job whose validation metric has stalled. Distributed training cuts wall-clock time when one machine is too slow and the metric is still improving.

Overfitting memorizes the training set. Underfitting never learns it. Catastrophic forgetting is a fine-tune that drops earlier skills. Combine models when different slices need different specialists. Epoch count, step count, and batch size are the fundamental knobs. A batch that does not fit in memory has to shrink.

Prompt engineering is the light customization. Fine-tuning is the heavier one. Optimize the embedding model and the retriever when citations are wrong and the generator is fine on a gold passage. Extra Pruning, quantization-style compression, and “shrink the file” were removed from this task. Do not pick them as the MLA-C02 fix for overfitting.

Module 5: Evaluate

Record experiments with MLflow on SageMaker AI, Amazon Bedrock evaluations, or Bedrock Prompt Management. Set a baseline before you call a drop a regression. Compare a shadow variant when users must not see the candidate yet. Explain a single prediction when the stem asks why one decision happened. Debug a flat loss as a convergence problem, often the learning rate, before you resize the cluster.

Text metrics on this exam include BLEU, ROUGE, BERTScore, and semantic similarity. Human review needs a rubric. LLM-as-a-judge scores outputs with a model and still needs a check. For retrieval, measure whether the right passage came back, not only whether the final sentence sounded fluent.

Module 6: Deployment targets

Real-time inference answers a live request. Batch inference scores a stored dataset. A multi-model endpoint hosts several rarely used models behind one endpoint. A multi-container endpoint runs pieces together, such as a preprocessor and a model. Pick a Bedrock deployment option for a foundation model you are not training yourself. Custom Model Import brings in a model that was built outside AWS.

Agents need a task, tools, and a way to communicate. Configure retrieval strategy and reranking when the order of passages matters. Extra SageMaker Neo, the edge compiler, was removed from this task.

Module 7: Capacity and agent infrastructure

Use on-demand capacity for uneven work and provisioned capacity for a peak you must hit. Build containers for ML runtimes and store images in Amazon ECR. Place a SageMaker AI endpoint in a VPC when traffic must stay private. Deploy with the SageMaker AI SDK, the AWS CLI, or Boto3 when the stem asks for automation rather than a click.

Scale on a metric that matches the bottleneck, such as invocations per instance for a real-time endpoint, and scale GPU capacity for GPU workloads. A Bedrock knowledge base needs a vector store, an index, and a refresh when documents change. Agent state is what the agent remembers between steps. Amazon Bedrock AgentCore is the in-scope agent platform. Extra “Bring your own container with SageMaker” was removed as a named skill. Prefer a built-in algorithm or script mode when they fit.

Module 8: Pipelines

Automate the deploy and keep a rollback to the last good version. AWS CodePipeline orders the stages. AWS CodeBuild builds. AWS CodeDeploy deploys. AWS CodeCommit and AWS CodeConnections cover the repository connection. Test classical models and prompts in the pipeline, including a golden question set. Retrain when drift says the model is stale, not on a timer that ignores the metric.

SageMaker Model Registry and MLflow record the model version. Bedrock Prompt Management records the prompt. Agent deploys need their own version. A fine-tuned model’s version should move through the same gate. Refresh a knowledge base as a pipeline stage and test retrieval before you cut over.

Module 9: Monitor inference

Amazon CloudWatch, including generative AI observability, and Amazon Bedrock evaluations watch production quality. Drift detection catches an input distribution that no longer matches training. An A/B test splits users across variants. A shadow variant copies traffic and does not answer the user.

Agent monitoring adds coordination failures, truncated streams, and tool failures. A fluent answer after a failed tool call is a defect. AWS X-Ray and AgentCore Observability show which dependency consumed the time. Fix that dependency before you add GPUs to a model that is waiting.

Module 10: Cost and performance

Choose an inference instance family that meets latency at a lower price. Dashboards belong in CloudWatch. AWS Budgets and Cost Explorer watch spend. Spot fits interruptible training. It is a poor fit for a user-facing endpoint that cannot stop. Savings Plans and reserved capacity fit a steady fleet.

Foundation-model cost is not only the instance hour. Count tokens, embedding computation, and vector-index storage. A prompt template that grew without an accuracy gain is a cost bug. Duplicate chunks in a vector index are a storage bug.

Extra The guide removed EventBridge-as-infrastructure-monitoring and the old “provisioned concurrency and service quotas” troubleshooting example from this task. EventBridge remains an in-scope integration service. Auto scaling metrics still belong with provisioning.

Module 11: Secure the workload

Fail the pipeline when Amazon Inspector or Amazon CodeGuru finds a vulnerability you do not accept. Least privilege on the training role and the endpoint role means the artifact bucket and the actions that job needs, not s3:* on every bucket. Limit iam:PassRole to the roles the notebook may launch.

IAM policies and roles are the default credential. The exam also allows an Amazon Bedrock API key, stored as a secret, not in Git and not as the root key. CloudTrail records who called the API. AWS Config records whether the resource stayed in the approved shape. Put endpoints in private subnets and security groups that name the client.

Amazon Macie finds sensitive data already in S3. Amazon Bedrock Guardrails filter denied topics and sensitive values on the model path. Logs should not keep a raw identifier the guardrail just stripped. KMS encrypts the artifact. Encryption does not replace the IAM allow.

Module 12: What changed from MLA-C01

The four-domain shape stayed. Domain 2 moved from 26% to 24% of scored content, and domain 3 moved from 22% to 24%. Domain 1 is still 28%. Domain 4 is still 24%. The names now say ML and AI, and foundation models sit beside classical models.

Additions you should be able to use: vector stores, embeddings, RAG chunking, fine-tune and distillation data, Bedrock model choice, BLEU and ROUGE, knowledge bases, agent state, prompt versioning, token and vector cost, Bedrock API keys, and Guardrails. Removals you should not spend study time on: SageMaker Neo, pruning as the way to shrink a model, and mounting EFS or FSx as the required trainer load step.

MLS-C01, the Machine Learning Specialty exam, is a different retired certification. Its last day was March 31, 2026. Studying it will not book you a seat.

Frequently asked questions

Service names and prices change. Use the current AWS exam guide as the authority when it differs from these notes.