Root Cause Analysis
Four signals, four questions: the one model for picking telemetry
A dashboard spike, a single failed transaction, slow latency across services, a config that changed with no deploy: four symptoms, and AWS answers each with a different signal. Naming which signal a DVA-C02 troubleshooting stem is really asking for, before you touch any code, is the entire skill here. Everything later on the page is a deeper look at one of these four signals.
The single mental model: AWS root cause analysis reads four complementary signals, and each answers exactly one question. Match the symptom in the stem to the question, and the service falls out.
- Amazon CloudWatch Logs answers "what exactly happened in this one request?", the line-level event record. A log group is the container for a resource's log streams (for example
/aws/lambda/<function-name>); the SDK and managed services write timestamped text or JSON events into it. Reach for logs when you have a single failed transaction and need its exact error. - Amazon CloudWatch metrics answer "how often, how much, and is it trending?", numeric time series such as error count or p99 latency, on which you set alarms. A metric is a published time series identified by a namespace plus a set of dimensions[1]. Reach for metrics when a dashboard shows a spike but no one request is obviously at fault.
- AWS X-Ray answers "where in the call chain did time or errors accumulate?", one request followed across every service it touches. Reach for it when end-to-end latency degrades across microservices but no single component looks broken.
- AWS CloudTrail answers "who called which AWS API, when, and from where?", the control-plane audit trail. Reach for it when behavior changed without a deployment you initiated, or a call started being denied.
The DVA-C02 reflex is this mapping itself. A spike on a dashboard is a metrics question; a single failed transaction is a logs question; slow end-to-end latency across services is a trace (X-Ray) question; an unexplained configuration or permission change is a CloudTrail question. The most common exam time-waster, and the most common distractor, is reaching for the wrong signal, such as grepping raw logs to judge whether errors are trending (a metrics job) or treating CloudTrail as an application debug log (it records API calls, not your code's internal events). The mechanics sections below define each signal's searchable key; the decision flow at the end of the page turns this mapping into a branch you can walk under time pressure.
Logs and metrics: querying events vs. measuring trends
This section explains the first two signals in depth, what Amazon CloudWatch Logs and Amazon CloudWatch metrics let you do once data is flowing, and the two tools that bridge them. It assumes only that your function or service is already writing to a CloudWatch log group.
CloudWatch Logs Insights, the reactive hunt. Once logs are centralized, CloudWatch Logs Insights[2] is an interactive query engine with a purpose-built query language (the fields, filter, stats, sort, and limit commands). You run it ad hoc during an investigation to find the exact log events behind one failure, and it can query up to a number of log groups at once[2]; a query that has not finished is stopped after it runs for 60 minutes[3]. Use Logs Insights when the question is "what was the error on request X?"
Metric filters, turning log text into a measurable trend. A metric filter[4] is the opposite of an ad hoc query: it is a standing rule that scans every incoming log event for a pattern (for example the term ERROR or a 5xx status) and emits a numeric CloudWatch metric you can then alarm on. So Logs Insights is the reactive hunt; a metric filter plus a CloudWatch alarm is the proactive trigger that pages you before you start hunting. This is the resolution of a tension the exam likes: if the stem asks you to spot a trend or rate, do not grep logs by hand, convert the log pattern into a metric with a metric filter, then alarm on the metric.
CloudWatch metrics, namespaces, and dimensions. A metric is a numeric time series. It is identified by a namespace (a container that isolates one service's metrics from another's, for example AWS/Lambda) plus up to a fixed number of dimensions (name/value pairs such as FunctionName=checkout that scope the series), as described in the CloudWatch concepts[1]. Dimensions are how you narrow a metric to one resource on a dashboard.
Reading the Lambda metrics, the triage that decides which layer to debug. For AWS Lambda the published metrics[5] split the problem before you read any code:
- Errors counts invocations that ended in a function error, an unhandled exception or a timeout raised inside your handler. A rising Errors line points at your code.
- Throttles counts invocations rejected because concurrency was exhausted. A throttle is an invocation error that happens around your code, surfaced as
TooManyRequestsException/ HTTP429, and it is counted in Throttles, not Errors. The reconciliation to hold onto: a flat Errors line with rising Throttles means concurrency was exhausted (raise reserved or provisioned concurrency, or the account limit); it does NOT mean your handler is broken. - Duration measures execution time; a climbing Duration that nears the configured timeout is what eventually turns into Errors (timeouts) and is the metric to watch for performance regressions.
The rule that ties logs and metrics together: metrics tell you that something is wrong and bound the time window; logs (via Insights) then tell you exactly what failed inside that window.
X-Ray and CloudTrail: locating the slow hop vs. auditing the change
This section covers the remaining two signals, AWS X-Ray for distributed tracing and AWS CloudTrail for the API audit trail, and why neither substitutes for the other. It assumes you have a multi-service request path (the case where logs and metrics alone are not enough to localize the fault).
X-Ray: segments, subsegments, and the service map. X-Ray records each request as a segment, the work done by one service for that request, and breaks the calls that service makes to downstream resources into subsegments. Segments that share one trace ID are stitched into a service map, the visual graph of services and the edges between them, color-coded for errors, faults, and throttles[6]. The service map is what lets you see where in the call chain time or errors accumulate, rather than just that they did.
Annotations are indexed and filterable; metadata is not. To make traces searchable you attach annotations, key-value pairs that X-Ray indexes (up to 50 per trace) and that you can use in filter expressions and the GetTraceSummaries API[6]. Metadata is also key-value, but it is NOT indexed, so it rides along with the trace for context yet cannot be searched on. The exam-load distinction, stated where it bites: put the field you will filter or group by (such as userId or orderType) in an annotation; put bulky diagnostic detail in metadata. If a stem asks you to filter traces by a value and proposes recording it as metadata, that is the wrong answer, metadata is not searchable.
Default sampling, why not every request is traced. To control cost and overhead, X-Ray does not trace every request. The default sampling rule records the first request each second plus five percent of any additional requests[7]. The consequence for RCA: a low-volume intermittent error may simply not have been sampled, so absence of a trace is not proof the request succeeded, you raise the sampling rate (or add a targeted rule) when you need fuller capture.
CloudTrail: the API-level audit, for the question the others can't answer. When a resource changed or a call was denied at the control plane (a security group edited, an IAM role's permissions changed, an unexpected API call), none of logs, metrics, or traces records it, because they observe your application, not the AWS control plane. CloudTrail[8] is the audit trail that does: it records management-event API activity, who (the principal), what (the API call), when, and from where (source IP). The reconciliation the exam tests: CloudTrail is not an application debug log. It tells you that someone called ModifySecurityGroupRules at 14:02; it does not contain your function's internal log lines or request payloads, for those you read CloudWatch Logs. Use CloudTrail only for the "who or what changed it" question.
Correlating signals and exam-pattern recognition
This section is the page's payoff: how to combine the four signals into one diagnosis, and how to map recurring DVA-C02 stem cues to the right answer. Each rule restates a fact from the sections above (see those for the full explanation). The decision flow below turns this into one branch you can walk under time pressure: name the question, take its branch to the one signal, then converge on correlating the rest by timestamp.
Correlate, never trust one signal alone. The disciplined RCA loop is to align timestamps across signals: when a metric spikes, note the exact time window, then pull the matching X-Ray traces, the CloudWatch Logs lines, and any CloudTrail event inside that same window. A 5xx spike whose window lines up with a CloudTrail ModifySecurityGroupRules event is a configuration regression, not a code bug, a conclusion no single signal could reach.
HTTP status triage, whose fault is it. A 4xx generally means the caller (your application or the client) sent a bad request; a 5xx means the serving side faulted. On a multi-service path, narrow a 5xx by which hop reported it on the X-Ray service map before assuming the fault is yours.
The stem-cue table:
- Cue: "a specific request failed," "one transaction returned the wrong result," "find the exact error." → CloudWatch Logs Insights to pull that request's log lines (and its X-Ray trace if instrumented). Distractor that fails: CloudTrail, it audits AWS API calls, not your application's internal events.
- Cue: "error rate / latency / throttles jumped on the dashboard," "is it trending," "alert me automatically." → CloudWatch metrics with an alarm, and a metric filter if the signal lives only in log text. Distractor that fails: grepping raw logs by hand to judge a trend, slow and error-prone.
- Cue: "end-to-end latency is high across microservices," "which service is slow," "trace the request path." → AWS X-Ray service map and trace timeline. Distractor that fails: reading a single service's logs, it cannot show where across hops the time accumulated.
- Cue: "need to filter/group traces by a field (userId, orderType)." → record that field as an X-Ray annotation (indexed). Distractor that fails: recording it as metadata, which is not indexed and cannot be searched.
- Cue: "something changed with no deployment," "a call started being denied," "who edited this resource." → CloudTrail management-event history (principal, API, time, source IP). Distractor that fails: CloudWatch Logs, it does not record control-plane API activity.
- Cue: "Lambda throttling," "
TooManyRequestsException," "429," "errors metric is flat but invocations are failing." → read the Throttles metric, not Errors; the cause is exhausted concurrency, raise reserved/provisioned concurrency or the account limit. Distractor that fails: debugging handler code because you looked at the Errors metric, which a throttle does not increment. - Cue: "unhandled exception / timeout inside the handler." → a function error, counted in the Errors metric; fix the code, and watch Duration if a climbing runtime is hitting the timeout. Distractor that fails: raising concurrency, which addresses throttles, not a code fault.
The through-line: name the question the stem is really asking (what / how-often / where / who), pick the one signal that answers it, then correlate the others to confirm, exactly the flow the decision diagram above walks.
CloudWatch Logs vs CloudWatch Metrics vs AWS X-Ray vs CloudTrail: what each answers
| Dimension | CloudWatch Logs | CloudWatch Metrics | AWS X-Ray | CloudTrail |
|---|---|---|---|---|
| Question it answers | What happened in this request? (event detail) | How often / how much / is it trending? (numeric) | Where in the call chain did time or errors accumulate? | Who called which AWS API, when, from where? |
| Data shape | Timestamped text/JSON log events | Numeric time-series metrics | Segments/subsegments stitched into traces + service map | API activity audit records (management/data events) |
| Primary tool to investigate | Logs Insights queries; metric filters to alarm | Dashboards and alarms on metrics | Service map, trace timeline, filter expressions on annotations | Event history / lookup, log file delivery to S3 |
| Search / filter key | Logs Insights QL over log fields | Namespace + dimensions | Annotations (indexed); metadata is NOT searchable | Event name, principal, source IP, time |
| Best for in RCA | Pinpointing one transaction's exact failure | Spotting a spike and bounding the time window | Isolating the slow or failing hop across services | Tracing a config/permission change that caused the issue |
Decision tree
Sharp facts the exam loves — give these one last read before exam day.
Cheat sheet
Sharp facts the exam loves — scan these before test day.
- Match the symptom to the question, and the right telemetry signal falls out
AWS root cause analysis reads four complementary signals, each answering exactly one question: CloudWatch Logs answers "what happened in this one request?", CloudWatch metrics answer "how often, how much, is it trending?", X-Ray answers "where in the call chain did time or errors build up?", and CloudTrail answers "who called which AWS API, when, from where?". The DVA-C02 reflex is this mapping itself, so when a stem describes a symptom, name the question first and the service follows. A spike on a dashboard is a metrics question, a single failed transaction is a logs question, slow end-to-end latency across services is a trace question, and an unexplained config or permission change is a CloudTrail question.
Trap Grepping raw logs by hand to judge whether errors are trending up. That is a metrics job, and scanning log text for a rate is slow and error-prone.
- Reach for CloudWatch Logs when you need the exact error on one request
CloudWatch Logs holds the line-level event record: the timestamped text or JSON your code and managed services emit into a log group such as
/aws/lambda/<function-name>. It is the signal that tells you what exactly happened inside a single failed transaction, down to the stack trace or message. Use it when you already know which request failed and need its precise cause, rather than when you are trying to spot a trend across many requests, which is a metrics question.Trap Reaching for CloudWatch Logs to spot a trend across many requests, when a rate or count over time is a metrics question and Logs answers what happened inside one specific request.
- Use Logs Insights as the reactive hunt for one request's failure
CloudWatch Logs Insights is an interactive query engine over your log groups with a purpose-built query language built from
fields,filter,stats,sort, andlimitcommands, run ad hoc during an investigation to surface the exact events behind a failure. It can search multiple log groups in one query, and a running query is stopped after 60 minutes if it has not completed. Reach for it when the question is "what was the error on request X?", the reactive hunt, in contrast to a metric filter, which is a standing rule that watches every new log event.Trap Standing up a Logs Insights query to continuously watch for and alert on new failures, when Insights is an on-demand reactive hunt and a metric filter plus an alarm is the proactive standing rule.
7 questions test this
- A developer is troubleshooting an Amazon API Gateway REST API that returns intermittent 504 Gateway Timeout errors. The developer has…
- A developer operates a microservices application on Amazon ECS with Fargate. The application uses the awslogs log driver to send container…
- A developer is troubleshooting a microservices application running on AWS Lambda. The application generates structured JSON logs containing…
- A developer is troubleshooting an AWS Lambda function that is experiencing intermittent timeout errors. The developer needs to identify…
- A company's API Gateway REST API is experiencing high 4XXError rates according to CloudWatch metrics. The development team suspects that…
- A developer is troubleshooting intermittent timeout errors in an AWS Lambda function that processes orders from an Amazon SQS queue. The…
- A developer is troubleshooting a Python Lambda function that processes payment transactions. The application uses structured JSON logging…
- Convert a log pattern into a metric with a metric filter, then alarm on it
A metric filter is a standing rule on a log group that scans every incoming event for a pattern (say the term
ERRORor a5xxstatus) and emits a numeric CloudWatch metric you can alarm on. This is how you turn unstructured log text into a measurable trend that pages you proactively, rather than discovering the problem only when you go hunting. Logs Insights is the reactive query you run after the fact; a metric filter plus an alarm is the proactive trigger set up in advance.Trap Trying to detect a rising error rate by repeatedly running a Logs Insights query: use a metric filter feeding an alarm so the trend pages you instead of you polling for it.
- A CloudWatch metric is identified by its namespace plus dimensions
A metric is a numeric time series (error count, p99 latency, throttle count) identified by a namespace that isolates one service's metrics (for example
AWS/Lambda) plus a set of dimensions, which are name/value pairs such asFunctionName=checkoutthat scope the series to one resource. Dimensions are how you narrow a chart from "all functions" down to the single function that is misbehaving. You set alarms on metrics, and a metric spike bounds the time window you then drill into with logs and traces.Trap Treating a namespace/dimension pair as interchangeable with another after the fact, forgetting that a dimension is part of a metric's identity, so an alarm created on a specific dimension combination does not retroactively cover series you never published with those exact dimensions.
4 questions test this
- A company uses Amazon API Gateway to expose a REST API. The operations team notices a spike in the 4XXError CloudWatch metric but cannot…
- A developer notices that an Amazon API Gateway REST API is returning intermittent 429 Too Many Requests errors to some clients during peak…
- A development team is investigating intermittent 429 Too Many Requests errors returned by their Amazon API Gateway REST API. The team wants…
- A development team manages an Amazon API Gateway REST API that serves multiple client applications. The team notices a spike in 4XXError…
- A flat Lambda Errors line with rising Throttles means concurrency is exhausted, not broken code
Lambda's Errors metric counts only function errors (an unhandled exception or a timeout raised inside your handler) so a rising Errors line points at your code. Throttles is a separate metric that counts invocations rejected because concurrency was exhausted, surfaced as
TooManyRequestsException/ HTTP429, and a throttle does NOT increment Errors. So if Errors stays flat while Throttles climb, the fix is to raise reserved or provisioned concurrency or the account concurrency limit, not to read your handler code.Trap Debugging handler logic because invocations are failing, when the Errors metric is flat and only Throttles is rising. A throttle is a concurrency limit, not a code fault.
- Watch Lambda Duration climbing toward the timeout before it turns into errors
Lambda's Duration metric measures execution time per invocation, and it is the early-warning signal for a performance regression: a Duration line trending up toward the configured timeout is what eventually turns into function errors once invocations actually hit the ceiling and time out. Treat a rising Duration as the metric to investigate proactively, since by the time it shows up as timeouts in the Errors metric the user impact has already begun.
Trap Waiting for the Errors metric to rise before investigating a slowdown, since by then invocations are already timing out, whereas a Duration line climbing toward the configured timeout warns you while there is still headroom to act.
- Distinguish a function error from an invocation error before you touch code
A function error is raised inside your handler (an unhandled exception or a timeout) and is the case where you actually debug your code. An invocation error happens before or around your code: a throttle when concurrency is exhausted (
TooManyRequestsException/429), or a permission/configuration failure such as the caller lackinglambda:InvokeFunction. Read the error type and the metric first, because the two classes point at different layers: fix the handler for a function error, but fix concurrency limits or IAM permissions for an invocation error.Trap Reading application logs to fix a "not authorized to perform lambda:InvokeFunction" failure. That is an invocation-level permission problem, not a bug in your handler.
- Use X-Ray to find where in a multi-service call chain the time or errors accumulate
X-Ray records each request as a segment (the work one service did) and breaks that service's downstream calls into subsegments, then stitches all segments sharing one trace ID into a service map, the visual graph of services and the edges between them, color-coded for errors, faults, and throttles. Reach for it when end-to-end latency is high across microservices but no single component looks broken: the service map shows which hop accumulates the time, which a single service's logs cannot reveal.
Trap Reading one service's logs to explain high end-to-end latency across services: logs from a single hop cannot show where across the chain the time was spent.
- X-Ray default sampling traces the first request each second plus 5% of the rest
To control cost and overhead X-Ray does not trace every request: the default sampling rule records the first request each second (the reservoir) plus five percent of any additional requests that second. The consequence for root cause analysis is that a low-volume intermittent error may simply not have been sampled, so the absence of a trace is not proof the request succeeded. When you need fuller capture, raise the sampling rate or add a targeted sampling rule for the affected path.
Trap Concluding an intermittent request succeeded because no X-Ray trace exists for it, when default sampling captures only the first request each second plus 5% of the rest, so a missing trace may just mean it was not sampled.
- Use CloudTrail to find who called which AWS API, when, and from where
CloudTrail is the control-plane audit trail: it records management-event API activity, the principal who made the call, the API action, the timestamp, and the source IP. Reach for it when behavior changed without a deployment you initiated, or a call suddenly started being denied, for example a security group was edited or an IAM role's permissions changed. Logs, metrics, and traces observe your application, so none of them captures a control-plane change; CloudTrail is the only one of the four signals that answers "who or what changed it?".
Trap Hunting through CloudWatch Logs or X-Ray for the cause of an unexplained permission or config change, when those observe your application and only CloudTrail records who called the AWS API that changed it.
- CloudTrail is an API audit, not an application debug log
CloudTrail records that a principal called an API (say
ModifySecurityGroupRulesat 14:02 from a given IP) but it does not contain your function's internal log lines, error messages, or request payloads. For those you read CloudWatch Logs. Keep the boundary sharp: CloudTrail answers the "who changed the configuration" question only, and using it to debug application behavior will turn up nothing because it never sees inside your code.Trap Searching CloudTrail for an application's runtime error or request payload: it logs AWS API calls, not your code's internal events; read CloudWatch Logs for those.
- Triage a status code by side: 4xx blames the caller, 5xx blames the server
A 4xx status generally means the caller (your application or the client) sent a bad request (malformed input, missing auth, a 429 throttle), while a 5xx means the serving side faulted. This split tells you which direction to investigate first. On a multi-service path do not assume a 5xx is your code: narrow it by checking which hop reported the 5xx on the X-Ray service map, since the failing service may be a downstream dependency rather than your own.
Trap Assuming every 5xx on a multi-service path is your own service's fault, when the X-Ray service map may show the 5xx originated at a downstream dependency you call rather than in your code.
The disciplined RCA loop aligns timestamps across signals: when a metric spikes, note the exact window, then pull the X-Ray traces, the CloudWatch Logs lines, and any CloudTrail event inside that same window. A 5xx spike whose window lines up with a CloudTrail
ModifySecurityGroupRulesevent is a configuration regression, not a code bug, a conclusion no single signal could reach. Metrics localize the blast radius and bound the window; logs and traces then explain exactly what failed inside it.Trap Declaring a code bug from a 5xx spike alone, when correlating its time window against CloudTrail can reveal a
ModifySecurityGroupRulesevent that points at a configuration regression rather than your code.- Read an ECS task failure from stopCode and stoppedReason first, and treat exit code 137 as out-of-memory
When ECS tasks start then stop, run describe-tasks and read the stopCode (the category, such as ResourceInitializationError or EssentialContainerExited) and stoppedReason (the human-readable detail) before anything else. The most common container-level cause is a SIGKILL: a container exit code of 137 (128 + 9) means the OOM killer terminated it for exceeding its task-definition memory limit, so the fix is to raise the allocated memory, not to debug code. A ResourceInitializationError points instead at setup failing before the app ran (logging-driver or secrets connectivity).
Trap Reading container application logs to explain a 137 exit when there are none: the process was killed by the OS for exceeding its memory limit, which surfaces in stopCode/stoppedReason, not in app output.
4 questions test this
- A developer is troubleshooting an Amazon ECS service where tasks repeatedly stop shortly after starting. The developer uses the…
- A developer observes that an Amazon ECS service is failing to maintain its desired task count. Tasks start but quickly stop with an exit…
- A developer is troubleshooting an Amazon ECS Fargate service where containers are exiting unexpectedly. The developer examines a stopped…
- A developer is troubleshooting an Amazon ECS Fargate service where tasks are repeatedly stopping shortly after startup. The developer needs…
- A Fargate awslogs ResourceInitializationError is missing connectivity or task-execution-role permission, not bad code
Fargate tasks using the awslogs driver fail at startup with ResourceInitializationError / 'failed to initialize logging driver' for one of two reasons. First, connectivity: a task in a private subnet with no NAT gateway needs an interface VPC endpoint for CloudWatch Logs (com.amazonaws.region.logs), and that same private-subnet rule causes silently missing logs even when the task stays up. Second, permission: the task execution role must allow logs:CreateLogStream and logs:PutLogEvents, which the AmazonECSTaskExecutionRolePolicy managed policy grants. The identical 'unable to pull secrets or registry auth' failure needs the task's security group to allow outbound HTTPS (443) to Secrets Manager / ECR.
Trap Assuming a NAT gateway is enough on its own: the task's security group must still permit outbound 443 to reach CloudWatch Logs, Secrets Manager, or ECR endpoints.
6 questions test this
- A development team deployed an application on Amazon ECS using AWS Fargate. The tasks start but stop unexpectedly after a few seconds. The…
- A developer is running an Amazon ECS service on AWS Fargate and notices that container logs are not appearing in Amazon CloudWatch Logs.…
- A development team deploys an application to Amazon ECS on AWS Fargate. The team configures the task definition to use the awslogs log…
- A developer deploys an Amazon ECS service on AWS Fargate in a private subnet without internet access. The task definition uses the awslogs…
- A developer deployed an Amazon ECS service on AWS Fargate, but tasks are failing to start with a ResourceInitializationError. The stopped…
- A developer has configured an Amazon ECS Fargate task with the awslogs log driver to send container logs to Amazon CloudWatch Logs. The…
- Lambda emits two X-Ray segments: AWS::Lambda is the service, AWS::Lambda::Function is your code
Active tracing makes Lambda record two segments per invocation, so the X-Ray service map shows two nodes for one function. AWS::Lambda is the Lambda service overhead (scheduling, creating or unfreezing the execution environment, and downloading function code) and an error or latency here is a platform/init problem (including throttling), not your code. AWS::Lambda::Function is the handler running, so an error or high latency on that node is in your application logic. Decide which node shows the symptom before you touch anything.
Trap Debugging handler code because the function's X-Ray node shows latency, when the slow node is AWS::Lambda, which is service overhead like cold-start init or throttling rather than your application logic.
5 questions test this
- A developer is investigating a Lambda function failure using AWS X-Ray. The X-Ray trace shows two segments for the function - one with…
- A development team has deployed an AWS Lambda function that is invoked by Amazon API Gateway. The team has enabled active tracing on both…
- A company has deployed a microservices architecture using multiple AWS Lambda functions that call each other and external APIs. Users…
- A developer is troubleshooting performance issues in a serverless application that uses Amazon API Gateway and AWS Lambda. The developer…
- A company runs a serverless application with multiple Lambda functions integrated with API Gateway. Users report intermittent errors from…
- Create a field index to make Logs Insights lookups on a high-volume field fast and cheap
When Logs Insights queries that filter on a field such as requestId scan huge volumes and run slow and expensive, create a field index policy for that field (PutIndexPolicy / console) without changing the log format. Queries written as
filter requestId = 'value'(equality) then skip log events known not to contain the indexed value, cutting scan volume and cost. Across hundreds of log groups, thefilterIndexcommand restricts a query to only the groups indexed on that field.Trap Expecting a field index to speed up a non-equality filter such as a wildcard or range on requestId, when the index only lets an equality match skip events known not to contain the indexed value.
5 questions test this
- A developer is optimizing a Lambda function that processes high-volume transactions. The developer implements structured JSON logging with…
- A company has implemented structured JSON logging across their Lambda functions to improve debugging capabilities. A developer notices that…
- A development team manages a microservices application with multiple Lambda functions writing logs to CloudWatch Logs. They frequently…
- A company runs Lambda functions that process financial transactions. Developers need to debug performance issues across 500 log groups…
- A company has a high-volume microservices application generating terabytes of logs daily across multiple CloudWatch Logs log groups.…
- Build Logs Insights trend queries with stats and bin, and match timeouts on 'Task timed out'
To turn logs into a time series, use stats with aggregations grouped by bin(5m) (for example
stats count(*) by endpoint, bin(5m)); bin() creates the time buckets a chart needs. Filter HTTP errors by numeric range (filter status>=400 and status<=499for 4XX). Lambda timeouts do NOT log as ERROR (they emit a 'Task timed out' message) so capturing all failures needs a pattern likefilter @message like /ERROR/ or @message like /Task timed out/.Trap Filtering only on /ERROR/ to count all Lambda failures, when a timeout emits a 'Task timed out' message rather than ERROR, so it slips past that filter unless you match it explicitly.
6 questions test this
- A developer is troubleshooting API Gateway REST API errors and needs to find all 4XX client errors from the past hour. The developer has…
- A developer is debugging a production issue where some API requests are failing intermittently. The application uses structured JSON…
- A developer needs to create a CloudWatch Logs Insights query to analyze API Gateway access logs and identify latency trends. The query…
- A company's API Gateway REST API is experiencing high 4XXError rates according to CloudWatch metrics. The development team suspects that…
- A developer is troubleshooting intermittent failures in an AWS Lambda function that processes batch data. Some invocations are failing but…
- A developer is troubleshooting intermittent timeout errors in an AWS Lambda function that processes orders from an Amazon SQS queue. The…
- Use VPC Reachability Analyzer to find the blocking component without sending traffic
VPC Reachability Analyzer performs static configuration analysis between a source and destination (it sends no packets) making it the tool to proactively prove a path or pinpoint what blocks it (security group, network ACL, or route table) before any traffic flows. It reports the blocking component with explanation codes. Because it snapshots configuration at analysis time, after you change a rule you must run a NEW analysis; re-reading the old result still shows the stale, pre-change state.
Trap Expecting an existing Reachability Analyzer result to update after you fix a security group: it is a point-in-time snapshot, so you must analyze the path again.
6 questions test this
- A development team is using VPC Reachability Analyzer to troubleshoot why an Amazon EC2 instance in a private subnet cannot connect to a…
- A development team deployed a web application on Amazon EC2 instances behind an Application Load Balancer. After updating security group…
- A developer is troubleshooting connectivity issues between two Amazon EC2 instances in the same VPC. The developer has enabled VPC Flow…
- A developer needs to verify that a web application running on an Amazon EC2 instance is reachable from the internet through an Application…
- A developer runs VPC Reachability Analyzer to test connectivity from an internet gateway to an EC2 instance on port 443. The analysis shows…
- A developer's application running on EC2 instances cannot connect to an Amazon RDS database in a private subnet. The developer needs to…
- A VPC Flow Log REJECT means a security group or NACL denied the traffic
An action=REJECT record means a security group or network ACL blocked the traffic; check the security group first since it is evaluated first and is the most common cause. Because NACLs are stateless, return traffic is evaluated independently and needs explicit ephemeral-port rules. This is why one direction can show ACCEPT while the reply shows REJECT, or why an SG that allows a port still fails when the subnet's NACL does not. Flow logs only appear once there is real traffic on the interface (the log group is created on first traffic), and custom formats can add fields like instance-id, subnet-id, flow-direction, and tcp-flags.
Trap Blaming the security group for a REJECT on return traffic when the outbound request showed ACCEPT, when NACLs are stateless, so the reply needs its own explicit ephemeral-port rule that the SG, being stateful, does not require.
6 questions test this
- A developer is analyzing VPC Flow Logs published to Amazon S3 to understand traffic patterns for an EC2 instance. The developer needs to…
- A developer created a VPC Flow Log for an EC2 instance to troubleshoot connectivity issues. The flow log shows status as Active in the…
- A developer is troubleshooting why an EC2 instance in a public subnet cannot be accessed via SSH from the internet. The instance has a…
- A developer is analyzing VPC Flow Logs to troubleshoot intermittent connectivity issues between an Amazon EC2 instance and an on-premises…
- A developer is troubleshooting an issue where an Amazon EC2 instance in a private subnet cannot connect to an Amazon RDS database in…
- A developer is troubleshooting an application that fails to connect to an external service endpoint from an EC2 instance in a private…
- API Gateway logging needs an account CloudWatch role, and some client errors are never logged
Before any API Gateway logs appear, the account must have an IAM role (trusting apigateway.amazonaws.com, with AmazonAPIGatewayPushToCloudWatchLogs) configured in API Gateway Settings; without it, requests succeed but produce no logs. Even with logging on, API Gateway does not write certain client errors to execution logs (such as 403 from a wrong resource path, and excessive 429 or 413 errors), so enable custom access logging to capture them. To surface integration failures behind 500 errors, add $context.integrationErrorMessage to the access log format.
Trap Assuming execution logging captures every failed request, then concluding a missing 403 means the call never reached API Gateway, when certain client errors like a wrong-path 403 are absent from execution logs unless you turn on custom access logging.
4 questions test this
- A developer is setting up CloudWatch monitoring for an Amazon API Gateway REST API but notices that no logs appear in CloudWatch Logs after…
- A developer is investigating intermittent 500 Internal Server Error responses from a REST API in Amazon API Gateway that uses a Lambda…
- A developer is troubleshooting 403 errors returned to clients from an Amazon API Gateway REST API. The CloudWatch execution logs for the…
- A developer notices that their Amazon API Gateway HTTP API is not generating CloudWatch metrics or logs for some failed requests. The…
- Read RDS ReplicaLag carefully: -1 means replication is broken, and an undersized replica lags
A ReplicaLag value of -1 does not mean zero lag: it means replication is not active or the lag can't be determined (for MySQL/MariaDB, equivalent to Seconds_Behind_Master = NULL), so check the replication state and event log for errors. Sustained positive lag often comes from a read replica with a smaller instance class than the primary: a replica replays the primary's full write load and must match or exceed its compute/storage to keep up. For consistent reads, the app can check ReplicaLag and route time-sensitive queries only to replicas below an acceptable threshold.
Trap Reading a ReplicaLag of -1 as zero lag and a perfectly healthy replica, when the value actually signals replication is not active or cannot be measured, so the replica may be receiving nothing.
4 questions test this
- A developer is troubleshooting performance issues with an Amazon RDS for MySQL read replica. The application reports stale data when…
- A developer manages an Amazon RDS for MySQL database that is experiencing slow response times during peak hours. CloudWatch metrics show…
- A company runs an Amazon RDS for MySQL database with multiple read replicas to handle read-heavy workloads. The development team notices…
- A developer manages an Amazon RDS for MySQL database with three read replicas in the same AWS Region. The application sends read traffic to…
Also tested in
References
- Amazon CloudWatch: Metrics concepts (namespaces, metrics, dimensions)
- Analyzing log data with CloudWatch Logs Insights
- CloudWatch Logs quotas (query timeout and limits)
- Creating metrics from log events using metric filters
- Using CloudWatch metrics with Lambda (Errors, Throttles, Duration)
- AWS X-Ray concepts (segments, subsegments, service map, annotations vs. metadata)
- AWS X-Ray: Configuring sampling rules (default reservoir + 5% rate)
- What Is AWS CloudTrail? (management-event API audit trail)