There’s a gap between “code works on my machine” and “code works in production.” Monitoring is supposed to bridge it. Most of the time it doesn’t—because it’s built as an ops tool rather than something developers reach for when things break.
What Bad Monitoring Looks Like#
It’s not the absence of tools. It’s:
- Blind debugging: A developer spends three hours on a bug that would’ve taken five minutes with access to production logs
- Customers finding your bugs: You learn about failures from support tickets, not alerts
- Alert fatigue: 200 alerts a day, most irrelevant, so everyone ignores all of them
- Slow incident response: Something’s broken, and the first 30 minutes are spent figuring out what instead of fixing it
The pattern is the same every time—monitoring exists, but it doesn’t connect to how developers actually work.
What You Actually Need#
Metrics, Logs, and Traces—Connected#
- Metrics tell you something is wrong (error rate spiked, latency jumped)
- Logs tell you what happened (stack traces, request payloads, error messages)
- Traces tell you where it happened (which service, which endpoint, which database call)
These are only useful when they’re linked. When an alert fires, you should be able to click through from the metric to the relevant logs to the trace. If developers have to manually correlate timestamps across three different tools, the setup isn’t working.
Dashboards Developers Will Open#
Most monitoring dashboards show CPU usage, memory, and disk I/O. These matter for capacity planning but don’t help debug a 500 error.
Build dashboards that show:
- Business metrics next to technical metrics (signups/hour alongside error rate)
- Recent deployments on a timeline overlay
- The top 5 errors in the last hour, with links to logs
- Latency percentiles (p50, p95, p99), not just averages
If developers don’t open the dashboard on their own, it’s not useful enough.
Alerts That Tell You What to Do#
Every alert should answer two questions:
- What’s broken?
- Where do I start looking?
Bad alert: “High CPU on web-server-3”
Good alert: “Error rate > 5% on /api/payments since 14:32. Last deploy: 14:15 by @sarah. [View logs] [View trace]”
Other things that help:
- Baselines over thresholds: Alert when behavior deviates from normal, not when it crosses an arbitrary number
- Routing by ownership: Payment errors go to the payments team, not the whole org
- Grouping: One Slack message for a cluster of related errors, not 50 individual alerts
Fast Feedback#
The time between “something broke” and “a developer knows about it” should be under a minute. That means:
- Real-time log streaming, not batch processing every 5 minutes
- Deployment markers on dashboards so you can spot if a release caused the issue
- Distributed tracing that works across service boundaries
Build Monitoring for Developers, Not Ops#
The biggest mistake DevOps teams make: building monitoring for themselves.
Talk to Your Developers#
Before picking tools:
- Sit with developers during a debugging session. Watch what they do, what they search for, where they get stuck.
- Ask what questions they have during an incident. “Which service?” “What changed?” “What does the request look like?”
- Find out what metrics matter for the product. Not CPU—things like checkout completion rate, search latency, file upload success rate.
Remove Friction#
If instrumenting a new service takes a day of work, developers won’t do it. Provide:
- Libraries that auto-instrument common frameworks (Django, Express, Spring)
- Copy-paste templates for dashboards and alerts
- Self-service tools so developers can add their own metrics without filing a ticket
Keep It Maintainable#
- Store monitoring config in version control
- Test your alert rules—verify alerts fire when they should
- Plan for multi-region from the start if you’re headed there
Choosing Tools by Team Size#
Small Teams on AWS: Start with CloudWatch#
If you’re under 50 engineers and running on AWS, CloudWatch is the right starting point.
Zero-config infrastructure metrics: EC2, Lambda, RDS, ECS all report to CloudWatch automatically. You get visibility without writing instrumentation code.
Low cost: $10-50/month for small teams. No fixed platform fees.
Everything in one place: Metrics, logs, traces (via X-Ray), and alarms. Less tool sprawl.
Fast to set up: Meaningful alerts and dashboards in hours, not weeks.
Making CloudWatch Work Well#
- CloudWatch Logs Insights is underrated. You can query logs without setting up ElasticSearch:
fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)Composite alarms reduce noise. Alert when error rate is high AND response time is degraded, not on each condition separately.
Custom metrics are where the value is. Track business metrics (signups, transactions, feature usage) alongside infrastructure metrics using the CloudWatch SDK.
CloudWatch Synthetics runs canary tests that simulate user journeys on a schedule. You find out critical paths are broken before users do.
X-Ray integration gives you distributed tracing with minimal setup. Good enough for most microservice architectures.
When to Move On#
CloudWatch starts showing its limits when:
- You go multi-cloud
- You need advanced anomaly detection
- Dashboard customization gets painful
- You have 50+ engineers and need better collaboration features
Larger Teams: DataDog#
DataDog is expensive ($20-100K+/year for large orgs), but it earns its keep at scale.
Cross-platform: Monitors AWS, Azure, GCP, on-prem, containers, serverless, databases, and frontend—all in one view.
Automatic anomaly detection: Watchdog flags unusual patterns without manual threshold configuration. You can’t manually watch thousands of services.
Team collaboration: Team-specific dashboards, RBAC, shared investigation notebooks, PagerDuty/Opsgenie integration.
Advanced alerting: Multi-condition alerts, forecasting (predict when you’ll breach a threshold), anomaly detection that adapts to traffic patterns, maintenance windows.
Deep APM: Code-level profiling, security monitoring tied to traces, cost attribution by service, auto-generated service maps.
Rolling Out DataDog#
Start small: Instrument critical services first. Use tagging to organize by team and environment.
Set standards early: Naming conventions for metrics, dashboard templates, alert severity levels, SLO definitions. This saves pain later.
Integrate with everything: CI/CD deployment markers, incident management, Slack notifications, ITSM ticketing.
Train your teams: Internal docs, team champions, workshops on APM and profiling. DataDog is powerful but has a learning curve.
Watch costs: Filter out noisy metrics, sample traces in high-volume services, audit feature usage regularly, tag everything for cost allocation.
Alternatives#
- New Relic: Similar to DataDog, sometimes cheaper for high-volume tracing
- Dynatrace: Strong AI/AIOps, popular in financial services
- Splunk: Best-in-class log analysis, especially if you already use it for security
- Grafana Cloud: Open-source friendly, great if you’re already on Prometheus/Loki
Hybrid Approach#
Many teams combine tools:
- CloudWatch for AWS-native services (automatic, cheap)
- DataDog for applications and cross-platform visibility
- DataDog ingests CloudWatch metrics for a unified view
This is often the most practical setup.
Common Pitfalls#
Tool overload: Three tools that work together beat six that don’t. Don’t adopt every monitoring product.
Metrics without context: A graph showing “requests per second” means nothing without baselines. Is 500 rps normal or a 10x spike?
Nobody uses it: If developers don’t open the dashboards, the monitoring isn’t working. Make it part of the workflow, not a separate system.
Monitoring the wrong things: Track what matters for users and business outcomes. “Disk I/O on a stateless container” probably doesn’t.
No monitoring for your monitoring: If your alerting system goes down during an incident, you’re blind when it matters most. Build in redundancy.
Getting Started#
If you’re setting up monitoring from scratch or fixing a broken setup:
- Talk to developers first. Find out what they struggle with during incidents.
- Define SLOs. What does “working” mean for your most important user flows? Set targets.
- Instrument the critical path. Start with your most important user journeys—login, checkout, search, whatever drives your business.
- Add tracing. Distributed tracing gives you the biggest debugging ROI in microservice architectures.
- Write runbooks. Every alert should link to a doc explaining what to check and how to fix common causes.
- Review quarterly. Remove stale alerts, update thresholds, check that dashboards still reflect the current architecture.
Good monitoring isn’t about having the fanciest tools. It’s about giving developers the information they need to fix problems fast.