Success is not just about building a product. It’s about defining what success means and measuring the right signals aligned to business goals.
Measuring product success is not guesswork. The actual job is to start with business goals and then translate those into measurable indicators across the customer journey. If you cannot define what success looks like in your context, you cannot measure it.
This lesson uses Amazon Echo as a case study to show how to break down success into business objectives, customer actions, and metrics. The approach you learn here applies broadly—whether you are working on hardware, SaaS, or digital platforms.
The trap of vague metrics
I have watched thousands of PMs stumble on success measurement because they jump straight to metrics without defining the business goals first. They list traffic numbers, app installs, or feature usage without tying it back to why the business exists.
Here is the uncomfortable reality: Metrics without context are noise. If you cannot answer "What business outcome does this metric move?" you are not ready to claim success.
Start with business goals
Any measurement system begins with business goals. For Amazon Echo, there are two broad customer segments with distinct goals:
- Consumer households: Buy and use Echo devices to improve their daily life and consume Amazon services.
- Alexa Skills developers: Build and monetize voice apps that increase device utility and ecosystem value.
Amazon’s primary goal is to grow revenue from consumer purchases through the Echo device. The secondary goal is to expand the Alexa Skills ecosystem, driving revenue for AWS through developer activity and skill monetization.
These business goals set the direction for all success metrics.
Map the customer journey to success metrics
Once business goals are clear, map the customer journey stages to the metrics that reflect progress toward those goals.
The customer journey stages for Amazon Echo roughly are:
- Awareness
- Acquisition (purchase)
- Engagement (usage)
- Retention (continued usage)
- Monetization (revenue generation)
Each stage has distinct behaviors that indicate success or failure.
Awareness
Awareness is about making potential customers curious and informed about Echo. Success here is measured by increased interest and research activity.
Key metrics:
- Website visits to Echo product pages
- Keyword search volume for "Amazon Echo"
- Social buzz and word-of-mouth mentions
- Click-through rates on marketing campaigns
These metrics show whether marketing efforts are effectively driving visibility.
Acquisition (Purchase)
Acquisition means customers buy the Echo device. Here, segmentation matters: Prime vs. Non-Prime customers behave differently.
Key metrics:
- Number of Echo units sold monthly
- Revenue generated from Echo sales
- Conversion rate of Non-Prime to Prime subscribers post-purchase
Amazon’s business model encourages Prime conversion because Prime users have higher lifetime value.
Engagement
Engagement reflects how customers use the device and its features. For Echo, engagement includes:
- Frequency of purchases through voice commands
- Number of Alexa Skills used per customer
- Subscriptions to Amazon Music Unlimited via Echo
Key metrics:
- Monthly active users (MAU) of Alexa Skills
- Top skills by usage frequency and revenue
- Number of purchases initiated through Echo vs. website sessions
Engagement metrics measure whether the device is driving the intended customer behaviors.
Retention
Retention tracks whether customers keep using Echo over time, a critical factor for sustainable revenue.
Key metrics:
- Daily active users (DAU) and monthly retention rates
- Average days between purchases via Echo
- Skill usage frequency over time
- Device return rates and trends
Retention indicates long-term product health and satisfaction.
Monetization
Monetization converts engagement into revenue. Amazon’s monetization levers include:
- Additional purchases made via Echo
- Revenue from Alexa Skills transactions (developer revenue share)
- Amazon Music Unlimited subscriptions via Echo
- New Amazon Prime subscriptions triggered by Echo ownership
Key metrics:
- Incremental revenue attributable to Echo voice purchases
- Revenue share from developer-hosted Skills on AWS
- Subscription growth rates for Amazon Music Unlimited and Prime
These metrics tie product usage back to financial outcomes.
The developer ecosystem goal
Amazon also needs to measure success in growing the Alexa Skills developer ecosystem. This supports the consumer business by increasing device utility.
Key metrics:
- Number of active developers building Skills
- Monthly growth rate of Skills published
- Average number of Skills per developer
- Revenue generated from AWS hosting and compute services for developers
Measuring developer engagement ensures the platform remains vibrant and competitive.
Cohort analysis for deeper insights
To understand how usage and revenue evolve, segment users into cohorts based on when they purchased Echo. Track changes in behavior as cohorts age.
For example, compare purchases, skill usage, and retention rates of users who bought Echo in January vs. June.
Why cohort analysis? It reveals whether new features or campaigns improve long-term engagement and monetization.
The structured approach to success measurement
The structured way is:
- Define business goals clearly.
- Break down the customer journey into stages.
- Identify behaviors and actions that indicate success at each stage.
- Choose metrics that measure those behaviors.
- Use cohorts to track changes over time.
- Prioritize metrics that tie directly to revenue and growth.
MeetingScene: Product strategy review at an Indian SaaS startup
Strategy meeting at a Series B SaaS startup in Bangalore
CEO: “We launched a new feature last quarter. How do we know if it’s successful?”
You (PM): “First, let’s revisit the business goals the feature supports. Is it acquisition, engagement, or revenue growth?”
CEO: “Mainly engagement and retention.”
You (PM): “Then we track metrics like monthly active users for the feature, frequency of use, and churn rates before and after launch. We should also segment users by how long they’ve used the product.”
VP Engineering: “Can we instrument those metrics in the next sprint?”
You (PM): “Yes, and we’ll set targets so the team knows what success looks like.”
Clear alignment on goals and metrics helped avoid the trap of vanity metrics.
The team risks celebrating usage spikes without understanding business impact.
SlackChat: Discussing success metrics for a new voice assistant feature
FieldExercise: Define success metrics for your product
- Write down your product’s primary business goals (e.g., revenue growth, user acquisition, retention).
- Break down your customer journey into 3-5 stages relevant to your product.
- For each stage, list 2-3 customer actions that indicate success.
- Identify metrics you can track for each action.
- Prioritize metrics that directly impact your business goals.
- Share your list with a peer or mentor for feedback.
JudgmentExercise
You are a PM at a Series C Indian consumer electronics startup launching a smart speaker similar to Amazon Echo. The CEO expects a monthly report on product success metrics. Your engineering team has limited capacity for instrumentation.
The call: Which metrics do you prioritize to measure success, and how do you justify your choices to leadership?
Your reasoning:
PracticeExercise
You are a PM at a Series C Indian consumer electronics startup launching a smart speaker similar to Amazon Echo. The CEO expects a monthly report on product success metrics. Your engineering team has limited capacity for instrumentation.
Your task: Which metrics do you prioritize to measure success, and how do you justify your choices to leadership?
your reasoning:
FromTheField: Reflections on success metrics from Pragmatic Leaders training
Test yourself: The success metrics prioritization
You are the PM for a new smart home device launching in Mumbai. The CEO wants a success report after 3 months. Your engineering team can only instrument 5 metrics initially.
Which metrics do you instrument first?
Where to go next
- Build a product vision that drives metric selection: Product Vision and Strategy
- Learn to conduct effective user research: User Research Methods
- Master data-driven decision making: Metrics and KPIs
- Understand product-market fit dynamics: Product-Market Fit