What are the KPIs for GSM

Atlassian uses the framework of goals, signals and metrics to develop new features

on October 28, 2019

The “Goal, Signals, Metrics Framework” is about defining success from the perspective of the user. Two product managers from Atlassian explain how they used the model for new features in the popular Jira and Confluence tools.

  • In this text you will learn
  • how to focus on the success of the user.
  • how to define metrics that measure the right thing.
  • how you can use signals to recognize whether you are on the right track.
  • what distinguishes goals, signals, metrics from OKRs and how they complement each other.

A common problem in product development is that teams tend to define the success of their product in terms of the amount of newly developed features. It is all too easy to forget that the aim is to satisfy the user. The framework from Goals, Signals, Metrics (GSM) helps teams focus on users again. The focus is on three questions:

  1. What does success look like for our users?
  2. How do we measure this success?
  3. How do we know that our product really supports the user?

A false start with a happy ending

Isha Mehta was very pleased with herself when the new language versions for Confluence came out. One of the goals of the product manager at the software company Atlassian was to internationalize the popular collaboration tool and thus improve the use of Confluence for non-English-speaking users. As Confluence in ten new languages was rolled out, this goal seemed to have been achieved. Then the first reactions of the users came in on Twitter.

Secure your ticket now The technology giant Atlassian, based in Sydney, Australia, builds the standard developer tools Jira and Confluence, among other things. The Trello digital whiteboard is now part of Atlassian, and the company also holds shares in the Slack collaboration platform. Round 150,000 corporate customers use Atlassian software. But even this large number of customers does not protect against losing sight of the user every now and then, as Isha Mehta had to learn the hard way. Because the reactions to the new language versions were anything but positive. “It's such a terrible and bad translation that it hurts,” wrote a user on Twitter after the new Confluence versions were live.

The hard feedback had its good side. It reminded Isha and her colleagues that putting yourself in the user's shoes is more important than simply knocking out a number of new features. To ensure that the language fiasco does not repeat itself, they rearranged their processes and their objectives. Instead of output-oriented KPIs, there was a system of goals, signals and metrics.

The advantages of goals, signals, metrics

In addition to Isha Mehta, Josephine Lee also works intensively with the GSM framework at Atlassian. Josephine is a product manager in the Confluence and Jira Notifications team. Both have now gained a wealth of experience on how to set the right goals and measure whether you are on the right path to achieve them. The advantages of GSM are obvious for both:

  1. It forces you to put the user first instead of thinking about which feature to release next.
  2. You are able to quickly adjust your goals and metrics when you receive new information.
  3. Avoid spending weeks or months optimizing on metrics that are out of reach or don't really measure what you want to measure.

Isha and Jo recommend the following procedure for teams that want to work with GSM.

How to set goals

First of all, the whole team should agree on the mission. This is important in order to even understand what you are working towards. Isha says, “A typical trap that teams fall into is that you define success in terms of metrics related to output.

"We are used to optimizing for typical KPIs such as daily active users, monthly active users, retention or conversion rate."

These metrics do not tell you anything about whether you have actually helped the user with a new feature. And it is also very difficult to prove in data analysis that this one feature really contributed to the change in a metric. Instead one defines which would be a success for the user. Only in the next step do you clarify how you can make this success measurable.

Bringing out ten new language versions for Confluence has proven to be a bad goal for Isha's team. It was based purely on the output. It worked better when they set themselves the goal of enabling non-English-speaking users to use Confluence for successful collaboration. Even if the way to get there is relatively clear, there is enough freedom to try and test different things. Previously, it was just a matter of ticking off language versions from a list. Now the team was asked to think about how to make this goal measurable.

How to define indicators

Now the demanding part begins: It is necessary to determine the indicators by which you can recognize that you are on the right track. These indicators may or may not be quantifiable. It can also be observable changes in behavior. Important: Signals are no guarantee of success. What they are guaranteed to indicate is if you are walking in the completely wrong direction. In other words, if the signals are red, it is almost certain that you have to go back to the whiteboard. But just because the signals are all green doesn't mean you can sit back and relax.

In the case of Confluence, signals for a successful introduction of language versions were that

  1. non-English speakers Switch users to the new language version and
  2. Confluence permanently in the national language to use.

It is important with these signals that they have not yet been provided with an exact KPI. With the signals, the first thing is to develop a feeling for the users and their behavior. Time plays an important role here, says Isha: “After the release of a new feature, we always give ourselves a few weeks to collect data and feedback first. We combine quantitative methods with qualitative user surveys. "

Signals are supposed to show whether what you expected is actually happening. If that's the case: good. Now you can work on achieving the goals set for a successful feature. If, on the other hand, the signals show that users are not behaving as expected, you are on the wrong path and you don't even have to try to optimize for success.

How to set success conditions

You only know whether a feature is actually successful once you have met the tough conditions for success. Here again, it is important to ask yourself what success would be like for the user? In the case of the language versions, the Confluence team wanted to improve the product for non-English speaking users. That means: The users should not only switch, but ideally also stay with the new version permanently. In order for these signals to become clean metrics, they must now operationalized be provided with a number. Isha and her team set two goals:

  1. At least 30 percent of non-English speaking users should use Confluence in the same language in which they use the browser.
  2. At least half of users who switch to a translation should not switch back to English in the next six months.

The key is to set these metrics so that they actually indicate success. So they shouldn't be too easy to get to. At the same time, they shouldn't be completely utopian to avoid frustration. For the Atlassian team, for example, it would have been simply unrealistic to plan to get 100 percent of non-English-speaking users to use the new translation.

In order to arrive at the correct measurement variable, one can either fall back on experience with comparable products or features. If there is no experience, Isha recommends listening to your gut instinct: “To start with, it is enough to simply set a number that you trust. In the course of the process you will notice whether you were really wrong, or actually had a good nose for your users. "

A second way to find an output metric is to do some kind of reverse engineering from within the team, recommends Josephine.

"You can ask the team what value would disappoint them."

Most teams have a very good feel for their users and can assess what can realistically be achieved, but still represent an exciting challenge.

In contrast to signals, reaching a metric signals that a feature or product has actually been successfully launched. That doesn't mean you can stop working completely at that moment. But now you are allowed to take your foot off the gas. A good metric should feel like an important milestone. If that is achieved, you will automatically think about the next steps. And the process basically starts over.

Constant iteration

An important factor in the GSM framework is that nothing is set in stone. The goal you want to achieve should of course be stable (otherwise it would mean starting from scratch again). A.Signals and metrics can change in the course of a process. User feedback plays a major role here. So it can happen that you find that the metric you use to measure the success of the users is not exactly accurate.

For example, Josephine and her team discovered that for Jira email notifications it was not enough to track the open rate to check whether users were satisfied with the feature. Some notifications require a response, such as being mentioned in a comment or assigned a task. In these cases, a message is only successful if the user has actually clicked on and executed the assigned task. During the process, Josephine and her team changed the corresponding metric several times. In the end, they agreed to track whether users actually click on the issue after notifications that initiate an action and stay on the issue for at least 40 seconds. This duration was considered to be an indicator that the users actually accept the task at hand.

But be careful: You can optimize yourself to death, warns Isha: “You have to be careful not to get lost in the metrics. The priority is always to develop the product or feature. "

Be careful with Countermetrics

What you should keep an eye on are so-called countermetrics. These are metrics that you don't want to negatively influence, just to bring another metric forward. It is certainly convenient for users to have fewer email notifications sent to them. But if that leads to users being less active in the product, that is counterproductive. The goal of sending users less irrelevant messages should not lead to a decrease in the number of active users. Here it is important to find the right balance between two metrics.

Delimitation and integration of OKRs

At first glance, the GSM framework seems to be nothing more than a slightly modified variant of Objectives and Key Results. You have a goal, an objective. And uses various metrics, i.e. key results, to check whether this goal is being achieved. OKR professionals also recommend working more output-oriented and focusing on the user and their behavior.

In fact, the two frameworks are very similar. But it is not a question of either / or. Isha and Jo believe that OKRs and GSM complement each other well. Atlassian also has an OKR system. These are checked quarterly and are based on the large, cross-company BHAG, the “Big, Hairy, Audacious Goal”.

Goals, signals, metrics allow the teams to set goals even more closely on the product and project-related, detached from the rhythm of the OKRs. While the OKRs pay into the big picture, goals, signals, metrics flexible to day-to-day business break down. If you imagine OKRs like fixed stars in the sky that you can use for orientation, GSMs are more like comets that you follow through the solar system at high speed.

Support with data science

Another important factor with GSMs in contrast to OKRs is that they are very close to the product and are almost always based on a hypothesis. "If our users do XY, then we assume that they are satisfied with the product." That means: "Whenever possible, we try to involve colleagues from the data science team," says Josephine. Especially when developing the right metrics to measure success, it helps Call in experts for data analysis. Otherwise typical statistical misinterpretations can easily occur, such as equating correlation with causality. The best way to protect yourself against misinterpretations and false correlations remains to be in contact with the user. “Regardless of which metric you measure: you will very quickly notice whether it is the wrong one if you speak to the users early on,” says Jo. Or when they send you angry tweets.

(Cover photo: Atlassian)