Define, instrument and measure Success Metrics for the logins component

[rfk]

We want to understand whether users are achieving their goals around login management in the wild. Let's identify a small number of core success metrics that will help us understand that, figure out how to measure them, and then get them on a dashboard somewhere for review.

How is this different from the "Quality Metrics" requested in #1748? The "Success Metrics" here should be more about whether users are getting value out of the Logins data type in our consuming apps. Are they saving and retrieving logins? Are they using them across devices? Things like that.

We may find that we can't track this effectively at the component level, or that it's better left to each individual consuming application. But let's give it a try.

We will probably need to work closely with the Lockwise team on this one...

┆Issue is synchronized with this Jira Story
┆Story Points: 0
┆Epic: Logins Component 1.0
┆Sprint: SYNC - end 2019-11-22

[rfk]

I figure the metrics we care about here will be almost entirely a subset of the metrics that Lockwise cares about from an app perspective. One place we could start here, is looking at the telemetry events from Lockwise for inspiration.

@eliserichards is there a top one or two metrics that Lockwise looks at to tell whether the application is being used successfully?

Most of the things I can think of here seem like vanity metrics, e.g. "number of passwords stored" or similar.

[eliserichards]

We aren't doing much analysis of the metrics that we have yet. Things I would like to look at would be:

• the number of autofill requests coming from each user per day/week/month
• the number of times a user opens Lockwise per day/week/month (e.g. to find/copy a password)

I'm not sure how hard it would be with our current metrics to match our telemetry events to a single user (guessing the UID work that landed recently would help with this).

[eliserichards]

Sandy's Personal Data Protection Weekly Times doc has some rough data in it:

Personal Data Protection Weekly Times: Lockwise & Monitor - Issue #12
...
The Data (Measure)

• Lockwise iOS # of credentials accessed: Last week we saw 35,433 credentials accessed on iOS; you can view the trend over time in our iOS credentials accessed dashboard.
• Lockwise Android # of credentials accessed: Last week we saw 68,422 credentials accessed on Android. You can view the trend over time in our Android credentials accessed dashboard.

[rfk]

Some more graphs sourced from the Data Protection Weekly Times:

• Count of Clients Taking Action (Desktop)
• "PWMGR 2019" Top-level Dashboard, including things like:
  • saved passwords per client per week over time
  • % of clients with at least N logins saved

I'm quite liking the combination of three core usage metrics for a first attempt at this:

• count of saved logins
• distribution of timeLastUsed of saved logins
• number of reads of saved logins per time period
• number of writes of saved logins per time period

These seem to line up OK with metrics we already have on Desktop, which would make for a nice comparison. Next step here should be to try to make a "Logins Success Dashboard" from the data submitted by desktop, perhaps based on some of the queries linked above. That will give us a reference point for what we want the logins component dashboard to look like.

[rfk]

There are two different kinds of metric here, which might affect the details of how we gather them.

The read/write metrics are "event based" which I think we understand how to track in an abstract sense, by calling some increment function. But count and timeLastUsed are "snapshot" metrics that we would need to probe for explicitly. AFAICT desktop achieves this by listening for a gather-telemetry notification, so perhaps we'll need to expose a gather_telemetry function of some sort that the application can call periodically.

[rfk]

AFAICT desktop achieves this by listening for a gather-telemetry notification,

Looks like this mechanism is considered legacy and unreliable, ref https://bugzilla.mozilla.org/show_bug.cgi?id=1546443

[linabutler]

Closing this bug because we've split it up into multiple sub-issues.