Define, instrument and measure Quality Metrics for the logins component

[rfk]

We want to understand whether the logins component is working well for users in the wild. Let's identify a small number of core quality metrics that will help us understand that, figure out how to measure them, and then get them on a dashboard somewhere for review.

This...has the potential to be a pretty big issue, and will almost certainly require collaboration with other teams. Finding out what metrics are currently tracked by our existing consumers seems like a good first step here.

┆Issue is synchronized with this Jira Story
┆Story Points: 0
┆Epic: Logins Component 1.0
┆Sprint: SYNC - end 2019-11-22

[rfk]

A bit of brainstorming of quality metrics which I think have a clear value prop to monitor on an ongoing basis:

• Rate of unexpected errors in the API, from things like disk IO errors, bad input data, etc.
• Sync error rate, both in aggregate (number of sync errors / number of sync attempts) and on a per-user basis (number of unique users with a sync error / number of unique users who tried to sync).
• Performance data on how long important API calls take (such as saving a login, reading all logins), ideally as a function of the number of items in the store.

To resolve this issue, we'll need to first figure out what we want to measure, then figure how whether we should measure it according to our Data Principles, then figure out how to actually measure it in practice using deployed clients like Lockwise and Firefox for iOS.

@eliserichards and @garvankeeley, I'd be interested in your perspective as consumers of the Logins component - what could we measure and report on that would give you confidence in the overall quality of the Logins component?

[garvankeeley]

The areas you mentioned cover the things I would associate with quality for this. For Firefox iOS, performance of Logins is a non-issue (i.e. there is no perf requirement from our end on Logins), but Sync error rate would be a good measure I think.

[eliserichards]

Those three areas are definitely top of my list. We have the sync data already from Lockwise, but I would definitely be interested in seeing the other errors and performance as those would have a significant impact on any new Lockwise users.

[cse-test-sync]

➤ Ana Medinac commented:

10/21: Could defer this to another week.

[linabutler]

Closing this bug because we've split it up into multiple sub-issues.