Skip to content

Add analytics sampling - #98

Merged
alecgeatches merged 1 commit into
planned-release/1.0.11from
add/analytics-sampling
Nov 7, 2024
Merged

Add analytics sampling#98
alecgeatches merged 1 commit into
planned-release/1.0.11from
add/analytics-sampling

Conversation

@alecgeatches

Copy link
Copy Markdown
Contributor

Description

See #96. Load analytics are recorded for VIP sites on each governance configuration load, which happens when the WordPress editor is opened. This amount of data isn't necessary and can have a small effect on editor load time.

This PR adds time-based sampling to usage analytics to reduce the number of analytics calls by 90%.

Steps to Test

  1. Debug the plugin in a local environment. Use either a breakpoint or logging in the Analytics::record_usage() call to determine when analytics would normally fire.
  2. Open any existing post in the admin interface, or a new post through Admin → Posts → Add New Post.
  3. Refresh the page a few times. Analytics should only fire 10% of requests, when ( $epoch_time % 10 ) === 0.

@alecgeatches alecgeatches self-assigned this Nov 4, 2024
@alecgeatches
alecgeatches requested a review from a team as a code owner November 4, 2024 18:20
@alecgeatches

Copy link
Copy Markdown
Contributor Author

@smithjw1 As part of the work in #97 to update the latest supported WordPress version, it'd be great if we could also include analytics sampling as you suggested in #96. This adds a super simple sampling technique based on the current time that should remove 90% of our current analytics calls, while still leaving a "heartbeat" of usage metrics during sampling seconds. Let me know what you think!

$epoch_time = floor( microtime( true ) );

if ( ( $epoch_time % 10 ) === 0 ) {
// Sample results. Only send analytics on 10% of configuration loads.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So microtime returns seconds, so this will send when the editor is loaded and it happens to be 0, 10, 20, etc.

I'm a little more concerned about this undercounting than the Block Data API, which when in high use is called almost constantly.

But we can check it out. The median number of MC stat hits for October has 348 so we would expect it to be 35 going forward, that still leaves enough room to notice changes.

But we should test that assumption after launch to see if the stats drop by more than 90 percent.

@alecgeatches alecgeatches Nov 7, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, Jacob! You're right this this is a little more risky, but I think we're still covered statistically if the editor is being used more than a handful of times per day. From my basic math, if an editor is opened 20 times in a day, we have an 88% chance of registering a data point (1-((1-.1)^20)). If it goes up to 50 times (1-((1-.1)^50)), we're 99% likely to catch a point. This will affect sites where the editor is opened infrequently like a test site, but for any "enterprise" level of usage it should be noticeable and logged.

@alecgeatches
alecgeatches merged commit 6c06120 into planned-release/1.0.11 Nov 7, 2024
@alecgeatches
alecgeatches deleted the add/analytics-sampling branch November 7, 2024 17:04
This was referenced Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants