Change #1108821 restored by Jdlrobson:

[mediawiki/extensions/CheckUser@master] CheckUser must declare modules that are loaded on page load

https://gerrit.wikimedia.org/r/1108821

Change #1108821 abandoned by Jdlrobson:

[mediawiki/extensions/CheckUser@master] CheckUser must declare modules that are loaded on page load

Reason:

Okay - this has now been descoped to https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1108842

https://gerrit.wikimedia.org/r/1108821

Change #1108842 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/VisualEditor@master] VisualEditor must declare modules that are loaded on page load

https://gerrit.wikimedia.org/r/1108842

In T355837#10438413, @Michael wrote:

Wait what? That needs to be a huge warning message in all related documentation places, because that is extraordinarily counter-intuitive and not at all obvious!
If I search for microtime( true ) in extensions, I see them mixing tracking seconds and milliseconds all over the place, so I'm not the only one.

Accidental and intentional misuse of timers is a long-standing problem. We attempted to help guide it all in a better direction in PHP StatsLib by introducing helper functions that did the math under the hood.

In T355837#10438413, @Michael wrote:

[…]
If I search for microtime( true ) in extensions, I see them mixing tracking seconds and milliseconds all over the place, so I'm not the only one.

Change #1108824 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/NavigationTiming@master] Track bundlesize of module loaded on article page load

https://gerrit.wikimedia.org/r/1108824

Change #1108820 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/core@master] Tests: Remove hardcoded exclusion list from PerformanceBudgetTest

https://gerrit.wikimedia.org/r/1108820

Change #1108821 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/CheckUser@master] CheckUser must declare modules that are loaded on page load

https://gerrit.wikimedia.org/r/1108821

In T355837#10438043, @Krinkle wrote:

In T355837#10431981, @Michael wrote:

[…]

Thank you so much for the work here and the documentation!

[…] https://wikitech.wikimedia.org/wiki/Performance.wikimedia.org/Runbook#How_it_works […] explicitly says to track milliseconds for Graphite. But if I understand the upstream best-practices correctly, then we _should_ use seconds with Prometheus, correct?

The short answer is: Yes, name it _seconds but remember to pass milliseconds as value, same as we did before the migration. This applies to MediaWiki PHP and MediaWiki JS equally.

The longer answer is:

• Graphite is being replaced with Prometheus. MediaWiki PHP and JS emit StatsD-formatted messages, same as before.
• At WMF, we ingested these into statsite which translated them to Graphite data points once a minute (ref wikitech:Statsd, wikitech:Graphite#Extended_properties).
• For the "new" style stats, WMF ingests these into prometheus/statsd_exporter, which translates them to Prometheus.
• For the "new" style stats, we follow Prometheus naming conventions instead of Graphite naming conventions, which mostly boils down to: no dots in names, include unit suffix in names (_total, _seconds), and permit optional use of dogstatsd "tags" on statistics that are translateable to Prometheus labels.

The trade-off here (between consistency, transparent naming, and developer familiarity/compatibility), for both MediaWiki PHP and MediaWiki JS, is to have timers named as mediawiki_foo_bar_seconds, but with values that are milliseconds, since that's what the statsd protocol requires. This is convenient in JavaScript, where performance.now() uses milliseconds, and our existing stats in MW extension JS all use milliseconds today. Once consumed by prometheus/statsd_exporter, these are converted to seconds, and that's how they'll come out in Grafana.

In T355837#10431981, @Michael wrote:

[…]

Thank you so much for the work here and the documentation!

[…] https://wikitech.wikimedia.org/wiki/Performance.wikimedia.org/Runbook#How_it_works […] explicitly says to track milliseconds for Graphite. But if I understand the upstream best-practices correctly, then we _should_ use seconds with Prometheus, correct?

In T355837#10417760, @Krinkle wrote:

A few updates:

• The infra is now ready for use and available on all wikis as of today (per today's group2 deployment of the 1.44.0-wmf.8 train).
• I've converted mw:Stats, from a soft-redirect to Wikistats, into a disambiguation page that includes StatsLib (PHP) and mw.track stats (JS): https://www.mediawiki.org/wiki/Stats
• Wrote usage docs and example at https://www.mediawiki.org/wiki/ResourceLoader/Core_modules#mw.track and https://wikitech.wikimedia.org/wiki/Statsv. Cross-linked from https://www.mediawiki.org/wiki/Manual:Stats#mw.track.
• Filed T382549: (mw.track) Migrate MediaWiki.edit_recovery.* to Prometheus with a task description describing the current state of affairs, JS-specific instructions, and relevant documentation.
• Propagate new instructions for mw.track, to replace the non-applicable PHP StatsLib instructions, to task descriptions of T359341: (mw.track) Migrate MediaWiki.AdvancedSearch.event.* to statslib, T359237: (mw.track) Migrate MediaWiki.cx.campaign.*.accept to statslib, T359467: (mw.track) Migrate MediaWiki.EntitySchema to statslib, T359347: (mw.track) Migrate MediaWiki.echo.* to statslib, T359251: [REPO][SW][GRAFMIGR] (mw.track) Migrate MediaWiki.wikibase.repo.* to statslib.
• Re-opened T359244: (mw.track) Migrate MediaWiki.RevisionSlider.* to statslib and T359261: mw.track() Migrate MediaWiki.TwoColConflict.* to statslib, and update their task descriptions, as their PHP portions were completed but not their JS portion.
• Added missing Phab tags to each of these tasks so that they can be discovered and triaged by relevant teams.

Example of first migrant: T382549: (mw.track) Migrate MediaWiki.edit_recovery.* to Prometheus.

A few updates:

Mentioned in SAL (#wikimedia-operations) [2024-12-19T21:53:04Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1105735|Make AuthManagerAutoConfig configuration key more distinctive (T369180)]], [[gerrit:1105739|SUL3: Disable more auth providers in the local leg of SUL3 login (T369180)]], [[gerrit:1105742|[noop] Update private/readme.php (T369180)]], [[gerrit:1105367|Enable $wgWMEStatsBeaconUri (T355837)]] (duration: 21m 34s)

Mentioned in SAL (#wikimedia-operations) [2024-12-19T21:37:35Z] <tgr@deploy2002> krinkle, tgr: Backport for [[gerrit:1105735|Make AuthManagerAutoConfig configuration key more distinctive (T369180)]], [[gerrit:1105739|SUL3: Disable more auth providers in the local leg of SUL3 login (T369180)]], [[gerrit:1105742|[noop] Update private/readme.php (T369180)]], [[gerrit:1105367|Enable $wgWMEStatsBeaconUri (T355837)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-12-19T21:31:30Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1105735|Make AuthManagerAutoConfig configuration key more distinctive (T369180)]], [[gerrit:1105739|SUL3: Disable more auth providers in the local leg of SUL3 login (T369180)]], [[gerrit:1105742|[noop] Update private/readme.php (T369180)]], [[gerrit:1105367|Enable $wgWMEStatsBeaconUri (T355837)]]

Change #1105367 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable $wgWMEStatsBeaconUri

https://gerrit.wikimedia.org/r/1105367

Change #1105454 merged by jenkins-bot:

[performance/statsv@master] Further limit regex to only counter type flag

https://gerrit.wikimedia.org/r/1105454

If the use case to join with data lake has a high priority, I suggest pulling data from Prometheus rather than intercepting it on the way to Prometheus.

Now on to some technical comments / clarifications :)

Change #1105372 merged by Cwhite:

[operations/puppet@production] webperf: Enable --dogstatsd on statsv.py

https://gerrit.wikimedia.org/r/1105372

Thank you so much Timo for the details!

In T355837#10147260, @Ottomata wrote:

Oh, it's POST? With a POST body? or is the data just urlencoded query params?

If we can use POST body for the data, then my proposed new pipeline would be: […]

Change #1105423 merged by jenkins-bot:

[performance/statsv@master] update kafka-python version to match webperf hosts

https://gerrit.wikimedia.org/r/1105423

Change #1105454 had a related patch set uploaded (by Cwhite; author: Cwhite):

[performance/statsv@master] Further limit regex to only counter type flag

https://gerrit.wikimedia.org/r/1105454

Change #1098570 merged by jenkins-bot:

[performance/statsv@master] statsv: Add dogstatsd/Prometheus support

https://gerrit.wikimedia.org/r/1098570

Change #1105423 had a related patch set uploaded (by Cwhite; author: Cwhite):

[performance/statsv@master] update kafka-python version to match webperf hosts

https://gerrit.wikimedia.org/r/1105423

Change #1105372 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] webperf: Enable --dogstatsd on statsv.py

https://gerrit.wikimedia.org/r/1105372

Change #1105367 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] Enable $wgWMEStatsBeaconUri

https://gerrit.wikimedia.org/r/1105367

Change #1105074 merged by jenkins-bot:

[mediawiki/extensions/Math@REL1_43] Fix for Math Popup not working in newer version of Popup-Extension

https://gerrit.wikimedia.org/r/1105074

Change #1105074 had a related patch set uploaded (by Krinkle; author: FrederikHennecke1):

[mediawiki/extensions/Math@REL1_43] Fix for Math Popup not working in newer version of Popup-Extension

https://gerrit.wikimedia.org/r/1105074

Change #1098570 had a related patch set uploaded (by Krinkle; author: Krinkle):

[performance/statsv@master] statsv: Add dogstatsd/Prometheus support

https://gerrit.wikimedia.org/r/1098570

Change #1099821 merged by Cwhite:

[operations/puppet@production] webperf: set statsd exporter timer type to histogram

https://gerrit.wikimedia.org/r/1099821

Change #1099796 merged by Cwhite:

[operations/puppet@production] webperf: disable statsd-exporter relaying flag

https://gerrit.wikimedia.org/r/1099796

Change #1099720 merged by Cwhite:

[operations/puppet@production] webperf: set statsv.py --statsd to statsd.eqiad.wmnet

https://gerrit.wikimedia.org/r/1099720

Change #1099822 merged by Cwhite:

[operations/puppet@production] prometheus: restart statsd-exporter on config change

https://gerrit.wikimedia.org/r/1099822

Change #1098521 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] statsd: Create a `stats` mw.track topic for dogstatsd/Prometheus

https://gerrit.wikimedia.org/r/1098521

Change #1098517 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Make use of built-in packageFiles `config` feature

https://gerrit.wikimedia.org/r/1098517