I think @BTullis' idea is great - there's a few unknowns regarding how to keep what runs via airflow in sync with what runs on k8s via helm charts, but that's something I'd rather invest my time in than writing some shim to run mwscript on a physical host. That was a solution I proposed to reduce effort overall but I think it makes sense for multiple reasons, first of all the operational toil of the DE SRE team, to go with his proposal.

In T377497#10240939, @Urbanecm_WMF wrote:

For example, video2commons (as one of the major sources of Server-side-upload-request) first attempts to upload the file itself. If it receives an error such as backend-fail-internal, it prompts the user to request a server side upload instead (see source for more details). Of course, if we can identify and fix all of those bugs, that would be amazing and far better than figuring out how to continue doing server side uploads.

In T377497#10248513, @Pppery wrote:

That documentation isn't quite accurate. The goal of server-side uploads as they are used today is to work around the fact that uploads of large files are flaky for various reasons, and more likely to flake the larger the file gets.

Most of the user stories for this epic have been implemented. Some were out of scope for this quarter's hypothesis and will be the focus of the new hypothesis for the new quarter.

The basic read-only interface is published, we can solve this task and open a new one for the read-write interface

In T377497#10240939, @Urbanecm_WMF wrote:

In T377497#10240682, @Joe wrote:

[...]
And finally, by far my favourite option:

• Given now uploads by url are async, just raise the file size limit for those to more than 5 GB and stop doing server side uploads, which is an archaic idea that generates toil work for a small and precious slice of our community members

Can you clarify this, please? I do not understand how would raising this limit help to avoid server side uploads.

I should add, this is yet another example of how the unmaintained and substantially abandoned parts of MediaWiki, like the file uploads and manipulation stack, are a constant source of technical debt and slow down every single migration/change we want to make to MediaWiki.

I have thought of a few options for this:

In T377218#10230170, @Volans wrote:

In T368389#10221202, @SCherukuwada wrote:

Questions:

3. How can MW code use this model?

In T376976#10220538, @akosiaris wrote:

We 've already discussed this in a 1on1 and just for transparency's sake, this finds me in agreement.

Conftool2git is now running in production; the git synced repository is cloned across all puppetservers and could in theory be served (minus the requestctl actions) via config-master if anyone wanted to take a look.

In T352650#10179932, @dr0ptp4kt wrote:

@Joe checking - is Q2 FY 24-25 still looking good for Service Ops containerization work? Or is it looking more like Q3? Either way, any guesses on rough range of dates in which this effort might start?

Also, having read all the task, I'd bet the problem is first of all a network instability, as @bd808 has suggested.

Please next time you have a problem with name resolution, run dig with +trace as well, so that we can see what the authdns is actually saying.

In T376140#10191908, @Ladsgroup wrote:

FWIW, lots of people could reset their passwords, that worksTM. I need to figure out what's causing this.

Reason: besides the AI blurb, I don't think this is a reasonable task to open if someone isn't going to do the work themselves, or has the resources to do the work.

I'm not sure what is this extension doing, but http requests to our edge (so, using the public graphite url) will for sure fail, and even if we use the internal url, we'd need to open a firewall rule for it.

Looks like a request to an external source from MediaWiki that doesn't make use of the url-downloader as a proxy, thus it's firewalled.

In T368654#10160566, @Lucas_Werkmeister_WMDE wrote:

In T368654#10156498, @Joe wrote:

Yeah that is slightly surprising

I guess your “slightly surprising” is our “pretty important for site performance” ;) I don’t know if we track the cache hit rate anywhere, but the total request volume to Special:EntityData is in the millions per day. If you’re not using that cache, then I agree you probably want a more restrictive limit on those calls 👍

An update after the meeting with SRE, Security and the wikifunctions people:

In T368654#10143463, @Lucas_Werkmeister_WMDE wrote:

In T368654#10137263, @Joe wrote:

Special pages are not cached at the edge, so there is no caching for that url, independently of indicating a revision or not:

$ curl -Is https://www.wikidata.org/wiki/Special:EntityData/Q42.json  | grep cache-control
cache-control: private, s-maxage=0, max-age=0, must-revalidate
$ curl -Is https://www.wikidata.org/wiki/Special:EntityData/Q42.json?revision=1600533266  | grep cache-control
cache-control: private, s-maxage=0, max-age=0, must-revalidate

Some caching seems to be happening:

$ curl -Is 'https://www.wikidata.org/wiki/Special:EntityData/Q42.json'  | grep x-cache
x-cache: cp3073 miss, cp3073 pass
x-cache-status: pass
$ curl -Is 'https://www.wikidata.org/wiki/Special:EntityData/Q42.json?revision=1600533266'  | grep x-cache
x-cache: cp3073 miss, cp3073 hit/1
x-cache-status: hit-front

Setting to high and not UBN because we don't have this version installed anywhere.

In T374662#10154543, @Krinkle wrote:

When the excimer timeout exception is thrown, MediaWiki replaces the response with an error page. However, the reason for having excimer at this level, is so that we can render a skinned and localised error page, which includes various bits of information from the database (e.g. sitenotice, notification count, etc). It also involves running whatever deferred updates were queued post-send.

As such, it is not impossible for an excimer timeout to happen "at the right time" but also have long execution, if there are badly behaved features queueing code to happen post-send.

In T371782#10114735, @TBurmeister wrote:

Hi @Joe, do you have a screenshot or any mockups of what the web UI for this tool will look like, or a list of all the functionality the tool will implement? I saw in a recent Asana update that the web interface, while still under construction, can already read/update/delete any requestctl object, and that you're planning to implement the ability to create objects and to commit changes. In working on T372095, it would be helpful for me to know:

• Which user actions or workflows documented at https://wikitech.wikimedia.org/wiki/Requestctl may become irrelevant due to being handled behind the scenes by the tool?
• Which user actions or workflows would potentially still require command line examples even after the web application is deployed?
• Are there any new user actions or workflows that will be made possible thanks to the new UI?

Thanks!

I would personally much prefer the second option, and I'd rather focus on keeping a good audit log one can find on logstash rather than working to keep the current git structure. While i understand it could be useful to inspect quickly when/how a change was made, I feel like the major pain point all users of requestctl, expert or otherwise, point out is the trial and error of creating a new object in an editor. I am confident the experience using the web interface is much less stressful even in this initial version, but it would keep getting better as we add new functions in the coming quarters.

Also, if you have a hierarchy of tasks that you want to run in a coordinated way, you should always account for the possibility one step fails for external reasons (for instance, the database master crashes suddenly), or for software bugs.

I think that when you have a list of tasks to execute that have side effects like creating databases and tables, you want the task to be idempotent, or - alternatively - to have a rollback mechanism in case of failure.

In T372604#10142394, @Scott_French wrote:

Alas, among the ways mw-debug is special, it configures an extra NodePort service that skips envoy (i.e., direct to apache) and in theory permits serving via HTTP for testing / benchmarking purposes:

service:
  deployment: production
  expose_http: true
  port:
    nodePort: 8444

See, e.g., https://gerrit.wikimedia.org/r/703739.

As I forgot about this, I failed to override the nodePort on the "next" release to a distinct value, which of course failed to apply:

Service "mediawiki-next" is invalid: spec.ports[0].nodePort: Invalid value: 8444: provided port is already allocated

Two options come to mind:

1. I can simply re-spin https://gerrit.wikimedia.org/r/1071945 with a nodePort: 8453 override in values-next.yaml (port doesn't seem to be used for anything else per code-search and follows the same +4000 convention).
2. I'm not 100% sure this is serving any useful purpose at the moment, given that any HTTP request directed at a wiki would receive a 302 to the canonical HTTPS URI. If that's the case, I can remove this service from mw-debug.

@akosiaris - Any preference among these options? Or @Joe if you have any context to share on current applicability of the original use case, that would be greatly appreciated as well.

A big difference between php-fpm and HHVM is that PHP doesn't have natively a wall clock based timeout. So for instance max_execution_time only refers to time spent executing php bytecode, so not for things like db queries or http requests.

If you want to use change dispatching from wikidata, which is a proven mechanism, then you'd probably be better off keeping the lexeme data within the wiki structure, and pass it to the orchestrator as a parameter of the function. That would allow you to re-parse the wiki page using the normal mechanism that's already established for wikis, and solve a lot of problems for you (including I think how to fetch the items, but I'd let @LucasWerkmeister comment on that).

In T368654#9942854, @DMartin-WMF wrote:

In Wikidata:Data_access, it says:

The following URL formats are used by the user interface and by the query service updater, respectively, so if you use one of the same URL formats there’s a good chance you’ll get faster (cached) responses:

https://www.wikidata.org/wiki/Special:EntityData/Q42.json?revision=1600533266 (JSON)

Does that mean we would have to specify a specific revision to get the benefit of caching?

Special pages are not cached at the edge, so there is no caching for that url, independently of indicating a revision or not:

$ curl -Is https://www.wikidata.org/wiki/Special:EntityData/Q42.json  | grep cache-control
cache-control: private, s-maxage=0, max-age=0, must-revalidate
$ curl -Is https://www.wikidata.org/wiki/Special:EntityData/Q42.json?revision=1600533266  | grep cache-control
cache-control: private, s-maxage=0, max-age=0, must-revalidate

In T374394#10133179, @Ladsgroup wrote:

In T374394#10132257, @jeremyb wrote:

That's not possible, the whole point of the API query is to decide whether or not to show the banner. (on enwiki the categories I'm filtering on for article (ns0) views are on the corresponding Talk. (ns1) so they're also not available in wgCategories.) I was worried more about performance for the API requests. I tested pretty thoroughly on a few devices so I was less worried about JS errors but of course WP has an enormous variety of clients.

CN banners work on a different level, one API request might be fast but if you add it to let's say enwiki CN, you will trigger a flood of several thousand requests a second every second. Reaching billions more requests to production in a day. That can easily bring down everything specially since API requests don't get CDN cached.

If something is not possible natively in CN, it's better not to circumvent it by API calls.

For the record, the cause was a relatively aggressive crawler filling up all resources. While we've rate-limited this bot, I think we should use robots.txt to ban crawling from most pages.

I'm finally seeing bounces get processed in logstash https://logstash.wikimedia.org/goto/3d34190bb82088f19669b0c66331d7c9

I still see the errors in the logs, and it's baffling. In fact, I've tried the command now listed in exim's configuration:

The check is done using Webrequest::getIP() which uses REMOTE_ADDR as a source for the address, and then overrides that using X-Forwarded-For only for trusted proxies.

Let me add some prospect, as I've heard people are complaining about this.

In T371633#10102614, @ItamarWMDE wrote:

Is this somehow related to T362084: [SW] [WBQC] shellbox-constraints returning 500 on preg_match error?

For the record, the reason we wanted to support large file uploads was not to worsen the performance of upload-by-url, which has since been fixed by making the process asynchronous. Better performance handling large files would be welcome, though.

I don't think we really need this, because I can't remember one episode in which this check has acutally fired and it wasn't expected/a false positive.

Now prometheus only reports scraping the correct ports https://prometheus-eqiad.wikimedia.org/k8s/targets?scrapePool=k8s-pods-metrics&search=statsd-exporter

In T371885#10049340, @Scott_French wrote:

Edit: Whoops, I completely missed T371885#10048618 onward before posting this. In any case, question still stands re: the annotation :)

For the endpoints marked down: it looks as if prometheus is scraping both container ports - i.e., 9102 (correct) and 9125 (statsd listen port, incorrect).

Not sure if that could somehow cause problems like those described in the task description, but it would at least explain T371885#10048571.

I wonder if we need to add an explicit prometheus.io/port annotation to ensure only 9102 is scraped?

I think we need to be able to pass to the release script a set of base images for mediawiki and web, and build the final images for each of those pairs.

I should add, wgLocalHTTPProxy and all those mechanisms are hacks, and if we want to do such things the right way, we should have a proper configuration table for read-only and read-write paths for different domains, and then instruct the application writers to use one or the other. I'd avoid replicating the hack we have at the traffic layer, because say one request gets erroneously sent to the wrong datacenter - now instead of paying 30 ms in total of penalty, we'll pay 30 ms per query to the mysql master.

There's quite a bit of incorrect information in the wall of text above, but to actually keep it short:
We went with just pointing to the read-write api because there's no system, within MediaWiki, to split requests between write and read and we didn't want to add ad-hoc logic to the service mesh just for that.

Thanks for the thorough explanation! I know the traffic folks were a bit worried about controlling stick tables from requestctl but I think this format is ok.