Returns templates to me...

Like I said, it's intermittent. ImageTaggingBot appears to encounter this once every thousand requests or so.

In T117332#1771593, @Carnildo wrote:

Like I said, it's intermittent. ImageTaggingBot appears to encounter this once every thousand requests or so.

Can you capture the HTTP headers next time it happens? It may be a screwed up proxy or appserver

Server response headers from https://en.wikipedia.org/w/api.php?action=query&format=xml&titles=File:Kerbisher2.gif&tllimit=500&meta=userinfo&rvprop=content&prop=templates|revisions&redirects=1&uiprop=hasmsg

Cache-Control: private, must-revalidate, max-age=0
Connection: close
Date: Sun, 01 Nov 2015 19:05:14 GMT
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Age: 0
Server: nginx/1.9.4
Content-Type: text/xml; charset=utf-8
Client-Date: Sun, 01 Nov 2015 19:05:14 GMT
Client-Peer: 198.35.26.96:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
Client-SSL-Cert-Subject: /C=US/ST=California/L=San Francisco/O=Wikimedia Foundation, Inc./CN=*.wikipedia.org
Client-SSL-Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL
Client-Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Analytics: WMF-Last-Access=01-Nov-2015;https=1
X-Cache: cp1053 miss (0), cp4017 miss (0), cp4017 frontend miss (0)
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Powered-By: HHVM/3.6.5
X-Varnish: 823044711, 2788295235, 841757361

And a repeat an hour later, same page:

Cache-Control: private, must-revalidate, max-age=0
Connection: close
Date: Sun, 01 Nov 2015 20:06:01 GMT
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Age: 0
Server: nginx/1.9.4
Content-Type: text/xml; charset=utf-8
Client-Date: Sun, 01 Nov 2015 20:06:01 GMT
Client-Peer: 198.35.26.96:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
Client-SSL-Cert-Subject: /C=US/ST=California/L=San Francisco/O=Wikimedia Foundation, Inc./CN=*.wikipedia.org
Client-SSL-Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL
Client-Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Analytics: WMF-Last-Access=01-Nov-2015;https=1
X-Cache: cp1068 miss (0), cp4018 miss (0), cp4017 frontend miss (0)
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Powered-By: HHVM/3.6.5
X-Varnish: 1128580727, 2878596088, 845302430

In T117332#1772336, @Carnildo wrote:

Server response headers from https://en.wikipedia.org/w/api.php?action=query&format=xml&titles=File:Kerbisher2.gif&tllimit=500&meta=userinfo&rvprop=content&prop=templates|revisions&redirects=1&uiprop=hasmsg

The API is correctly reporting the fact that there are no templatelinks recorded for the title:

db1052 [enwiki]> select * from templatelinks where tl_from=48433757;
Empty set (0.00 sec)

db1052 [enwiki]> select * from pagelinks where pl_from=48433757;
Empty set (0.01 sec)

db1052 [enwiki]> select * from categorylinks where cl_from=48433757;
Empty set (0.01 sec)

db1052 [enwiki]> select * from imagelinks where il_from=48433757;
Empty set (0.00 sec)

db1052 [enwiki]> select * from externallinks where el_from=48433757;
Empty set (0.00 sec)

db1052 [enwiki]> select * from iwlinks where iwl_from=48433757;
Empty set (0.00 sec)

The question then becomes "why aren't there templatelinks recorded for this title"? I know @aaron has been making changes related to pushing links updates into jobs, so CCing him in case he has any insight.

Change 253077 had a related patch set uploaded (by Aaron Schulz):
Race condition fixes for refreshLinks jobs

https://gerrit.wikimedia.org/r/253077

Change 253077 merged by jenkins-bot:
Race condition fixes for refreshLinks jobs

https://gerrit.wikimedia.org/r/253077

Is it possible to painlessly backport the fix to wmf.7? Per https://www.mediawiki.org/wiki/MediaWiki_1.27/Roadmap, all wikis are stuck on wmf.7 for the next two weeks.

It seems to me that the code fix will solve the problem partially only. But unsure if this bug should be reopened or another one created for this.

After the bug in code is fixed, we will still have a bunch of pages with missing / incorrect entries in link tables distributed among all wikis. And I doubt the problem to disappear automatically after some period of time as it is database, not cache related. Null edits on all pages hurt by this bug should update the link tables and fix this. However:

• I see no point to do it before a fixed version is uploaded to servers (as no warranty that null edir really fixes the problem then)
• It is a big pain to null-edit all pages in all wikis (for enwiki it may take years; even on en/fr.wikisource it would be many months)
• I have no idea how to locate the affected pages, or at least locate the pages that are suspected to be affected.

Anybody is able to estimate when the bug started to appear (the affected pages wuold be the pages editted is a specific period of time, and not later)? Or suggest another database query that should contain all affected pages among result (but not all pages on a wiki)?

Things like this aren't always easy to test in development. People make "best effort" towards fixes, and then they need testing in production,

The bug was reported at the end of October. You can probably knock a couple of weeks of that for pages to nulledit. I suspect it wouldn't take that long using a bot working 24/7 at a reasonable "edit rate"

Note that this is not even deployed yet…

Change 256269 had a related patch set uploaded (by Reedy):
Race condition fixes for refreshLinks jobs

https://gerrit.wikimedia.org/r/256269

Change 256269 abandoned by Jforrester:
Race condition fixes for refreshLinks jobs

Reason:
Let's use the train rather the pull it forward a couple of days.

https://gerrit.wikimedia.org/r/256269

Hi, if I understand correctly, this means any affected pages that were tagged for speedy deletion (e.g. on enwiki) are not showing up in the speedy categories, and hence may not get deleted. This is a rather serious issue for attack pages and copyright violations. I was talking to some folks on IRC, perhaps we could go by edit summary to automate null edits on pages that were tagged for speedy by our known new page patrollers and bots? Is there an easier technical solution on WMF side for this?

In T117332#1871699, @MusikAnimal wrote:

Hi, if I understand correctly, this means any affected pages that were tagged for speedy deletion (e.g. on enwiki) are not showing up in the speedy categories, and hence may not get deleted. This is a rather serious issue for attack pages and copyright violations. I was talking to some folks on IRC, perhaps we could go by edit summary to automate null edits on pages that were tagged for speedy by our known new page patrollers and bots? Is there an easier technical solution on WMF side for this?

IMO you are right. We have exactly this problem.

My tests performed 1-2 weeks ago on plwikisource showed that at least 1-2% pages editted were affected by this bug. But ratio can be different for a large traffic wiki like enwiki.

Current estimates of potentally affected pages (from quarry):

select count(distinct rev_page) from enwiki_p.revision where rev_timestamp < '20151209015000' and rev_timestamp > '20151001000000';

count(distinct rev_page)
2877079

The number is likely overestimated as the '20150915000000' timestamp is arbitrary (@Reedy said above: "The bug was reported at the end of October") and any page already editted after '20151209015000' should have correct link tables entries.

Assuming a single bot operating with rather high 50/min ratio should fix it in 2877079/(50*60*24) = 39 days (of continuous bot work)

In T117332#1817443, @Anomie wrote:

Let's call this "resolved". If the issue still occurs on wikis running 1.27.0-wmf.8 (you can check that using Special:Version), feel free to reopen.

This (or similar) problem still happens. The 6 pages listed in https://pl.wikisource.org/wiki/Specjalna:Nieskategoryzowane_strony :

https://pl.wikisource.org/wiki/Strona:Gustaw_Daniłowski_-_Lili.djvu/57
https://pl.wikisource.org/wiki/Strona:Gustaw_Daniłowski_-_Lili.djvu/58
https://pl.wikisource.org/wiki/Strona:Gustaw_Daniłowski_-_Lili.djvu/59
https://pl.wikisource.org/wiki/Strona:Gustaw_Daniłowski_-_Lili.djvu/60
https://pl.wikisource.org/wiki/Strona:Gustaw_Daniłowski_-_Lili.djvu/61
https://pl.wikisource.org/wiki/Strona:Gustaw_Daniłowski_-_Lili.djvu/62

were created yesterday and are not properly linked to the category:

https://pl.wikisource.org/w/index.php?title=Kategoria:Nieskorygowana&pagefrom=Gustaw+Dani%C5%82owski+-+Lili.djvu%2F50%0AGustaw+Dani%C5%82owski+-+Lili.djvu%2F50#mw-pages

I can't check database at the moment as Quarry seems to be stuck. Any hints?

See also https://gerrit.wikimedia.org/r/#/c/258445/ which will be in the next release. That fixed a bug where new pages would have bad links until the link job was retried sometimes (around 1 hour).

This doesn't appear to have been fixed in -wmf8. ImageTaggingBot reported errors on https://en.wikipedia.org/wiki/File:Elitch_Gardens_Logo.png (query https://en.wikipedia.org/w/api.php?action=query&format=xml&titles=File:Elitch_Gardens_Logo.png&redirects=1&meta=userinfo&tllimit=500&prop=templates|revisions&uiprop=hasmsg&rvprop=content) and https://en.wikipedia.org/wiki/File:Charity_Standish.jpeg (which has since been edited), both of which were uploaded earlier today.

In T117332#1876188, @aaron wrote:

See also https://gerrit.wikimedia.org/r/#/c/258445/ which will be in the next release. That fixed a bug where new pages would have bad links until the link job was retried sometimes (around 1 hour).

wmf9 is out everywhere since yesterday.

Not fixed in -wmf9: https://en.wikipedia.org/wiki/File:JoJo_-_LOVEJO2_(Official_EP_Cover).png (query https://en.wikipedia.org/w/api.php?action=query&format=xml&prop=templates|revisions&titles=File:JoJo_-_LOVEJO2_(Official_EP_Cover).png&tllimit=500&meta=userinfo&rvprop=content&uiprop=hasmsg&redirects=1), uploaded yesterday morning, still has problems.

For those who saw this issue, could you re-check if you still experience this issue? There were quite a lot of changes around jobs and related topics since this task was first opened, it could have been fixed.

ImageTaggingBot's most recent encounter with this issue is from June 6, for https://en.wikipedia.org/wiki/File:Tennis_A_Team_1926.jpg. Since the bot has been averaging only a few reports of this a month, any fixes since then won't be obvious yet.

This happened again with https://en.wikipedia.org/wiki/User:TimSGearhart/sandbox. A speedy deletion tag was added on April 16 but the page was never put in the category. I have since made a null edit and that fixed it.

Maybe we could run a query to look for Template:Db (and variants) transclusions where the page is not in Category:Candidates for speedy deletion?

In T117332#4269288, @MusikAnimal wrote:

This happened again with https://en.wikipedia.org/wiki/User:TimSGearhart/sandbox. A speedy deletion tag was added on April 16 but the page was never put in the category. I have since made a null edit and that fixed it.

Maybe we could run a query to look for Template:Db (and variants) transclusions where the page is not in Category:Candidates for speedy deletion?

Was the page being tracked in "what links here" of the template, before you did the null edit? It would be strange to have the templatelinks addition and not the categorylinks, for the same edit. If it was not in templatelinks neither, then that query won't give any useful result.

In T117332#4269299, @Ciencia_Al_Poder wrote:

Was the page being tracked in "what links here" of the template, before you did the null edit? It would be strange to have the templatelinks addition and not the categorylinks, for the same edit. If it was not in templatelinks neither, then that query won't give any useful result.

Oh good point. No it probably was not in "what links here", then. I should have checked.

Can you think of a way to find these uncategorized pages?

In T117332#4269316, @MusikAnimal wrote:

Can you think of a way to find these uncategorized pages?

There's no way other than reparse all the pages again (running refreshLinks.php ), or retrieve all page content and search for that template (either xml dump or api query, the latter may take a lot of time)