Commons talk:Tracking external file usage

Previous discussion

Early discussion about this activity was at Commons:Village pump/Proposals/Archive/2022/06#File usage on openstreetmap.org, The following page uses this file and Commons:Bots/Requests/Usage Bot. --bjh21 (talk) 23:01, 17 August 2022 (UTC)

Proposed target: OpenStreetMap Wiki

Previous, approved application.

I'd like to propose tracking files used on the OpenStreetMap Wiki. The OpenStreetMap Wiki is used for documenting the workings of OpenStreetMap, and specifically for describing the meanings of keys in the OpenStreetMap database. It uses a lot of files from Commons (just under 30,000 at the moment), mainly as examples of features that it describes. This use is educational, as defined by COM:EDUSE, so it would be useful to track it on Commons to assist with deletion decisions. We currently track files used in the OpenStreetMap database itself at Commons:Files used on OpenStreetMap, but not those used on the wiki. At a technical level, I would just add the OpenStreetMap Wiki to the targets tracked by Usage Bot, and have it construct galleries under Commons:Files used on the OpenStreetMap Wiki. I've put a sample gallery with a random thousand files at User:Bjh21/Usage demo. If this proposal is accepted there would initially be 30 such galleries. I will also mention this proposal at Commons:Village pump/Proposals and osmwiki:Talk:Wiki. --bjh21 (talk) 14:43, 18 August 2022 (UTC) As OSM contributor and very active person on OSM Wiki, especially with files: this is a great and highly welcome idea! Mateusz Konieczny (talk) 20:18, 18 August 2022 (UTC) Thanks for this proposal. I've run into multiple situations where an image link on the OSM Wiki was broken when someone here renamed an image without leaving a redirect, thinking it would have no effect. It happens in categories where editors have a very strong preference for consistent file names. Unfortunately, the OSM Wiki doesn't have a mechanism to automatically update links to an image when it gets renamed here, so your proposal would avoid misunderstandings. Note that the OSM Wiki also includes a Wikibase instance: the image (P28) property is set to a file name, typically a Commons file name. (OSM Carto image (P39) can also be set to an actual Commons image, but I think that's less common.) – Minh Nguyễn 💬 05:44, 19 August 2022 (UTC) As a Commons and OSM Wiki contributor, I find it important to see reuse on projects as important as OSM Wiki. I am totally for this initiative — Koreller (talk) 10:02, 19 August 2022 (UTC) @Mxn: The Wikibase instance will be a bit of a problem, since it doesn't seem to show up in the list of places where a file is used on the OpenStreetMap Wiki. Wikidata doesn't have this problem, so I expect it's soluble. --bjh21 (talk) 10:12, 19 August 2022 (UTC) @Bjh21: Unlike image (P18) on Wikidata, which is typed as a Commons file, image (P28) on OSM Wikibase is typed as a string; some site JavaScript is dynamically embedding an image. I don't know if that's typical of third-party Wikibase instances, but it means the usual Special:WhatLinksHere mechanism isn't quite comprehensive. Sophox can query images and has a SPARQL endpoint just like the Wikidata Query Service. Would that suffice? – Minh Nguyễn 💬 10:38, 19 August 2022 (UTC) @Mxn: I expect I can make that work. It's quite a lot of extra effort, though, so I'll start with files used on the wiki itself. --bjh21 (talk) 11:14, 19 August 2022 (UTC) It's a week since the last comment, and everyone seems to be in favour, so I've set up Commons:Files used on the OpenStreetMap Wiki and the bot is currently running to populate it. Thank you all for your input! --bjh21 (talk) 13:28, 26 August 2022 (UTC)

Proposed targets: Wikimedia chapter wikis

Previous, approved application.

Several Wikimedia chapters have their own MediaWiki wikis. Most of these are hosted on Wikimedia infrastructure, so they appear in Special:GlobalUsage. There are a few, though, that are hosted elsewhere and that don't appear in GlobalUsage. Most of these could be easily supported by Usage Bot. I propose to add tracking of file usage on these wikis. The targets I propose covering initially are: Chapter Wiki Proposed base gallery # Files m:Wikimedia Österreich wmat: Commons:Files used on Wikimedia Österreich Mitglieder 1770 m:Wikimedia Australia wmau: Commons:Files used by Wikimedia Australia 345 m:Wikimedia Czech Republic wmcz_old: Commons:Files used by Wikimedia Czech Republic 209 m:Wikimédia Magyarország wmhu: Commons:Files used by Wikimédia Magyarország 75 m:Wikimedia UK wmuk: Commons:Files used by Wikimedia UK 2671 m:Wikimedia District of Columbia wmdc: Commons:Files used by Wikimedia District of Columbia 265 I've omitted Wikimedia Australia because the bot's API requests to it time out. I've omitted Wikimedia Italia because it doesn't allow anonymous API access. I've omitted Wikimedia South Africa because it's using a version of MediaWiki too old for the current version of Pywikibot. Most of these wikis don't have their own names apart from the name of their chapter. This is why the names of the galleries are "by" rather than "on". Please tell me if this is incorrect. They also all have closed registration, so I haven't been able to mention this on the target wikis. Since I'm just using public APIs to bring them closer to parity with Wikimedia-hosted wikis, I don't think this is a serious problem. As with the last one, I'll post a link to this proposal on Commons:Village pump/Proposals. --bjh21 (talk) 15:15, 24 September 2022 (UTC)  Support.   — Jeff G. ツ please ping or talk to me 15:36, 24 September 2022 (UTC)  Support not used any of them but may help this people to reduce how often they get broken files due to unneeded redirect destruction (or just moving file without leaving redirect) Mateusz Konieczny (talk) 22:10, 24 September 2022 (UTC)  Support Looks useful. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 18:19, 25 September 2022 (UTC)  Support, whatever can be supported should be supported by the "Special:GlobalUsage" software. --Donald Trung 『徵國單』 (No Fake News 💬) (WikiProject Numismatics 💴) (Articles 📚) 21:12, 25 September 2022 (UTC)  Support --Achim55 (talk) 20:25, 26 September 2022 (UTC)  Question@Bjh21: What API call is giving you the WMAU error? It looks like e.g. generator=allfileusages is working (albeit a bit slowly). (I'm one of the WMAU sysadmins so would be happy to help get this working properly!) — Sam Wilson ( Talk • Contribs ) … 09:35, 27 September 2022 (UTC) @Samwilson: The request that times out is https://wikimedia.org.au/w/api.php?generator=allfileusages&gafunique=&prop=imageinfo&iiprop=&action=query&indexpageids=&continue=&gaflimit=500&meta=userinfo&uiprop=blockinfo%7Chasmsg&maxlag=5&format=json. With the API Sandbox I find that gaflimit=25 takes just under 25 seconds, so maybe if I extend the timeout to 600 seconds that'll be enough. --bjh21 (talk) 20:12, 27 September 2022 (UTC) @Bjh21: Sounds good! But I do wonder what's taking it so long. It'd be nice if there was a specific (perhaps more efficient) 'InstantCommons Usage' API; it is a feature of core after all. :-) — Sam Wilson ( Talk • Contribs ) … 00:14, 28 September 2022 (UTC) @Samwilson: I've worked out how to have the bot make smaller batch requests for certain sites, which seems more polite and robust than increasing the timeout to several minutes. This means I can include WMAU in this batch. Apart from the fact that it returns information about local files as well, generator=allfileusages with gafunique= is exactly the API I need. Adding filtering by repository would help somewhat, but I think there must be some fundamental inefficiency here. With how slow WMAU is, I suspect a missing index or something like that. --bjh21 (talk) 16:15, 29 September 2022 (UTC) @Bjh21: Okay great, that's good. And yes, I'll dig into the WMAU set up and see what's going wrong. Thanks! — Sam Wilson ( Talk • Contribs ) … 06:24, 30 September 2022 (UTC)  Comment I've now added Wikimedia Australia to the list since it seems to be working. --bjh21 (talk) 16:17, 29 September 2022 (UTC) Since there have been no objections and I'm going to be travelling for the next two days, I've deployed the code to create galleries for the chapter wikis. It'll first run as part of the weekly cron job on Monday. Thank you all for your input! --bjh21 (talk) 16:55, 30 September 2022 (UTC)  Support --Häferl (talk) 12:43, 6 October 2022 (UTC)

Proposed target: RegiowikiAT

Previous, approved application.

Karl Gruber has suggested on my talk page that we should track file usage on RegiowikiAT. It appears to be run by Wikimedia Österreich, so it might fall within the proposal above, but it also uses over 11,000 Commons files, more than all the chapter wikis above combined. Would there be any objection to my having the bot create galleries under Commons:Files used on RegiowikiAT for Commons files used there? --bjh21 (talk) 00:00, 6 October 2022 (UTC)  Support I think that's an excellent idea. BR, Asurnipal (talk) 07:01, 6 October 2022 (UTC)  Support --M2k~dewiki (talk) 07:59, 6 October 2022 (UTC)  Support at the moment the tagging is done manually (e.g. File:GuentherZ 2006-06-15 2457 Stift Altenburg.jpg) which is tedious and error prone and as always incomplete. Save our human resources for more useful tasks. --Herzi Pinki (talk) 08:03, 6 October 2022 (UTC)  Support --Isiwal (talk) 09:46, 6 October 2022 (UTC)  Support.   — Jeff G. ツ please ping or talk to me 10:24, 6 October 2022 (UTC)  Support --Anton-kurt (talk) 12:15, 6 October 2022 (UTC)  Support --Häferl (talk) 12:42, 6 October 2022 (UTC)  Support --Eweht (talk) 15:50, 6 October 2022 (UTC)  Support Braveheart (talk) 16:49, 6 October 2022 (UTC)  Support --Krd 06:42, 7 October 2022 (UTC)  Support --Mfchris84 (talk) 18:55, 7 October 2022 (UTC) That's pretty convincing. I've updated the bot's code to support RegiowikiAT and it will populate the galleries tonight. Thank you all for you input! --bjh21 (talk) 10:51, 9 October 2022 (UTC) many thx -- K@rl (talk) Diskussion 11:19, 9 October 2022 (UTC)

Proposed target: AARoads Wiki

Previous, approved application.

Rschen7754 has suggested on my user talk page that Usage Bot should track file usage on the AARoads Wiki. It's a free-content educational project, so I think its uses of files are likely to be of interest to those judging whether a file is educationally useful. The AARoads wiki is using over 69,000 files from Commons, so this would require 70 galleries under Commons:Files used on the AARoads Wiki. The bot would update these galleries weekly, not least because it currently takes nine hours to get the entire list of used files. Would there be any objection to my having the bot start tracking usage on this wiki? As usual, I'll post a link to this proposal on Commons:Village pump/Proposals. --bjh21 (talk) 21:24, 14 September 2023 (UTC)  Comment There's a gallery of 1000 randomly chosen files currently in use on the AARoads Wiki temporarily at User:bjh21/Usage demo so people can see how the galleries will work. --bjh21 (talk) 21:31, 14 September 2023 (UTC)  Support as original proposer. --Rschen7754 00:01, 15 September 2023 (UTC)  Support as well. –Fredddie™ 00:05, 15 September 2023 (UTC)  Support SounderBruce 04:31, 15 September 2023 (UTC)  Support—Imzadi 1979 → 03:54, 16 September 2023 (UTC) Thank you all! I've updated the bot's code to support the AARoads Wiki, so if everything works it should create a load of galleries under Commons:Files used on the AARoads Wiki some time on Monday. --bjh21 (talk) 21:28, 23 September 2023 (UTC) ... and only a few Mondays later, the galleries are now in place. --bjh21 (talk) 19:18, 9 October 2023 (UTC)

Proposed targets: Wikimedia movement affiliates (and specifically Deoband Community Wikimedia)

TheAafi (DCW) has suggested User talk:Bjh21#Files used on DCW website that Usage Bot should track file usage on the Web site of Deoband Community Wikimedia (DCW). DCW is a Wikimedia user group which is a different kind of Wikimedia movement affiliate from the chapters I asked about above. Like chapters, some user groups have wikis on Wikimedia infrastructure that are covered by Special:GlobalUsage. Examples include wmbr: and wmge:. But other user groups, like DCW, have wikis that aren't covered by Special:GlobalUsage and Usage Bot could usefully track their file usage. Would there be any objection to my having Usage Bot track file usage on MediaWiki wikis run by all kinds of Wikimedia movement affiliate, including user groups like DCW? --bjh21 (talk) 17:34, 28 February 2024 (UTC)

•  Support As far as I'm concerned, any official Wikipedia group's content is within scope, regardless of the specifics of their charter and organization structure. The Squirrel Conspiracy (talk) 21:48, 28 February 2024 (UTC)
•  Support not as someone associated with DCW as a volunteer and a Program Lead (TheAafi (DCW)), but as TSC says above that regardless of charters and structures, every Wikimedia movement affiliate's content is within scope. Best regards, ─ Aafī (talk) 17:55, 29 February 2024 (UTC)
• Given that DCW is a recognized affiliate, I have no concerns as long as DCW maintains its status. 511KeV (talk) 16:07, 2 March 2024 (UTC)

Thanks all for your opinions. Commons:Files used by Deoband Community Wikimedia now exists and will be maintained by the Usage Bot. I'll see what other user group wikis I can find. --bjh21 (talk) 21:17, 18 March 2024 (UTC)

Fewer broken galleries; more files used by OSM

Enhancing999 informed me that broken galleries are not welcome on Commons, which meant that Commons:Files used on OpenStreetMap needed to change. I worked out how to have Usage Bot efficiently avoid putting broken links in its gallery pages, and that then allowed me to make its interpretation of values from OpenStreetMap more liberal. It now interprets OSM data roughly like MapComplete, but with compatibility with OpenStreetBrowser as well. The result is that it's removed a couple of hundred broken links and added about 22,000 new files.

It's unfortunate that this has lost us a convenient way of finding actually broken links on OpenStreetMap. I'll see if I can come up with a way of listing those that doesn't just end up full of mis-interpretations of tags. --bjh21 (talk) 13:09, 3 August 2024 (UTC)

That's indeed unfortunate as User:Ivanbranco is using the broken links to improve OSM tags. In the meantime, I use CommonsChecker4OSM to find broken links for my Maproulette challenge. --LennardHofmann (talk) 14:42, 4 August 2024 (UTC)

That’s a bummer. I was hoping to approach you at some point about adding support for OpenHistoricalMap, which has a similar data model. The site currently has built-in support for displaying arbitrary images based on image tags, but due to a serious amount of bitrot, I’ve proposed adding support for wikimedia_commons tags with the idea that a bot like yours could help us catch broken file references. [1][2] For what it’s worth, OHM has the same data model as OSM but a strong preference for keys like image:3 over semicolon value separators. – Minh Nguyễn ^💬 00:32, 5 August 2024 (UTC)

Add WikiConference North America wiki

Hi, could the https://wikiconference.org/ wiki be added as well? It's run by Wikimedia DC and a bunch of slide decks on Commons are referenced there. Legoktm (talk) 15:17, 5 October 2024 (UTC)

 Support. In scope. _Regards, Aafi ^(talk) 17:23, 5 October 2024 (UTC)

@Legoktm: According to m:WikiConference North America, WCNA is a user group in its own right, but either way it's covered by the decision above about Wikimedia movement affiliates. I've added support to the Usage Bot and manually run it to populate Commons:Files used by WikiConference North America. The galleries will be updated every Monday. Thanks for the suggestion! --bjh21 (talk) 22:01, 14 October 2024 (UTC)

Removal of Wikitech

Taavi has pointed out to me that Wikitech's use of Commons files is now properly reflected through Special:GlobalUsage. So there's no need for Usage Bot to maintain Commons:Files used on Wikitech any more. So I've removed the code to do that and requested deletion of the galleries. --bjh21 (talk) 22:18, 14 October 2024 (UTC)