This should be fixed now that ms-be2075 is taken out of the ring. Thanks to @MatthewVernon for doing all the heavy lifting.

In T382705#10422179, @BCornwall wrote:

(it's been quite a day for swift!)

@TheDJ That was a result of a separate issue that is now resolved (it's been quite a day for swift!)

Mentioned in SAL (#wikimedia-operations) [2024-12-23T20:16:56Z] <Emperor> weighted ms-be2075 to zero T382705 T382707

Ehm. it this a problem ? or a side effect of the depool taking effect after that 20:15 window ?

The depool won't entirely help (writes always go to both clusters), but diverting read traffic to eqiad swift should help mitigate user impact a bit. We should restore it before US staff stop work at the end of today, though.

Mentioned in SAL (#wikimedia-operations) [2024-12-23T14:10:02Z] <Emperor> depool codfw swift T382705

ms-be2075 will be effectively removed from the ring (weights set to 0), but a small snag: Swift rings have an enforced minimum time between changes for data integrity reasons and the next availability for application will be at 20:15 UTC. Unfortunately, we'll need to wait.

Change #1106303 merged by BCornwall:

[operations/puppet@production] Swift: Mark ms-be2075 as failed, remove from prod

https://gerrit.wikimedia.org/r/1106303

ms-be2075 has a data link reset a few times a minute:

oh, wrong link, and wrong screenshot, I copied from the wrong browser tab :D

Change #1106303 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] Swift: Remove ms-be2075 from prod hosts

https://gerrit.wikimedia.org/r/1106303

Thank you for reporting @Yann. I created T382705 for this one.

This happened again while trying to upload a new version of https://commons.wikimedia.org/wiki/File:The_Lion_of_the_Moguls_-_Le_Lion_des_Mogols_(1924)_dir._Jean_Epstein.webm

04369: finalize/189> Still waiting for server to publish uploaded file
04374: FAILED: stashfailed: An unknown error occurred in storage backend "local-swift-codfw".

To this date, the enwiki community has supported (the implementation of) restricting non-confirmed users (of enwiki) from migrating uploads into Commons.

Glad to hear it uploaded right now :)
There's no need to ping me specifically, I get all the sre-swift-storage items sent to me already.

I have uploaded this file successfully. If I got this error again, I'll add SRE-swift-storage project and ping you.

I'm afraid we don't keep swift logs far enough back to 7th November, so I can't provide any feedback from a swift point of view.
[you could try again to see if the upload now works]

Amusingly, Tim last touched that line in https://github.com/JamesHeinrich/getID3/commit/273e47678035658a6c1c6837448f1732026b77cb

@brennen ah yes my old nemesis, getid3's MPEG parser

@bvibber, @Umherirrender any thoughts on this one?

Alright, as noted in T379035#10318390, since ~ 17:50 UTC today all three job types that are critical to (async) uploads have been lifted out of the low-traffic consumer and into dedicated rules. Thus, they should no longer be exposed to this kind of isolation failure.

Yeah some systems can deal with that, some don't. We don't. If someone wants to figure out why, patches are definitely welcome.

none the less these files are handled fine by ffmpeg, VLC, the stock video player of my smart phone and web browsers on windows and linux. Also MW extracts the audio part.

These files lack cues (which provides time -> byteoffset mapping) and "Unless Matroska is used as a live stream, it SHOULD contain a Cues Element"