Would per-format canaries allow to allocate different hardware resources per file format? TIFF files would benefit to get more memory, for example.

With the power of buildpacks I was even able to update very easily to Java 23 :) https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/commit/c00aee97338ec2cb0af4051e2d82555aa8033e8f

@dcaro @Slst2020 have you enabled the cache already? I switched to Toolforge Build Service tonight in order to get Java 21 and the build is really fast! I don't need anything more to increase build performance, I'm very happy with the current situation :)

I managed tonight to:

• donwload jdk21 from Adoptium at build time on GitLab CI and cache it using GitLab cache mechanism. It's fast enough for me
• use jdk21 with toolforge build service / buildpacks. It's also pretty fast

For me the errors are gone (toolforge job service works, I was able to build and deploy my tool. No more DNS errors, everything looks fine).

Wow thank you Zache for having found this. I think this is definitively the way to go.

Probable duplicate of T373243

Same for my tool (pod spacemedia-6fdcc8d798-8sncn). Started to fail at 2024-08-25T17:38:18.469Z with error message "java.net.UnknownHostException: tools.db.svc.wikimedia.cloud"
I don't see name resolution problem on bastion nor my cloud vps instances.

new one, lot of issues this week, 600s of replag right now on s4 :(

OK, thank you, let's consider it fixed then. We'll try to solve the remaining issues one by one.

Status update: the system has been completely updated and restructured as follows:

• Reduced the number of celery workers to 4 workers per encoding instance
• Doubled the number of instances from 3 to 6

Thank you! I need this quota even after the migration to fix T365154: one of the many errors causing the permanent outages was too many celery workers per encoding instance, causing OOM errors under heavy load. in order to keep the same number of workers with a more stable system, I reduced the number of workers per instance to increase their available memory, and doubled the number of instances (from 3 to 6).

Update completed!

@fnegri thanks a lot, it works! Is there a procedure for me to follow to avoid this problem? I am going to recreate encoding06 instance.

@Andrew can I get support from WMCS team, I got something that doesn't work and I don't understand what?

Thank you Andrew!

Update: OK I understand that NFS and role::labs::lvm::srv are not related:

@Andrew in fact I need help.
I see the old encoding instances use the role::labs::lvm::srv puppet role to get more disk space.
I understand this is outdated and now we're using Cinder volumes directly from Horizon. Should I go this way? Does it mean the NFS server will no longer be necessary? I'm not sure to understand exactly how the storage part works.

@ Everyone: I wasn't able to work on v2c the past two weeks, starting from right now I'm going to update the whole infra to hopefully get rid of this problem for good.

In T369267#10002632, @dcaro wrote:

I think I've done that, but the way to change the branch was to delete the main and rename the master to main, let me know if that does not work

In T369267#10002632, @dcaro wrote:

If you give me your sonarcloud.io user I can add you manually

Sure, it's don-vip@github

Hi Andrew!
Sorry for the delay. Thanks a lot for the maintenance of the nfs server :)
For the rest of the activities, it's ok, I hope to complete them this week :)
I won't hesitate to ask for help if I face difficulties.

Thank you @dcaro! I managed to perform an analysis on spacemedia:

• https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/jobs/318094
• https://sonarcloud.io/summary/new_code?id=toolforge-repos_spacemedia&branch=main

@bd808 ok, noted! Thanks a lot @Slst2020!

I managed to fix almost all files except these two:

• https://commons.wikimedia.org/wiki/File:GolfoMexico_AWFI_20200505_210_069_L2_BAND161514.tif
• https://commons.wikimedia.org/wiki/File:Ontario_NY_PA_OH_MI_AMAZONIA_1_WFI_20220509_044_008_L2_BAND4321.tif

I found a way to solve the issue :)

In T366353#9965169, @Urbanecm_WMF wrote:

Let's avoid prematurely jumping into conclusions :). As showed above, your comment can be fully explained by the test not taking the TIFF extension into account (which is what reported the error here). As of now, I do not see any evidence on this task that would suggest it's specifically Commons (its config and/or infrastructure) that would be at fault here. As of now, I'm happy to take a look at what might be happening here, but I don't believe engaging in a discussion in which the infrastructure is repeatedly blamed without evidence would be productive. Thank you for your understanding.

I've setup a local mediawiki instance (1.39, using Ubuntu package) with default config:

Tried Brasilia 20170312a0416 Mediana B151413.tif

Thank you @Urbanecm_WMF!
I see the files have been imported, but the thumbnails have not been generated:

Hi @Urbanecm_WMF !
It's weird, I can download files without problem from Cloud VPS:

@Xqt I reviewed the code in gerrit and am OK with it. I'm pretty confident it will work :)

Thank you!

Sorry Andrew, still one request!

Yes, it works! https://codeclimate.com/github/toolforge/video2commons
Thank you :)

Hello Andrew, one last request (I hope).

My plan to solve this, with current status:

Is it possible to make this behaviour configurable? It broke video2commons in an horrible way, as we serialize API errors in our redis instance, we now have keys that contain whole serialized videos (hundreds of megabytes of binary data). It's really interesting to have most of the parameters (filename, mime type and so on) but we clearly don't want to have file contents in the string representation.

Caused by T333957 change in Pywikibot: https://github.com/wikimedia/pywikibot/commit/afab98e41f4366a98659ae97ba43c9428fa590ce

Found a big problem. After connecting to the Redis instance, it appears the db1 database contains 3.6GB of data for ~6000 keys, which is a lot for a database that only contains the current status of each task (the text displayed in the frontend).
If we look at the 15 biggest keys, they are between 86 Mb and 435 Mb, which is absolutely not expected (they usually only a few bytes/kilobytes of text).
The (truncated) contents of the biggest key is:

Credentials mistake fixed (thanks to Andrew). Now video2commons should be fully up again.

Thank you again!

... and to solve the mistake I made :/ v2c is temporarily unable to upload new files :(

Status update:

• Puppet configuration problem on video-redis-buster: fixed
• Logrotate configuration problem that filled the disk of all encoders: fixed
• Workers are still able to pickup tasks and transcode videos

@Andrew can you please also add me in the Toolforge project? (video2commons). I need this access to manage the frontend.

Second issue on encoder06 instance. Puppet agent is failing. When launching it manually:

First issue solved regarding the puppet alert firing continuously on video-redis-buster.video.eqiad.wmflabs instance for many months (prior to the reboot of 20th May).

I don't know. I'm just discovering how video2commons works, and trying to not break anything.

Thank you! I'm already receiving puppet alerts :)

Yes, taavi pointed us towards the change that repaired video2commons: https://phabricator.wikimedia.org/T365154#9915037

Thank you!

I confirm, this morning it works. It was not working for a few days. Feel free to close the ticket if nothing can be done to understand what went wrong.

Not sure how long the tool will remain functional, we already lost two of the four encoding instances:

Did this change fix T365154? video2commons became live again at the general surprise, around the time this change was applied.

The encoding instances became busy again around 2024-06-20T15:48:00