Page MenuHomePhabricator

[Quota increase]: globaleducation
Closed, ResolvedPublic

Description

Project Name: globaleducation
Type of quota increase requested: CPU and RAM
Amount to increase: 16 CPU and 32 GB RAM
Reason: Improve reliability of the data processing workers for outreachdashboard.wmflabs.org

While replacing servers running deprecated Debian versions recently, I took the opportunity to isolate the different parts of the Dashboard system more fully to reduce the frequency of system downtime and make data updates more reliable. In doing so, I used up all of the available quota, but now that I've run the system in this configuration for a few months, I have found that there is an area of bottlenecks that is preventing successful updates for some Programs & Events Dashboard events, which I could address by increasing the size (the RAM in particular) of several of the system components. (In particular, the processes that are intended to handle the least resources-intensive update tasks are sometimes assigned a task that involves a much larger dataset than expected, and these tasks are running that processes out of memory.)

I am in the midst of a major architecture update that I expect to greatly reduce the memory requirements of these processes, but in the meantime, a quota increase would help me sleep at night with less worry about the frequent "stuck" data updates and resulting complaints from end users.

Event Timeline

Done; please reopen the ticket when you no longer need the extra quota. :)