Even if it's a browser bug, since it's showing up in firefox and safari, I honestly think it's worth bypassing until the browser bug is fixed.

I will need to spend some time and dig deep on this but not sure I will able to do so right now. Maybe early next week.

The old reports are too old and the logs have been purged but for the Zimbawe $5 picture, I found two logs that might be useful:
https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.10.10?id=QIvhdZIBa-PL6vFeQ0-Z
https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.10.10?id=P4vhdZIBa-PL6vFeQ0-Z

I made some modifications and ran it again. I confirm that these hosts seems to have a different value and probably need to be recloned:

aawiki.page_restrictions {7: ['db1150.eqiad.wmnet:3313', 'db1154.eqiad.wmnet:3313', 'db1240.eqiad.wmnet:3313', 'db2139.codfw.wmnet:3313'], 8: ['db1157.eqiad.wmnet:3306', 'db1166.eqiad.wmnet:3306', 'db1175.eqiad.wmnet:3306', 'db1189.eqiad.wmnet:3306', 'db1198.eqiad.wmnet:3306', 'db1212.eqiad.wmnet:3306', 'db1223.eqiad.wmnet:3306', 'db2149.codfw.wmnet:3306', 'db2156.codfw.wmnet:3306', 'db2177.codfw.wmnet:3306', 'db2186.codfw.wmnet:3313', 'db2190.codfw.wmnet:3306', 'db2194.codfw.wmnet:3306', 'db2205.codfw.wmnet:3306', 'db2209.codfw.wmnet:3306', 'db2227.codfw.wmnet:3306', 'dbstore1007.eqiad.wmnet:3313']}

I'm repooling the host for now. If you need it shut down, let me know.

Force a reboot via mgmt console brought it back online. Worth checking what's going on.

Up now.

forced a reboot via mgmt console.

Nothing on SEL beside:

-------------------------------------------------------------------------------
Record:      63
Date/Time:   10/17/2024 13:13:00
Source:      system
Severity:    Critical
Description: A fatal error was detected on a component at bus 4 device 0 function 1.

I forgot. I do it right after lunch.

In T377442#10237518, @Lucas_Werkmeister_WMDE wrote:

replacing one class with a more lightweight subset is a minor optimization.

In T376565#10237476, @Lucas_Werkmeister_WMDE wrote:

What is the point of the LanguageCode class? Why not just use a string?

(The answer can’t be “it guarantees that the code is valid”, because you’ve made the class @newable, which means other code can’t assume that a LanguageCode contains a valid code – anyone is allowed to create a LanguageCode wrapping an invalid string.)

If you want to be able to edit wikitech, we can make it happen. If you don't want to, we can wait until end of November.

Confirmed. Deleted the whole mailing list.

It was docker. I rebuilt the image and now it's fixed.

I have applied the patch and restarted frontend service but it's solved which I'm 99% sure is because of server-side caching. Let me see how I can flush it.

In T706#10236098, @Milimetric wrote:

Hello, I am serving as interim Engineering Manager for the Data Products team. I need to create sprints and generally wield more power in Phab, please. Thank you!

I dropped 5K tables. After T355594#10235308

Give me a sec. I deploy it, then push the patch to gerrit?

The risk scenarios here are:

• You would have a lot of pages, like in order of millions, and only a handful of non-translation pages, this could become slow but the chance of it happening is really low.
• Adding more joins, after five or six starts to become hard for mariadb to find the correct join order and that's why we have a lot of "straight join" hints all over the code. It's not that adding one more join here is bad, it's that if it reaches joining of a lot of tables, then mariadb starts to act stupid.

@Robertsky You have no email connected to your wikitech account. I can set it for you if you can prove the account belongs to you (edit in a SUL wiki and wikitech to prove that). If that's not possible, let me see what I can do to prove ownership.

@Dominic3203 I forced a reset password on the email address connected to your wikitech account.

@Jc3s5h I can try to force a password reset email and you let me know if it worked. Does that work for you?

@CanonNi I can't find your user in wikitech. Are you sure you have a user there?

Thank you, I turn on the chainsaw now.

yeah, I was thinking of that but forgot. We definitely should do it.

After the cloning, the total number of mismatch in auto_increment value reported is now around 300. In almost all cases that I have checked, it's very very likely caused by replag and nothing to worry about. These stood out (via checking on cases that master has lower value than replicas):

eowikivoyage geo_tags {1: ['db2123 (codfw master)', 'db2178', 'db2192', 'db2213', 'db1213', 'db1200', 'db1230', 'db1185'], 5305: ['db1183 (eqiad master)', 'db2211', 'db2157', 'db2171', 'db2128', 'db1161', 'db1210']}
frwiki cur {1: ['db2214 (codfw master)', 'db2229', 'db2224'], 310635: ['db1201 (eqiad master)', 'db2217', 'db2193', 'db2158', 'db2169', 'db2180', 'db2151', 'db1165', 'db1180', 'db1187', 'db1168', 'db1231', 'db1173']}
frwiki ores_model {1: ['db2214 (codfw master)', 'db2229', 'db2224'], 12: ['db1201 (eqiad master)', 'db2217', 'db2193', 'db2158', 'db2169', 'db2180', 'db2151', 'db1165', 'db1180', 'db1187', 'db1168', 'db1231', 'db1173']}
gorwiktionary linter {19823: ['db2123 (codfw master)'], 19359: ['db1183 (eqiad master)', 'db2211', 'db2178', 'db2157', 'db2192', 'db2171', 'db2213', 'db2128'], 19879: ['db1213', 'db1200', 'db1230', 'db1161', 'db1210', 'db1185']}
jawiki cur {1: ['db2214 (codfw master)', 'db2229', 'db2224'], 283002: ['db1201 (eqiad master)', 'db2217', 'db2193', 'db2158', 'db2169', 'db2180', 'db2151', 'db1165', 'db1180', 'db1187', 'db1168', 'db1231', 'db1173']}
kabwiki linter {94756: ['db2209 (codfw master)'], 94729: ['db1189 (eqiad master)', 'db2190', 'db2149', 'db2177', 'db2227', 'db2156', 'db1157', 'db1166', 'db1223', 'db1212', 'db1198', 'db1175'], 94773: ['db2194']}
mniwiktionary linter {3354: ['db2123 (codfw master)', 'db2211', 'db2178', 'db2157', 'db2192', 'db2171', 'db2213', 'db2128'], 3364: ['db1183 (eqiad master)'], 3926: ['db1213', 'db1200', 'db1230', 'db1161', 'db1210', 'db1185']}
ptwiki ores_model {2220: ['db2204 (codfw master)'], 2326: ['db1222 (eqiad master)', 'db1233', 'db1246', 'db1182', 'db1156', 'db1229', 'db1188', 'db1197', 'db1162'], 2218: ['db2207'], 1994: ['db2189'], 2256: ['db2148'], 2144: ['db2126'], 2298: ['db2238'], 2158: ['db2175'], 2278: ['db2225']}
ruwiki cur {1: ['db2214 (codfw master)', 'db2229', 'db2224'], 57403: ['db1201 (eqiad master)', 'db2217', 'db2193', 'db2158', 'db2169', 'db2180', 'db2151', 'db1165', 'db1180', 'db1187', 'db1168', 'db1231', 'db1173']}
ruwiki ores_model {1: ['db2214 (codfw master)', 'db2229', 'db2224'], 17: ['db1201 (eqiad master)', 'db2217', 'db2193', 'db2158', 'db2169', 'db2180', 'db2151', 'db1165', 'db1180', 'db1187', 'db1168', 'db1231', 'db1173']}

Yeah, that'd be amazing.

In T375355#10227085, @Zache wrote:

Just as a update, update is live now.

@Ladsgroup, if there is no problems how we should handle the gradual increasing of the number of parallel edits? Ie. would it be too fast just to increase it with 1 per week until the number parallel writes is 5?

T377276: db2205 is stuck at "Shutdown in progress"

I can't even start mariadb on db2205 now :/

Thanks!

Is there anything we should do in downstream? Maybe make a strict CSP to minimize the impact? I can try to patch our setup but that's going to take a bit of time.

In T706#10227867, @Gehel wrote:

Hey! Could you please add @BTullis to Project Admins, so that he can create milestones in the Data-Platform-SRE project? He is a taking over from me while I'm on vacations.

Thank you!

In T376332#10226483, @Pppery wrote:

Maintenance bot still didn't create the post-creation work tree here for some reason.

wiki is live

wiki is live

wiki is live

wiki is live

wiki is live

wiki is live

In T375016#10225195, @Ladsgroup wrote:

wiki is live now. About to create five more.

wiki is live