I'm BDavis (WMF) on wiki, bd808 on irc & GitLab, and BryanDavis on Gerrit & Wikitech.
I've got a thing for 🦄s. Don't judge.
I work for or provide services to the Wikimedia Foundation, but this is my only Phabricator account. Edits, statements, or other contributions made from this account are my own, and may not reflect the views of the Foundation.
@thcipriani For your consideration. I would love some help thinking about what our measurable outcomes might be for this next phase of the project. It seems likely that picking the scope/role that we want to target before committing to the hypothesis would be helpful. Then we could say something like:
There is some related information at https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_user_roles_and_rights and https://wikitech.wikimedia.org/wiki/Help:Access_policies.
@SLyngshede-WMF The draft system also seems to be missing a public audit log of block/unblock actions and an admin provided description of each action. The Wikitech/MediaWiki functionality for this is exposed by https://wikitech.wikimedia.org/wiki/Special:Log?type=block.
Has work started yet on the inverse unblock action? It has been an important principal of this functionality up to now that it is easily reversible so that we can block first and sort out details later knowing that it will be simple to restore access if the block was found to be hasty.
I would like to sponsor @theprotonade for this process so they can be a part of the Toolforge-standards-committee
I would like to sponsor @SD0001 for this process so they can be a part of the Toolforge-standards-committee
I would like to sponsor @waldyrious for this process so they can be a part of the Toolforge-standards-committee
I would like to sponsor @LucasWerkmeister for this process so they can be a part of the Toolforge-standards-committee if a new NDA signing is needed.
I would like to sponsor @Pintoch for this process so they can be a part of the Toolforge-standards-committee
I would like to sponsor @JJMC89 for this process so they can be a part of the Toolforge-standards-committee
@SLyngshede-WMF I would like to understand how I and others will be able to use this new blocking functionality when the current system has been removed from Wikitech on (planned) October 1. A quick review of https://wikitech.wikimedia.org/wiki/Special:Log?type=block shows that @taavi, @JJMC89, and myself are the most common admins to deliver blocks recently, but a longer list when looking back over the past 12 months or so of activity.
I have a mostly working solution for this issue in a custom container image created with the Toolforge build service, but it currently needs T356016: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) to be fixed to make the container easier to use.
I can at least recreate this failure in my local environment, so hopefully I can figure out how to fix it.
My local working tree is full of a half done implementation for T372059: Specify the deploy window via URL in the scheduler tool, so for now I'll just remove the bot right on wikitech.
Apparently mwclient's Page.edit marks things as bot edits automagically.
[00:04] <logmsgbot_cloud> !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 (T359641)
Stashbot in the #wikimedia-cloud-feed channel expects !log messages to start with the project to assign the log event to. This is different than the assumption in most other channels that all the logging goes to the production SAL. Some info at https://wikitech.wikimedia.org/wiki/Tool:Stashbot#!log_processing
The total tool count remaining the same over that period seems suspect. The count has increased by 2 in subsequent runs however so maybe it was just a very strange coincidence?
My initial random guess at a proximal cause would be deferred updates blocking on insert locks at the db where they hang for a long time in iowait.
Things don't seem to be quite as bad as the 2024-07-23T21:03 last update date would imply. It looks to me like the OOM crash has been happening fairly late in the crawl and that records seen before the OOM are being updated. I have deployed a config change that increases the initial cpu and ram for the job and greatly increases the hard limits. The deploy happened just after the 2024-09-12T22:00 run crashed. Let's see what happens when the 2024-09-12T23:00 run fires.
The OOMKiller reaping the crawler is unexpected. Is this container undersized? It turns out that I was very conservative in the granted resources for the cronjob's container. It is set to:
resources: limits: cpu: 100m memory: 100Mi requests: cpu: 100m memory: 100Mi
I think we should start by just giving it more RAM.
$ ssh deployment.eqiad.wmnet $ kube_env toolhub eqiad $ kubectl get cronjob NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE toolhub-main-crawler @hourly False 1 22m 555d $ kubectl get job NAME COMPLETIONS DURATION AGE toolhub-main-crawler-28053180 1/1 17m 497d toolhub-main-crawler-28769340 0/1 23m 23m $ kubectl get po NAME READY STATUS RESTARTS AGE toolhub-main-75df574bdc-6vtvw 4/4 Running 8 (56m ago) 15d toolhub-main-75df574bdc-kct8n 4/4 Running 8 (57m ago) 15d toolhub-main-crawler-28769340-fk5qn 1/2 OOMKilled 0 23m $ kubectl logs toolhub-main-crawler-28769340-fk5qn toolhub-main-crawler ... {"@timestamp":"2024-09-12T17:08:27.063Z","log.level":"error","message":"Toolinfo toolforge-authors already seen at https://toolsadmin.wikimedia.org/tools/toolinfo/v1.2/toolinfo.json","ecs":{"version":"1.7.0"},"log":{"logger":"toolhub.apps.crawler.tasks","origin":{"file":{"line":94,"name":"tasks.py"},"function":"process_url"}},"process":{"name":"MainProcess","pid":8,"thread":{"id":140367692085056,"name":"MainThread"}},"service":{"type":"toolhub"},"trace":{"id":"none"}} {"@timestamp":"2024-09-12T17:08:27.788Z","log.level":"error","message":"Toolinfo mm_wdfist already seen at https://magnustools.toolforge.org/toolinfo.json","ecs":{"version":"1.7.0"},"log":{"logger":"toolhub.apps.crawler.tasks","origin":{"file":{"line":94,"name":"tasks.py"},"function":"process_url"}},"process":{"name":"MainProcess","pid":8,"thread":{"id":140367692085056,"name":"MainThread"}},"service":{"type":"toolhub"},"trace":{"id":"none"}} /srv/app/contrib/cronjob/crawler.sh: line 30: 8 Killed poetry run python3 /srv/app/manage.py crawl --quiet /srv/app/contrib/cronjob/crawler.sh: connect: Connection refused /srv/app/contrib/cronjob/crawler.sh: line 26: /dev/tcp/127.0.0.1/1666: Connection refused
Andrew are you asking for a shared URL on the IDP side that would return a "you are logged out from service X, would you like to log in again?" sort of page? This to me feels like a client concern rather than a provider concern, but I can see how it could be convenient to have this handled centrally. That does feel like the IDP provider side would need to know more about the workflows of the client applications it supports.
Before:
>>> import k8s.client >>> images = k8s.client.get_images(cached=True) >>> for image in images["items"].keys(): ... repo, name_tag = image.split("/", 1) ... if ":" not in name_tag: ... print(image, name_tag) ... tools-harbor.wmcloud.org/tool-sample-complex-app/frontend tool-sample-complex-app/frontend >>>
I have just emailed the Wikimedia Foundation's Trust and Safety team and asked them to inform me if anyone on the nomination list should be deemed ineligible for this trusted functionary role based on existing issues known to them.
I have just emailed the members of the admin tool to solicit their feedback on the candidates as the method for checking that these 8 individuals are all "active members in good standing of the Toolforge community." The Toolforge admins now have until 2024-09-12 end of day anywhere on earth (2024-09-13T12:00Z) to provide their feedback directly to me. Once the feedback period is closed I will inform this admins and the nominees who is invited to continue on to the final membership hurdle, signing the Volunteer NDA with WMF Legal.