@bking - thanks for the hint i'll try it out on the upcoming imports

Do you know a tool that will track CPU and Mem load of a server over time (on an Ubuntu server)?
Currently we have 3% Memory of 512 GB and 300% CPU Load (of 16 cores) while the indexing runs. Hannah Bast reported the memory needs are not at 128 GB any more these days but the peak i do not know. If we want to go try and error we could go with the current max and run things for a test.

@Harej
thank you for pointing out your infrastructure availabilty which is sort of already "priced" in in the calculation. We are seeking a third mirror environment sponsored by the Mediawiki foundation. As for the technical aspects: https://wiki.bitplan.com/index.php/Wikidata_Import_2024-10-17

it's 4 TB of SSD and 128 GB RAM for starters which will give you a single QLever instance to be indexable. If you want multiple instances e.g. QLever and blazegraph in parallel the footprint for indexing gets bigger. The runtime requirements are lower so it might be possible to just get the index result over but that would not be the idea of the game.

Hurray! Can we make [[ https://github.com/WolfgangFahl/snapquery | snapquery ]]an official middlware infastructure ? I assume that would be a separate ticket again.

The snapquery concept https://github.com/WolfgangFahl/snapquery with demo at https://snapquery.bitplan.com/ will happily support much larger queries and given them a proper Persistent identifier consiting
of Domain/Namespace/Name and accept meta info such as title, url and other details such as purpose or rating.

Given the upcoming graphsplit this issue is getting even more urgent. https://github.com/WolfgangFahl/snapquery has been developed at Wikimedia Hackathon May 2025 Tallinn to mitigate the issue. The Wikimedia foundation currently does not actively participate in this infrastructure but we hope this will happen soon.

see also https://github.com/fnielsen/scholia/issues/1439

Any action yet ?

One special case of this is publications of proceeedings of scientific conferences. I am intending to have a wikidata backend for my Proceedings Title Parser at ptp.bitplan.com anyways so i could add an "ISBN" input mode.

thx for pointing this out so my issue is linked to the wrong phabricator task. Is there already one for the "zombie" case i am experiencing where i can't delete a page?

https://www.openresearch.org/wiki/Undeletable/EUROCRYPT_2017 is an example for the problem.

@Xqt - thx for the reply and asking - i am fine with this schedule - thank you very much.

Thx. for the explanation. And no - testing the script just now with just the config2.py changed will be tough since the three CI pipelines i am using are all pulling directly from PyPi. For my local environment it would be sort of feasible but i fear the installation routines would override my local changes as they did when i was trying things out for almost a day recently. My current project priorities are also different -i need to make sure i don't have showstoppers and this one isn't any more.

looks like avoiding the exception
I do not know how to use the review tool. I logged in but have no clue what to do. I tried edit/marked as reviewed.

I am happy with my workaround for the time being. I also intend to change my app to a webbased app and use py2app to create an app that only communicates with the webbased app. That way i'll avoid any similar problems. Still to fix things upstream the default should be not to throw an exception but simply log the problem by default. So os.listdir might be wrapped in a try catch blog. Only if configured to specifically show start up issues should there be an exception.

The sample

% python
Python 3.8.8 (default, Mar 18 2021, 06:01:57) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from wikibot import wikibot
>>>

@Xqt https://github.com/WolfgangFahl/py-3rdparty-mediawiki/issues/43 has the issue about pycrypto...

see https://github.com/WolfgangFahl/py-3rdparty-mediawiki/commit/8ab0bd846d191fa93cfd661c89e2bc769d69f06b for trying this workaround

If you then do a

import pywikibot

I still have to use the awkward work around

import os
os.environ["PYWIKIBOT_NO_USER_CONFIG"]="1"
import pywikibot

With Mediawiki 1.35 the situation got worse. Now the code from my own github repository https://github.com/WolfgangFahl/S5SlideShow doesn't work any more and i didn't even follow up on the integration in the in mediawiki world yet. http://wiki.bitplan.com/index.php/S5SlideShow tries to describe the issues before the 1.35 LTS changes that broke the extension again.

I am sorry - i did not record the message. Today I tried with HitCounters-REL1_35-52da1b7.tar.gz and the problem didn't show up.

Happened e.g. today when trying to install extension HitCounters on MediaWiki 1.35.

See https://cr.bitplan.com/index.php/Main_Page for the example data shown here.

@Yaron_Koren - i'll certainly try the explanation and add more examples to make my point. As far as i understand the Form definition approach the only way to handle multiple instances is currently explained in https://www.mediawiki.org/wiki/Extension:Page_Forms/Defining_forms#Multiple-instance_templates. For this to work the context of these multiple instances need to be known in advanced. So if i have pages of type:

• Open Source Project
• Git Respository
• Internal Project

...
and I'd love to use the "commit" section in all these i'd need to repeate the definition in the forms.

Could partial forms be made working by putting the definition into a standalone context - e.g. defining a form page as partial and having the form definition be picked up automatically when a fitting macro/WikiSon notation is found?

I have never tried https://www.mediawiki.org/wiki/Extension:Page_Forms/Defining_forms#Partial_forms - does this feature still work?

Before getting rid of Page Forms all together I was thinking of we could have some improvements that would lead to a "workaround" for a start.

Hi, @Yaron_Koren - thank you for looking into this.

@Xqt - thanks for trying again. I still have some project based on pywikibot although I am trying to avoid it and use mwclient more these days due to this limited useability in intranet situations.

It's not clear for me how to open a "documentation" ticket from what https://www.mediawiki.org/wiki/How_to_report_a_bug has to offer. I searched for the keyword "documentation" in the page and couldn't find that keyword. What should i search for ? I never created a documentation ticket before and don't know what it looks like.

I don't know whether this a documentation problem or a bug. It's clearly an unexpected change in behavior that makes the system unusable without any warning i'd know of e.g. in a migration guide. So there are two options:

• fix the behavior
• change the documentation

IMHO it's a valid question and might be a bug if there is no proper documentation for answering the questions. I could't find the proper documentation that describe why the behavior changed and what the reason for this is.

The workaround is

# make sure we don't use "pretty urls"
$wgUsePathInfo=false;

https://github.com/WolfgangFahl/py-3rdparty-mediawiki/issues/35 will have to wait for the release on pypi see https://pypi.org/project/pywikibot/

looks great. How do i get rid of the "family and mylang ar not set ..." messages?

Wouldn't this be a good opportunity to generally add a link to a FAQ Wiki per error-message? I feel that following a link per error-message gives a lot more options for discussing individual situations in which an error might appear, linking to related Phabricator tasks and so on.

It happens when I try to change the protocol for accessinga wiki. What I try is to change the apache settings:

<VirtualHost *:80>
  ServerName server.domain.com
  Redirect permanent / https://server.domain.com/
</VirtualHost>
<VirtualHost *:443>
  ....
  ServerName server.domain.com 
</VirtualHost>

For the extensions https://www.mediawiki.org/wiki/Extension:Diagrams and https://www.mediawiki.org/wiki/Extension:Piwo the approach discussed here might be very useful.

same on my main wiki - showed up after some 3 weeks of migration to 1.35 LTS - it is a showstopper! Why is this low priority if it makes the wiki completly unusable?

Unfortunately https://www.mediawiki.org/wiki/API:Login is inconsistent. E.g. the python example uses login as an example but the text says you shouldn't use it any more but use clientLogin instead. I could not find sufficient information on the cookie issue either.

E.g. how would a curl access with cookie work?

Opening up the wiki for public read access:

One of the issues seems to be that I don't know how to get rid of the "You need read permission to use this module." messsage on MediaWiki1.35 so that things would at least work for my public wikis.

curl -s "http://test.bitplan.com/api.php?action=query&meta=siteinfo&siprop=general|namespaces|namespacealiases|statistics&format=json" | jq .

{
  "error": {
    "code": "readapidenied",
    "info": "You need read permission to use this module.",
    "*": "See http://test.bitplan.com/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes."
  }
}

If you try to debug the situation and click on "site" in the loginmanager a sequence of API call is initiated ... that makes it even harder to find out what is going on ...

If checking the version is the issue please user the information from the family file (and trust it) instead of trying to find out via Mediawiki API doing an http request before being logged in ...

There is a

WARNING: API error readapidenied: You need read permission to use this module.

warning with no details on what the API called is. When debugging it reveals that a meta=siteinfo is tried and if that fails a login is performed. This doesn't make any sense if a proper user/password combination is given. Why try to use the API - how can this be switched off because of course you can't use the API if you are not logged in ...

What is the proper procedure to tell pywikibot the password for login via API ? I am using

# needs patch as outlined in https://phabricator.wikimedia.org/T248471
           #self.site.login(password=self.getPassword())
           lm = LoginManager(password=self.getPassword(), site=self.site, user=self.user)
           lm.login()

and i get a WARNING:

With pywikibot 5.0.0 i have to start all over again ... sigh ..

The same secondary message might appear in other contexts. In my case it is:

I can't see where the problem is.

I tried debugging the line 595492 mentioned in the error message. See my stackoverflow answer to How to get few lines from a .gz compressed file without uncompressing

For me it's a showstopper. I'd love to work on https://github.com/somnathrakshit/geograpy3/issues/15 and currently I have the following options to run the SPARQL query for it

I retried the download which took another 16h but the problem persists

I am having a similar issue see http://wiki.bitplan.com/index.php/WikiData_Import_2020-09-11#Issue_2

tried

git pull 
mvn package

today and problem persists

Still the message could be improved into "File missing or config error" and having a pointer to the FAQ and may be such a checking code or extension.

The issue here is the undecipherable error message. Since the invalid status of this task is only based on the version MW 1.33.4 that i was trying things out on is EOL. It's not clear whether the error message has been improved so that "normal" people would understand what the real problem is in more recent versions of MW. I tried out different git states yesterday and always go the same error message so that i assume there is a "real" issue behind this task. It's just that the rules within MediaWiki foundation are such that the followup will not happen immediately for this task as it's reported based on a MW 1.33.4 experience. So i think it's good to have this task for reference so that other users that might run into a similar situation can reference it. Even the hint for the workaround https://www.mediawiki.org/wiki/Extension:SimpleMathJax - was a bit hidden in the documentation.

According to https://www.mediawiki.org/wiki/Manual:Math the work-around is to install https://www.mediawiki.org/wiki/Extension:SimpleMathJax which i did successfully.

I am not asking for a solution but a work-around. And it would be good if such situations could be avoided in the future by proper impact analysis which extensions break on API changes and not doing API changes with too much impact.

see https://phabricator.wikimedia.org/T224854
and work around

Bawolff asked on project support desk:
How would mediawiki distinguish between the config being wrong and the file just happening to be missing?

work around:

delete from logging where log_user_text="";

Maybe this could be a step in the update.php file to check that there is a reasonable file configuration if lots of files are pointed to in the SQL file but do not properly resolve ..

Is this a request to edit https://www.mediawiki.org/wiki/Manual:Errors_and_symptoms#Image_Thumbnails_not_working_and/or_appearing ? - yes thats's part of the issue and I added a section.

The issue is in the confusing message "File missing" - it's not true that the file is missing. The configuration is wrong and the pointer to the directory of files is not configured correctly. For a single such message it would not be a problem but if in "browse files" every single image shows the same misleading message "File missing" than that is an issue which should be fixed to allow administrators to get informed about the problem by their users much more quickly and more detailed than is the case right now.

same here ...

Please reopen this issue - marking it invalid since it's not reproducible in the Wikimedia Universe is not helpful for those having the problem in their own environment .

You trial was obviously on Windows and not Ubuntu 18.04 LTS