Sorry, but whatever was the problem is fixed, it's working now. Feel free to close this. Thanks.

I spoke too soon. It's happening again.

Sometimes, when I hit the "Submit query" button two or three times after no success, the query runs. Other times, no luck at all. "Intermittent" is a good description, I agree.

But sometimes I have to hit "Submit" 10 of 15 times before it runs.

Sometimes I'm getting the message with a slightly different IP number:

*Access denied for user 'quarry'@'172.16.2.252' (using password: NO)

I always get this error if I enter a ToolsDB database as the db name (e.g. s55771__wsstats_p).

If you want to see an example of this, try Category redirects and look at the history.

I could never get this one to run. I use to be able to submit a dozen or more times and the query would eventually run. No more. It looks like this is happening with other editors (see Recent queries list ) so how can it be resolved?

Thanks to whomever fixed this problem.

I suspect they are related yes. Maybe Quarry is trying to connect to another database but using ToolsDB credentials, or viceversa.

Ah well, looks like the deployment of https://github.com/toolforge/quarry/pull/40 didn't fully go through. The keys TOOLS_DB_USER and TOOLS_DB_PASSWORD are missing in config.yaml on the pods, although I could have sworn this wasn't the case the last time around when I looked into it.

The default values are empty. So quarry is trying to connect to ToolsDB with no password and username defaulting to the server username ("quarry"). The (using password: NO) in the error message checks out.

But this doesn't explain why the issue occurs when querying the wiki replicas.

Sorry to post a duplicate notice. I couldn't find this one when I wanted to post about this bug.

Well, for the last 21 hours no querry could run. So it's pretty bad.

Ah well, looks like the deployment of https://github.com/toolforge/quarry/pull/40 didn't fully go through. The keys TOOLS_DB_USER and TOOLS_DB_PASSWORD are missing in config.yaml on the pods

Good spot! I think I modified the wrong file, this one should fix it: https://github.com/toolforge/quarry/pull/46

siddharthvp opened https://github.com/toolforge/quarry/pull/47

In T365374#9880226, @SD0001 wrote:

But this doesn't explain why the issue occurs when querying the wiki replicas.

Figured out the cause. I've filed https://github.com/toolforge/quarry/pull/47.

https://github.com/toolforge/quarry/pull/46 and https://github.com/toolforge/quarry/pull/47 should probably fix the issue, but I'm not sure how to deploy those after they are merged.

I found the following instructions at https://github.com/toolforge/quarry/blob/main/README.md#deploying-to-production

From the quarry-bastion in the git checkout that has the state file. bash deploy.sh mysql -uquarry -h <trove hostname created in last step> -p < schema.sql In horizon point the web proxy at the new cluster.

I'm not sure either about how to deploy in the new k8s-based setup. However, /home/rook/quarry seems to be the git checkout referenced above.

I'm confused because that directory has a checkout of a non-main branch:

root@quarry-bastion:/home/rook/quarry# sudo -u rook git status
On branch T362213-queued

root@quarry-bastion:/home/rook/quarry# sudo -u rook git log --oneline |head
9fe166f auto update of  tag
573b9f0 Do not offer stop button on queued
c417f9f Add ToolsDB credentials (#40)
49862cc Tofu state to object store (#39)
dc3e220 Update helm on pr (#33)
5b2ff73 quarry on k8s (#31)
01d3c4a improve grammar of synchronization delay message (#32)
312990c minikube helm chart (#28)
f866d1f Link to new filename to make VM happy (#30)
03900ff Migrate from uwsgi to gunicorn + make dockerfile suitable for prod (#29)

siddharthvp closed https://github.com/toolforge/quarry/pull/47

FYI: runs in the last hour are passing now

@rook redeployed Quarry including the latest fixes https://github.com/toolforge/quarry/pull/46 and https://github.com/toolforge/quarry/pull/47.

Some of my queries are running okay now, others are just spinning their wheels, like they are running but they never conclude or take an unusual length of time to do so. So, a partial fix. And with one query, I get an error message

"Unknown column 'pl_title' in 'field list'"

which might mean that there is suddenly a code problem for some reason.

In T365374#9886036, @Liz wrote:

Some of my queries are running okay now, others are just spinning their wheels, like they are running but they never conclude or take an unusual length of time to do so. So, a partial fix. And with one query, I get an error message

"Unknown column 'pl_title' in 'field list'"

which might mean that there is suddenly a code problem for some reason.

@Liz this this is a separate issue that was a planned change see https://meta.wikimedia.org/wiki/Tech/News/2024/20 (i missed it as well so also have broken queries)

I'm not getting that message any more but some queries are just running endlessly and never finishing. I had one that ran for more than 3 hours before I finally cancelled it. Does this warrant a new case to be opened?

@Liz are the queries that never finish different from other queries, or are they similar but sometimes they randomly fail? Did similar queries work fine in the past?

If this keeps happening, please open a new task, adding an example of a query that is failing to complete.

In T365374#9890676, @Liz wrote:

I'm not getting that message any more but some queries are just running endlessly and never finishing. I had one that ran for more than 3 hours before I finally cancelled it. Does this warrant a new case to be opened?

See T367654.