⚓ T176336 Deploy drafttopic model to production ORES

	Subject	Repo	Branch	Lines +/-
	Provision the drafttopic model	mediawiki/services/ores/deploy	master	+14 -1

Status	Assigned	Task
Resolved	None	T176324 Scoring platform team FY18 Q2
Resolved	Halfak	T183198 Scoring Platform FY18 Q3
Resolved	awight	T176336 Deploy drafttopic model to production ORES
Resolved	Halfak	T123327 Train/test draft topic model (new article routing AI)
Resolved	Sumit	T172321 Build mid-level WikiProject category training set
Resolved	Sumit	T172325 Efficient method for mapping a WikiProject template to the WikiProject Directory
Resolved	Sumit	T172326 Create machine-readable version of the WikiProject Directory
Resolved	Sumit	T175037 Publish Machine-Readable WikiProjects Dataset
Resolved	Sumit	T179311 Generate mid-level WikiProject categories
Resolved	Sumit	T181166 Revscoring: Statistic for multilabel classification
Resolved	Halfak	T181163 Revscoring tune does not recognize a set of labels as target
Resolved	awight	T192293 Check drafttopic model memory usage
Resolved	None	T194295 Phabricator repo is not mirroring to gerrit as hoped
Resolved	awight	T194322 Drafttopic breaks ?features parameter
Resolved	awight	T195593 Investigate memory issues surrounding trial drafttopic deployment
Duplicate	None	T181855 scap support for git-lfs
Resolved	• mmodell	T180627 Support git-lfs in scap
Resolved	awight	T180628 Install git-lfs client (at least on scap targets & masters)
Declined	None	T182085 Connect Phabricator to swift for storage of git-lfs and file uploads.
Resolved	• mmodell	T192042 Create gerrit mirrors for all github-based ORES repos
Resolved	fgiunchedi	T192124 Deploy Scap 3.8.0 to production

Halfak created this task.Sep 20 2017, 4:55 PM

Restricted Application added a project: Machine-Learning-Team. · View Herald TranscriptSep 20 2017, 4:55 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Halfak added a subtask: T123327: Train/test draft topic model (new article routing AI).Sep 20 2017, 4:55 PM

Halfak added a parent task: T176324: Scoring platform team FY18 Q2.

Halfak triaged this task as Medium priority.Oct 9 2017, 9:13 PM

Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

Halfak added a parent task: T183198: Scoring Platform FY18 Q3.Dec 18 2017, 10:21 PM

Halfak moved this task from Parked to Non-Epic on the Machine-Learning-Team (Active Tasks) board.Feb 26 2018, 3:58 PM

Halfak renamed this task from Deploy topic model to ORES to Deploy drafttopic model to ORES.Apr 16 2018, 2:26 PM

Halfak closed subtask T123327: Train/test draft topic model (new article routing AI) as Resolved.

Halfak added a project: drafttopic-modeling.

MMiller_WMF subscribed.Apr 18 2018, 11:48 PM

Restricted Application added a project: Product-Analytics. · View Herald TranscriptApr 18 2018, 11:48 PM

• Phabricator_maintenance removed a project: Product-Analytics.Apr 19 2018, 12:22 AM

awight moved this task from Non-Epic to Pending deployment on the Machine-Learning-Team (Active Tasks) board.May 8 2018, 9:51 PM

Change 432000 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] [WIP] Provision the drafttopic model

https://gerrit.wikimedia.org/r/432000

gerritbot added a project: Patch-For-Review.May 8 2018, 9:51 PM

awight mentioned this in rORESDEPLOY39a752242e22: [WIP] Provision the drafttopic model.May 8 2018, 10:26 PM

awight added a subtask: T194295: Phabricator repo is not mirroring to gerrit as hoped.May 9 2018, 6:29 PM

Paladox closed subtask T194295: Phabricator repo is not mirroring to gerrit as hoped as Resolved.May 9 2018, 8:00 PM

awight mentioned this in rORESDEPLOY7064288c4f96: Provision the drafttopic model.May 9 2018, 8:52 PM

awight mentioned this in rORESDEPLOYbcb6b18348c7: Provision the drafttopic model.May 9 2018, 9:01 PM

awight mentioned this in rORESDEPLOYc0db10224d57: Provision the drafttopic model.May 9 2018, 9:31 PM

Change 432000 merged by Awight:
[mediawiki/services/ores/deploy@master] Provision the drafttopic model

https://gerrit.wikimedia.org/r/432000

awight added a subtask: T194322: Drafttopic breaks ?features parameter.May 10 2018, 1:19 AM

awight closed subtask T195593: Investigate memory issues surrounding trial drafttopic deployment as Resolved.May 28 2018, 11:19 AM

awight renamed this task from Deploy drafttopic model to ORES to Deploy drafttopic model to production ORES.May 28 2018, 11:26 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:31:40Z] <awight@deploy1001> Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:34:14Z] <awight@deploy1001> Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (duration: 02m 35s)

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:40:10Z] <awight@deploy1001> Started deploy [ores/deploy@d77e52c]: l ores2001.codfw.wmnet ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:40:15Z] <awight@deploy1001> Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:40:38Z] <awight@deploy1001> Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f (duration: 00m 22s)

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:48:39Z] <awight@deploy1001> Started deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision)

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:50:26Z] <awight@deploy1001> Finished deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision) (duration: 01m 47s)

awight added a subtask: T181855: scap support for git-lfs.Jun 4 2018, 9:03 PM

awight closed subtask T192293: Check drafttopic model memory usage as Resolved.Jun 4 2018, 9:28 PM

awight added a subtask: T180627: Support git-lfs in scap.Jun 6 2018, 6:32 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T10:15:14Z] <awight@deploy1001> Started deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336

Mentioned in SAL (#wikimedia-operations) [2018-06-06T10:15:21Z] <awight@deploy1001> Finished deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 00m 06s)

Mentioned in SAL (#wikimedia-operations) [2018-06-06T10:15:41Z] <awight@deploy1001> Started deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336

Mentioned in SAL (#wikimedia-operations) [2018-06-06T10:19:25Z] <awight@deploy1001> Finished deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 03m 44s)

Drafttopic code is live on ores2002, the word2vec file has cloned successfully after applying a workaround. I'll monitor memory for a few hours, it's taken an expected dip and will hopefully stabilize.

ores2002 looks very much like ores2001 and seems to be in a stable garbage-collection cycle. Minimum headroom has dropped from c. 20GiB to 14GiB. CPU and other metrics are unchanged, however "scores errored" went from 0.005/min to 0.5 or so, which is annoying.

Screenshot from 2018-06-06 14-32-31.png (714×1 px, 107 KB)

I've run the workaround creating ~deploy-service/.gitconfig on all production hosts.

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:01:08Z] <awight@deploy1001> Started deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:26:45Z] <awight@deploy1001> Finished deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336 (duration: 25m 37s)

awight closed this task as Resolved.Jun 6 2018, 3:28 PM

awight claimed this task.

awight moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.

awight closed subtask T194322: Drafttopic breaks ?features parameter as Resolved.Jun 7 2018, 1:39 PM

awight closed subtask T180627: Support git-lfs in scap as Resolved.Jun 12 2018, 3:00 PM

• Vvjjkkii reopened subtask T195593: Investigate memory issues surrounding trial drafttopic deployment as Open.Jul 1 2018, 1:08 AM

• Vvjjkkii reopened subtask T194322: Drafttopic breaks ?features parameter as Open.Jul 1 2018, 1:10 AM

• Vvjjkkii reopened subtask T194295: Phabricator repo is not mirroring to gerrit as hoped as Open.

CommunityTechBot closed subtask T194322: Drafttopic breaks ?features parameter as Resolved.Jul 1 2018, 2:38 PM

Vachovec1 closed subtask T194295: Phabricator repo is not mirroring to gerrit as hoped as Resolved.Jul 1 2018, 4:37 PM

CommunityTechBot closed subtask T195593: Investigate memory issues surrounding trial drafttopic deployment as Resolved.Jul 2 2018, 4:22 AM

Aklapper mentioned this in T360990: drafttopic has two issue trackers.Mar 26 2024, 10:36 AM