Maniphest T190457

Closed, ResolvedPublic
Actions

Assigned To

Authored By

	abian
	Mar 22 2018, 9:16 PM

Description

Please include text files with the hash values of the future entity dumps in https://dumps.wikimedia.org/wikidatawiki/entities/ in order to check data integrity. These files could be similar to the *sums.txt ones in https://dumps.wikimedia.org/wikidatawiki/latest/.

Details

	Subject	Repo	Branch	Lines +/-
	Add checksums for Wikidata entity dumps	operations/puppet	production	+12 -0

Customize query in gerrit

Event Timeline

abian created this task.Mar 22 2018, 9:16 PM

@hoo, can you fold this into the bash script without too much work?

In T190457#4079955, @ArielGlenn wrote:

@hoo, can you fold this into the bash script without too much work?

Piece of cake (I guess)… so yes, will schedule this for the week.

Would we want one hash sum file per (dated) folder, or one for everything? Or both?

If one for everything, should it contain just the base file names (like wikidata-20180323-truthy-BETA.nt.bz2), or the relative path (like 20180323/wikidata-20180323-truthy-BETA.nt.bz2).

One per folder I suppose, so that as a particular run finishes up, the hash info is available.

That's also the easiest option for users, I think.

Change 423353 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Add checksums for Wikidata entity dumps

https://gerrit.wikimedia.org/r/423353

gerritbot added a project: Patch-For-Review.Apr 2 2018, 1:12 AM

hoo claimed this task.Apr 2 2018, 1:14 AM

hoo added a project: Wikidata-Ministry-Of-Magic.

hoo moved this task from Tasks to Needs Review on the Wikidata-Ministry-Of-Magic board.

Change 423353 merged by ArielGlenn:
[operations/puppet@production] Add checksums for Wikidata entity dumps

https://gerrit.wikimedia.org/r/423353

• Jonas moved this task from Needs Review to Done on the Wikidata-Ministry-Of-Magic board.Apr 3 2018, 7:14 AM

hoo removed a project: Patch-For-Review.Apr 3 2018, 10:46 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-05T12:04:22Z] <hoo> Manually back-filled hashes for the Wikidata JSON dumps in https://dumps.wikimedia.org/wikidatawiki/entities/20180402/wikidata-20180402-*sums.txt (T190457)

First checksums are available: https://dumps.wikimedia.org/wikidatawiki/entities/20180402/wikidata-20180402-md5sums.txt and https://dumps.wikimedia.org/wikidatawiki/entities/20180402/wikidata-20180402-sha1sums.txt for https://dumps.wikimedia.org/wikidatawiki/entities/20180402/.

I manually added the JSON checksums, but the RDF ones were automatically added. I'll check next week to make sure this also correctly works for the JSON checksums, but I don't expect any surprises there.

Thank you!

JSON checksums look fine as well:

hoo@snapshot1007:/mnt/dumpsdata/otherdumps/wikibase/wikidatawiki/20180409$ md5sum -c wikidata-20180409-md5sums.txt 
wikidata-20180409-all.json.gz: OK
hoo@snapshot1007:/mnt/dumpsdata/otherdumps/wikibase/wikidatawiki/20180409$ sha1sum -c wikidata-20180409-sha1sums.txt
wikidata-20180409-all.json.gz: OK

Envlh awarded a token.Apr 16 2018, 3:03 PM

ArielGlenn moved this task from Backlog to Done on the Dumps-Generation board.May 8 2018, 7:49 AM

Include checksums in https://dumps.wikimedia.org/wikidatawiki/entities/Closed, ResolvedPublicActions

Description

Details

Event Timeline

Include checksums in https://dumps.wikimedia.org/wikidatawiki/entities/
Closed, ResolvedPublic
Actions