Archive of Samourai Wallet Blog/OXT Research Blog (WARC format)

Example of using warcutils with Apach Spark

Transform stream to read .warc or .warc.gz file member by member in nodejs

A search engine, but currently a filtering pipeline for WARC files. Legacy repo, look for abracabra repo.

Eventually a search engine, but currently a filtering pipeline for HTML and soon WARC files.

ES6 Class to read .warc or .warc.gz file member by member in nodejs

WARC archive crawler

This library is a very lightweight client to Common Crawl's WARC files.

ANS-108 implementation for the Archive The Web. This allows gateways on arweave to render ATW's warc files

Convert WARC files into Avro for big data processing

A tool to fetch binary content for tweets

Vim syntax highlighting for WARC files

Hadoop streaming EMR job

WARC parser for ocaml

CLI to extract named entities from web pages and link them to potential entity candidates in the WikiData knowledge base.

This is an old version of the WARC patches. Wget v1.14 and higher has WARC support.

A more compressed alternative to WARC web archival. Command line tool built in Kotlin.

This system evaluates a series of mementos (archived web pages) to determine which are off topic. The series can be part of an Archive-It collection, a single TimeMap, or stored in a WARC file.

Partition (W)ARC Files by MIME Type and Year

pull-streaming WARC file operations