EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
NLP Code Snippets and Conference related
WordPiece Tokenizer for BERT models.
Word/Image/Audio Embedding models, Tokenizer models, Ngram language models, MatrixModels, Corpus building, Vocabulary Building, Language modelling
This repository contains my coursework (assignments & semester exams) for the Natural Language Processing course at IIIT Delhi in Winter 2025.
Large Language Model Tokenizers from scratch
Add a description, image, and links to the wordpiece-tokenization topic page so that developers can more easily learn about it.
To associate your repository with the wordpiece-tokenization topic, visit your repo's landing page and select "manage topics."