tokenizers: Fast, Consistent Tokenization of Natural Language Text

Canonical link: https://rscholar.com/pkg/tokenizers

Lincoln Mullen • Os Keyes • Dmitriy Selivanov • Jeffrey Arnold • Kenneth Benoit • View description and downloads View dependencies GitHub project

2016 Published

0 Citations

5 Authors

8 Revisions

0.3.0 Version

MIT License

Referenced by ⇅	Year
textfeatures: Extracts Features from Text (Version 0.3.3)	2018
WhatsR: Parsing, Anonymizing and Visualizing Exported 'WhatsApp' Chat Logs (Version 1.0.6)	2023
DramaAnalysis: Analysis of Dramatic Texts (Version 3.0.2)	2019
WhatsR Logs: (Version )	0
blocking: Various Blocking Methods for Entity Resolution (Version 1.0.3)	2025
covfefe: Political Linguistics Toolkit - Gaffes, Phonetics, and Text Transformation (Version 1.0.0)	2017
deeplr: Interface to the 'DeepL' Translation API (Version 2.1.0)	2018
easieRnmt: Local Machine-Translation with 'EasyNMT' from 'R' (Version 0.0.5)	2026
edgarWebR: SEC Filings Access (Version 1.1.0)	2017
llmshieldr: Safety Guardrails for Large Language Model Workflows (Version 0.1.0)	2026
pdfsearch: Search Tools for PDF Files (Version 0.4.3)	2016
pkgmatch: Find R Packages Matching Either Descriptions or Other R Packages (Version 0.5.4)	2026
proustr: Tools for Natural Language Processing in French (Version 0.4.1)	2017
quanteda: Quantitative Analysis of Textual Data (Version 4.5.0)	2015
rslp: A Stemming Algorithm for the Portuguese Language (Version 0.2.0)	2016
rwig: Wasserstein Index Generation (WIG) Model (Version 0.2.0)	2026
textrecipes: Extra 'Recipes' for Text Processing (Version 1.1.0)	2018
tidypmc: Parse Full Text XML Documents from PubMed Central (Version 2.0)	2019
tidytext: Text Mining using 'dplyr', 'ggplot2', and Other Tidy Tools (Version 0.4.3)	2019
torchdatasets: Ready to Use Extra Datasets for Torch (Version 0.3.2)	2021
wactor: Word Factor Vectors (Version 0.0.1)	2019

RPKG Scholar presents a tabulation of an author's contribution in the development of R packages stored in the Comprehensive R Archive Network (CRAN). Within this site, we consider package dependencies (suggests,imports,depends,enhances) as citations because we believe that using one's package to develop another is tantamount to citing the author of the package being imported, suggested or enhanced.