Natural language tokenization and text processing libraries

Research date: 2026-10-09

This report selects 25 GitHub repositories for studying the engineering of natural-language tokenization and adjacent text processing: subword encoders, sentence boundaries, dictionary and morphological segmentation, Unicode normalization and boundaries, stemming, and text repair. General NLP frameworks are included only where their tokenization or text representation subsystem provides a substantial implementation to study. These are evidence-grounded engineering judgments, not a claim that every component is exemplary or a recommendation to adopt every project.

Criteria used throughout:

• C1 — Difficult correctness: meaningful invariants, concurrency, numerical semantics, adversarial inputs, or failure handling.
• C2 — Reusable abstractions: substantial interfaces and representations that support different applications or implementations.
• C3 — Performance with structure: concrete resource constraints addressed through understandable architecture.
• C4 — Sustained evolution: evidence across years of compatibility work, testing, or complexity management; age alone does not qualify.

Subword encoders and tokenizer runtimes

1. huggingface/tokenizers

Language/role: Rust with language bindings; configurable tokenization pipelines and subword models.

Study how normalization, pretokenization, model application, postprocessing, and decoding form distinct stages. The component documentation explains why text transformation needs alignment metadata and why an early split constrains what the model may subsequently merge. This is useful architecture for an engineer building interchangeable tokenization strategies rather than one fixed encoder.

• C1: Normalizers track alignment to the original text, while pretokenizers establish boundaries that the model cannot cross. These are observable semantic contracts, especially when normalization changes string length. Component documentation
• C2: Normalizer sequences and separate model/stage interfaces support different pipelines. The repository's current architecture additionally separates encoding and training responsibilities. Components, repository architecture and transition notes

Entry points: the component guide above and the repository's workspace overview. Version caveat: the inspected default-branch README describes a 1.0 release-candidate transition from 0.23 and lists features still being restored. Do not assume the documented established pipeline API and every current-main capability are identical.

2. google/sentencepiece

Language/role: C++ with bindings; trainable unigram and BPE segmentation from raw text.

The unigram implementation is especially instructive because inference, alternative segmentations, and sampling share a lattice while requiring different algorithms. Its normalization layer also makes the distinction between preserving a tokenized representation and preserving the original input explicit.

• C1: The lattice implements Viterbi, probability accumulation, and alternative-path search. Numerically careful sampling and bounded search behavior address problems that a simple greedy splitter never encounters. Unigram implementation
• C3: An optimized encoding path keeps best-path information without constructing every candidate node; alternative-path search limits its agenda on difficult inputs. These choices expose the tradeoff between richer output and memory/work. Implementation
• C2: Built-in and custom normalization mappings are independent configuration choices. Custom rules use leftmost-longest matching, and identity normalization is available. Normalization guide

Entry points: the unigram source and normalization guide above. Whitespace markers do not imply lossless recovery of arbitrary original text: the default normalizer changes text, and the guide documents limitations relative to complete NFKC handling.

3. openai/tiktoken

Language/role: Rust core and Python API; byte-oriented BPE encoding.

Study how a small public encoding abstraction contains vocabulary ranks, regex pretokenization, and separately controlled special tokens. The Rust comments are unusually useful explanations of performance choices, including rejected cache designs.

• C1: The Python API distinguishes permitted special tokens, forbidden special-token spellings, and ordinary text. Its unstable-prefix API specifies relationships between a stable token prefix and possible completions; handling incomplete Unicode and surrogate-containing input requires additional care. Python encoding API implementation
• C3: The core discusses regex scratch-space contention, cloned regex instances, and replacement of a lock-protected cache with a vocabulary-based approach. These are concrete concurrency and hot-path design decisions rather than unsupported speed claims. Rust core

Entry points: tiktoken/core.py and src/lib.rs above. The prefix-completion API explicitly describes itself as unstable; treat its contract separately from ordinary encoding.

4. microsoft/BlingFire

Language/role: C++ finite-state text processing compiler/runtime with bindings; word, sentence, and model-specific subword tokenization.

The distinctive study opportunity is compilation: tokenizer rules and vocabularies become packed finite-state artifacts consumed by a runtime. The BERT model-building walkthrough connects vocabulary IDs, lexical rules, normalization maps, and debugging of word/subtoken boundaries.

• C2: Model-specific behavior is represented by loadable binary data and a common runtime interface. The compiler walkthrough shows character mapping and lexical rules as separable model ingredients. BERT model-building guide
• C3: Combining tokenization and dictionary lookup in a finite-state machine moves work into compilation. The repository describes stateless model functions that can share models across threads; the engineering article explains the production latency motivation. Repository runtime description, Bing engineering article

Entry points: the model-building guide above and the runtime library tree. No numerical benchmark claim is needed to appreciate the compiler/runtime boundary.

5. dotnet/machinelearning

Language/role: C#; specifically the Microsoft.ML.Tokenizers subsystem, not the entire ML.NET monorepo.

This subsystem offers a useful managed-runtime counterpart to Rust/C++ tokenizer libraries. Its shared API handles encoding, counting, bounded tokenization, decoding, and the relationship between original and normalized text.

• C1: Bounded encoding reports characters consumed and normalized text; token-boundary index operations must refer to the correct text representation. Span-based decoding reports consumed IDs, written characters, and failure/status information rather than assuming an unlimited destination. Tokenizer abstraction source
• C2: An abstract tokenizer separates normalization and pretokenization from model implementations, with common operations spanning BPE and model-family tokenizers. Package documentation
• C3: ReadOnlySpan<char> overloads, pooled character arrays, and explicit output-buffer management expose allocation-aware implementation choices within the public abstraction. Tokenizer source

Entry points: the two files above, within the verified Microsoft.ML.Tokenizers source directory. This monorepo is counted once.

General text representations and sentence processing

6. nltk/nltk

Language/role: Python NLP toolkit; focus on nltk.tokenize, particularly Punkt.

Punkt is valuable for studying sentence segmentation as a learned collection of abbreviations, collocations, and sentence-start evidence. Its implementation separates training, learned parameters, language conventions, and segmentation rather than burying everything in one regex.

• C1: Abbreviations, orthographic evidence, and closing punctuation interact in boundary decisions. The source exposes boundary realignment and language-dependent patterns, including fixes motivated by regex and performance failure modes. Punkt source documentation
• C2: Incremental training, parameter objects, and overridable language variables make the implementation reusable across corpora and conventions. Punkt implementation
• C4: Release notes record sentence-tokenization accuracy/performance regression fixes in 2022, interpreter compatibility work in subsequent years, the 2024 move away from pickled models for security, and 2025 checksum changes. This is concrete evolution of compatibility and failure handling. Release history

Entry points: the executable Punkt source documentation and release history above. The selection concerns this substantial subsystem, not uniform quality across every NLTK module.

7. explosion/spaCy

Language/role: Python/Cython NLP framework; tokenizer, Doc/Span/Token, alignment, and retokenization.

The strongest engineering lesson is how downstream annotations depend on token boundaries. Tokenization is part of a shared document representation, and subsequent merges or splits must preserve text and reconcile dependencies.

• C1: The tokenizer preserves doc.text == input_text. Alignment requires compatible underlying text, while retokenization rejects overlapping merges and requires split strings to reconstruct the original token. Dependency heads must also be specified correctly. Tokenization and retokenization guide
• C2: A common document model connects configurable tokenizer rules, language exceptions, spans, and alignment between alternative segmentations. Buffered retokenization changes provide a controlled interface for updating that shared structure. Linguistic features guide

Entry point: the guide above, especially tokenization, alignment, and merging/splitting sections. It provides concrete contracts worth reading before studying integrations that assume token indexes remain stable.

8. winkjs/wink-nlp

Language/role: JavaScript; model-driven NLP pipeline and tokenizer.

This is a useful smaller JavaScript codebase for understanding the interaction of lexical caching, language-specific patterns, and optional analysis stages. Its pipeline documentation explains consequences of disabling stages rather than merely presenting a list of switches.

• C2: Pipeline stages have explicit dependencies and ordering: sentiment uses negation, lemmatization benefits from POS, and omitting sentence detection changes document interpretation. Configuration therefore represents a reusable processing pipeline with semantic consequences. Pipeline guide
• C3: The tokenizer checks cached lexemes and category patterns, handles punctuation, and falls back to more involved splitting. These visible fast paths make the relationship between language coverage and repeated work inspectable. Tokenizer implementation

Entry points: the pipeline guide and tokenizer source above. Performance interest comes from the staged and cached implementation; published throughput figures are not used as evidence here.

9. spencermountain/compromise

Language/role: JavaScript; English-oriented rule-based text analysis and transformations.

Study the distinction between a document and a view selecting parts of that document. Terms retain original text and surrounding whitespace alongside normalized forms and tags; selections and transformations build on those representations.

• C1: Views share their underlying document, so mutations affect other views; cloning provides independence. Preserving original and normalized representations separately is essential when transformations and matching coexist. Core concepts, normalization implementation
• C2: Selection views, plugins, compute hooks, and model extensions provide a reusable text-processing abstraction rather than a single tokenizer function. Core concepts
• C3: Explicit build tiers separate basic tokenization from tagging and richer selections. The documentation discusses bundle-size choices and why the interacting tagger limits finer-grained tree shaking. Build architecture

Entry points: the concepts document and normalization source above. Its normalization is application-oriented; do not equate all such transformations with Unicode canonical equivalence.

10. fnl/syntok

Language/role: Python; tokenization and sentence segmentation, principally for English, German, and Spanish.

This compact project is worth inspecting when token offsets and formatting matter as much as token strings. Its Token representation records value, preceding spacing, and absolute position, while the sentence segmenter consumes a token stream.

• C1: Hyphens, apostrophes, zero-width characters, and contractions complicate the relationship between token value and source text. The implementation makes that relationship visible through spacing and offset fields; configurable contraction and hyphen handling mean reconstruction is not an unconditional invariant. Tokenizer source
• C2: Tokenization and segmentation are separate modules with an explicit token representation and iterator-based processing. This supports applications needing either stage independently. Repository module description, tokenizer implementation

Entry point: syntok/tokenizer.py above. It is a focused alternative to a full NLP framework, but its documented language scope should not be mistaken for universal boundary detection.

11. diasks2/pragmatic_segmenter

Language/role: Ruby; rule-based multilingual sentence segmentation.

The processor exposes an ordered pipeline for protecting or transforming constructs before sentence scanning: lists, abbreviations, numbers, references, continuous punctuation, and other patterns. Its published boundary examples make the rule interactions concrete.

• C1: Decimal punctuation, abbreviations, quotations, and names require ordered handling; applying a plausible rule at the wrong stage changes boundaries. The processor and documented “Golden Rules” provide implementation and challenging examples to study together. Processor source, repository examples
• C2: Language-specific abbreviation and punctuation behavior plugs into a shared processor. The common pipeline therefore supports multiple language strategies without requiring a wholly separate engine for each. Processor implementation

Entry points: the processor source and repository's Golden Rules examples. These are heuristic decisions: examples document intended behavior, not proof that arbitrary semantic ambiguity can be resolved.

Language-specific segmentation and morphology

12. fxsjy/jieba

Language/role: Python; Chinese word segmentation with dictionary and unknown-word handling.

Study a readable implementation of prefix-dictionary candidate generation, a directed acyclic graph over character positions, and backward dynamic programming. Unknown spans can pass through an HMM-based segmenter, making the dictionary/statistical boundary especially clear.

• C1: Ambiguous segmentations are resolved using accumulated log-frequency scores and fallback frequencies. User dictionary changes alter candidate paths, and known versus unknown spans must be routed consistently. Core tokenizer source
• C2: Independent Tokenizer instances, user dictionaries, dictionary updates, and different cutting modes expose reusable policy choices around the same core machinery. The code keeps full segmentation, ordinary segmentation, and search-oriented alternatives distinguishable. Tokenizer implementation, usage documentation

Entry point: jieba/__init__.py, particularly DAG construction, route calculation, and dispatch to unknown-word handling. Inclusion reflects implementation value; a current maintenance commitment has not been established by this research.

13. atilika/kuromoji

Language/role: Java; standalone Japanese morphological analysis.

This repository provides a clear view of dictionary lookup feeding a Viterbi lattice. It is the standalone Atilika project, distinct from the Japanese analysis subsystem maintained within Apache Lucene.

• C1: Lattice construction only starts candidates at reachable positions and combines known, unknown, and user-dictionary entries. Mode-dependent treatment of unknown spans creates correctness obligations beyond selecting the cheapest isolated dictionary match. Viterbi builder
• C2: Shared core machinery supports dictionary-specific modules such as IPADIC, UniDic, and Juman, whose token feature APIs reflect the dictionary schema. This separates common segmentation mechanics from linguistic feature inventories. Module and API documentation

Entry points: the Viterbi builder and repository's dictionary-module documentation. Status caveat: treat this as a reference-oriented selection; the inspected README still presents 0.9.0 dependency coordinates, and this report does not establish active maintenance of the standalone project.

14. ikawaha/kagome

Language/role: Go; Japanese morphological tokenization with dictionary packages.

Kagome is a substantial Go implementation, not simply a C tokenizer binding. The tokenizer source connects dictionary configuration, lattice construction, forward/backward processing, and token construction in one readable path.

• C1: Output includes positions and boundaries derived from lattice nodes; character lengths use UTF-8 rune counts rather than byte length. BOS/EOS handling and normal, search, and extended modes change how paths become output tokens. v2 tokenizer source
• C2: System and optional user dictionaries feed a shared tokenizer API supporting morphological tokens, word-separated output, and graph inspection. Mode selection is an explicit policy within that API. Tokenizer implementation, repository usage guide

Entry point: the verified v2 tokenizer implementation above. The encoding/index conventions deserve attention when integrating its positions with byte-oriented Go string operations.

15. lindera/lindera

Language/role: Rust; dictionary-based morphological analysis for Japanese, Korean, and Chinese.

The current architecture separates dictionary data, lattice segmentation, analysis filters, and higher-level interfaces. Although the project documents an origin in kuromoji-rs, its present workspace and analysis architecture provide substantive independent material; that ancestor is not counted separately here.

• C2: Character filters, a minimum-cost lattice segmenter, and token filters form a configurable pipeline. Separate dictionary, training, segmentation, and analysis components allow applications to use different layers. Architecture
• C3: Dictionary lookups use a serialized double-array trie; optional memory mapping avoids some owned-data loading. A SegmentWorker reuses lattice/scratch storage and applies a shrinking policy to limit retained memory after large inputs. Architecture and memory management

Entry points: the architecture document and project overview. This is a useful contrast with the Java and Go implementations because storage ownership and worker reuse are explicit architectural concerns.

16. PyThaiNLP/pythainlp

Language/role: Python Thai-language toolkit; specifically word tokenization and the newmm engine.

Thai segmentation provides a different problem from whitespace-delimited languages: dictionary matches must coexist with valid character-cluster boundaries, ambiguity, and unknown or non-Thai spans. The engine exposes these concerns as a trie, candidate graph, and boundary checks.

• C1: The inspected newmm implementation restricts dictionary candidates using Thai Character Cluster boundaries and resolves ambiguous paths. Unknown spans and mixed-script text have explicit fallback behavior. Version 5.1 implementation documentation
• C3: Graph-size heuristics and a safe mode address expensive ambiguous or long inputs. Later release notes specifically record a fix for exponential path growth, demonstrating why ordinary-input speed is insufficient evidence of robustness. Implementation, release notes

Entry points: the versioned source documentation and releases above. The 5.1 source is an architectural study reference, not evidence that it contains subsequent fixes; match the implementation to the release being evaluated.

17. anoopkunchukuttan/indic_nlp_library

Language/role: Python; Indic-language normalization, tokenization, and related text processing.

The normalizer implementation is a useful study in combining shared script machinery with language-specific orthographic policy. Its transformations go beyond Unicode canonical normalization, so their semantics must be understood before they are used in a preprocessing pipeline.

• C1: Nukta handling, nasal conversion, vowel conventions, and context-dependent punctuation transformations can change linguistic distinctions. Options make some policies explicit; for example, Devanagari-specific rules distinguish danda and visarga treatment from generic punctuation cleanup. Normalizer implementation
• C2: A common normalizer interface and base class factor out shared handling while script subclasses implement their own transformations. Shared script offsets also support common transformations across related writing systems. Source documentation

Entry point: the normalizer module above. Documentation caveat: the generated documentation carries a 0.2 label; verify API correspondence with the intended repository revision. These are configurable orthographic transformations, not a promise of lossless text normalization.

18. CAMeL-Lab/camel_tools

Language/role: Python Arabic NLP toolkit; focus on morphological tokenization and its disambiguator interface.

Here tokenization means selecting and formatting a morphological analysis of an already word-tokenized input. This makes the code a useful counterexample to treating tokenization as a purely character-level boundary problem.

• C1: Different tokenization schemes and diacritization choices can change token counts, including whether standalone morphemes survive dediacritization. The implementation defines fallbacks for missing analyses and missing scheme values instead of assuming every word is analyzable. API guide, implementation
• C2: An injected disambiguator supplies analyses; scheme and output formatting remain tokenizer choices. This separates analysis strategy and dialect-specific resources from the interface producing tokens. Morphological tokenizer API

Entry points: the API guide and implementation above. The pretokenized-input requirement matters when composing this tokenizer with punctuation and whitespace processing.

Unicode boundaries and normalization infrastructure

19. unicode-org/icu

Language/role: C/C++ and Java; focus on ICU boundary analysis and the ICU4C UText abstraction. The monorepo is counted once.

ICU demonstrates how language-aware boundary services can operate over a common Unicode text interface while retaining specialized dictionary handling. Its storage abstraction is particularly valuable for engineers integrating segmentation with nontrivial text containers.

• C1: UText providers expose UTF-16 chunks while mapping indexes to the native representation. Correct chunk boundaries and native/UTF-16 index conversion are necessary when the underlying text uses another encoding. UText design
• C2: Break iterators offer rule-based and dictionary-assisted boundaries through a common service; UText allows these services to work with multiple storage representations and custom providers. Boundary analysis, UText
• C3: The boundary guide explicitly recommends reusing iterators with new text because construction has meaningful cost. Boundary iterator usage

Entry points: the two guides above. The selection concerns text storage and boundary machinery, not ICU's unrelated internationalization subsystems.

20. unicode-org/icu4x

Language/role: Rust; modular internationalization components, particularly icu_segmenter and its data-provider architecture.

ICU4X is worth studying independently of ICU because deployment data and resource tradeoffs are part of the component interfaces. Its word segmenter exposes alternatives whose CPU, data-size, and language-coverage implications are documented.

• C2: Segmenters can use compiled data or provider-supplied data, separating boundary APIs from how linguistic resources are packaged and delivered. Segmenter module, WordSegmenter constructors
• C3: Dictionary and LSTM approaches offer different data-size/computation tradeoffs. Explicit construction choices make the tradeoff inspectable rather than concealing it behind an undifferentiated “fast” API. WordSegmenter documentation
• C1: The same documentation warns that the LSTM option does not supply CJK segmentation coverage. Algorithm selection therefore affects semantics as well as resources. Coverage constraints

Entry points: the module and constructor documentation above. The linked latest pages are moving references; pin a release when evaluating deployment behavior.

21. JuliaStrings/utf8proc

Language/role: C; UTF-8 processing, Unicode properties, normalization, and case folding.

This library is useful for studying a compact, explicit boundary between Unicode data and caller-selected transformations. Its public header provides substantive contracts, flags, property structures, and error behavior rather than only declarations.

• C1: Malformed UTF-8, invalid option combinations, integer overflow, allocation failure, and unassigned code points have explicit failure semantics. Canonical versus compatibility transformations and optional removal of marks are distinct policies with different information-loss consequences. Public API and contracts
• C2: Property access, decomposition, composition, case folding, and mapping options provide reusable building blocks for different text pipelines. The interface also distinguishes API and ABI compatibility in its versioning contract. Header documentation

Entry point: utf8proc.h above, followed by the repository's implementations of the relevant operations. A shared normalization API does not make all flag combinations interchangeable or lossless.

22. unicode-rs/unicode-segmentation

Language/role: Rust; Unicode grapheme, word, and sentence segmentation.

The most distinctive abstraction is GraphemeCursor, which supports partial text and ropes instead of requiring one contiguous string. It makes the algorithm's need for surrounding context visible to callers.

• C1: A boundary decision can require preceding context or another chunk; callers must return consistent text at valid UTF-8 boundaries. Regional-indicator sequences illustrate why inspecting only the current chunk is insufficient. GraphemeCursor contract and examples
• C2: Ordinary iterators serve contiguous strings, while the cursor protocol supports editors and other chunked-storage applications. Explicit incomplete-result variants allow storage traversal to remain outside the segmentation algorithm. Cursor API, repository overview

Entry point: GraphemeCursor documentation above. It is especially useful for understanding why grapheme segmentation cannot always be implemented as an independent operation on arbitrary buffers.

23. adah1972/libunibreak

Language/role: C; Unicode line, word, and grapheme breaking.

The word-break source offers a direct implementation of Unicode boundary rules, with comments connecting state transitions to named rules. It also exposes how encoding-specific character traversal can be separated from a shared boundary engine.

• C1: Combining/format characters, newline exceptions, Hebrew quotations, and regional-indicator parity interact through state. The current repository also describes deferred fixups for line-break rules requiring lookahead. These are useful examples of rule ordering and context management. Word-break implementation, current implementation notes
• C2: UTF-8, UTF-16, and UTF-32 adapters feed common word-break processing through character-reader callbacks, preserving one rule implementation across encodings. Shared engine and adapters

Entry points: src/wordbreak.c and the repository's current Unicode-support notes. Search results contained older Unicode-version descriptions; this report uses the opened repository rather than those stale snippets. Unicode-version compatibility should be checked against the chosen release.

Stemming languages and repair pipelines

24. snowballstem/snowball

Language/role: Snowball domain-specific language and compiler, principally implemented in C, with generated stemmers for multiple target languages.

This is a substantial original compiler and algorithm collection, not a generated binding wrapper. Study how a small language captures common stemming operations while making cursor movement, matching, and mutation explicit.

• C1: Cursor and limit semantics become subtle when moving backwards through mutable text. The language distinguishes backwards execution from reverse tests, constrains mutation in reverse contexts, and defines matching/failure behavior for constructs such as among. Language and compiler manual
• C2: Linguistic stemming algorithms are expressed once in a dedicated language, with a compiler generating implementations for different runtimes. This separates algorithm definition from target-language plumbing. Manual, repository compiler and algorithm overview

Entry point: the manual above, especially cursor/limit behavior, backwards processing, and matching constructs. Stemming produces algorithmic stems; it should not be confused with a general morphological lemmatizer.

25. rspeer/python-ftfy

Language/role: Python; repair of mojibake and related Unicode text problems.

ftfy is useful for studying transformations that must be conservative about already-correct text. It combines evidence of likely corruption with an inspectable repair pipeline rather than assuming every unusual string has the same encoding problem.

• C1: Badness heuristics help decide when an apparent encoding repair is plausible and when to stop. The difficult boundary is between corrupted text and legitimate unusual characters, so heuristic evidence is part of correctness rather than a guarantee of perfect detection. Heuristic documentation
• C2: Repair can return an explanation made of encoding, decoding, transcoding, and named transformation steps; an operation plan can be inspected and reapplied. Configuration controls which classes of repair are enabled. Explanations and configuration
• C3: The text-level API splits processing into bounded segments, while the explanation API processes a whole segment. This makes a resource-management choice visible in the API. Processing documentation

Entry points: the heuristic and explanation guides above. This is text repair, not an unrestricted detector for arbitrary unknown byte encodings.

Search coverage, exclusions, and limits

Discovery used more than six distinct live search formulations. The main angles included BPE/WordPiece/unigram implementations; Unicode segmentation and normalization; Japanese and Chinese dictionary lattices; sentence-boundary engines; JavaScript text-processing pipelines; stemming and mojibake repair; Thai trie/cluster segmentation; Indic and Arabic normalization/morphology; and C#, Go, Ruby, and C alternatives. Follow-up searches inspected runtime/compiler design, model-building guides, buffer and offset contracts, release notes, and source implementations. Later searches increasingly returned bindings, overlapping ports, small demonstrations, or unsupported performance claims, so the selection stopped at 25 substantive implementations.

Every retained canonical GitHub repository page was opened, and at least one additional primary implementation or documentation source was opened and read for each. The linked material includes executable source, substantive API contracts, design explanations, or compiler documentation; search snippets alone were not used to establish inclusion. No repository stars or unverified benchmark multipliers contribute to the criteria. Each repository is counted once, including the explicitly scoped ML.NET and ICU monorepos. Lindera's documented ancestry is disclosed rather than presenting its ancestor as an additional independent selection.

Excluded material included awesome lists, tutorials, wrapper-only packages, superseded duplicate ports, generic lexical scanners for programming languages, font shaping/rendering, and broader search or model-training systems without a focused text-processing subsystem selected for inspection. Original code generators such as Snowball remain in scope because the reusable compiler and algorithms are the substantive implementation. No selected repository is being represented as an official mirror of an implementation hosted elsewhere. The report does not assert that every selected project is actively maintained; standalone Kuromoji is explicitly treated as a reference-oriented inclusion, and maintenance was not comprehensively audited for the others.

This was read-only web and source research: no candidate code was installed, built, tested, or benchmarked. Architectural value and criterion assignments are grounded in the cited material but remain engineering judgments. Default branches and latest documentation can change; the Hugging Face transition, versioned PyThaiNLP source, and older Indic documentation deserve particular care. Unicode conformance, language coverage, normalization loss, and offset conventions must be evaluated for the exact release and configuration an application intends to use.