API Reference

Load a BPE tokenizer from either a directory (vocab.txt + merges.txt) or a vocab file path.

Load a BPE tokenizer from explicit vocab + merges paths.

Load a byte-level BPE tokenizer from a directory (vocab.txt + merges.txt) or vocab path.

Load a byte-level BPE tokenizer from explicit vocab + merges paths.

Load GPT-2 / RoBERTa style BPE from vocab.json + merges.txt.

Example: load_bpe_gpt2("/path/to/vocab.json", "/path/to/merges.txt")

Load GPT-2 encoder variant from encoder.json + vocab.bpe.

Example: load_bpe_encoder("/path/to/encoder.json", "/path/to/vocab.bpe")

Load a Unigram tokenizer from unigram.tsv (file or directory).

Expected format (tab-separated): token<TAB>score[<TAB>special_symbol]

Load a WordPiece tokenizer from a vocab file path or a directory containing vocab.txt.

Examples:

• load_wordpiece("/path/to/vocab.txt")
• load_wordpiece("/path/to/model_dir")

Load a SentencePiece .model file.

Supported inputs:

• standard SentencePiece binary protobuf .model/.model.v3 payloads
• Keemena text-exported model files:
  • key/value lines (type=unigram|bpe, whitespace_marker=▁, unk_token=<unk>)
  • piece rows: piece<TAB>token<TAB>score[<TAB>special_symbol]
  • bpe merge rows (for type=bpe): merge<TAB>left<TAB>right

Examples:

• load_sentencepiece("/path/to/tokenizer.model"; kind=:auto)
• load_sentencepiece("/path/to/tokenizer.model.v3"; kind=:bpe)

Load a tiktoken encoding file (*.tiktoken).

The expected format is line-based: <base64_token_bytes><space><rank> where ranks are non-negative integers.

Examples:

• load_tiktoken("/path/to/o200k_base.tiktoken")
• load_tiktoken("/path/to/tokenizer.model") (when file contains tiktoken text lines)

Load a Hugging Face tokenizer.json tokenizer in pure Julia.

Expected files:

• tokenizer.json directly, or
• a directory containing tokenizer.json.

Examples:

• load_hf_tokenizer_json("/path/to/tokenizer.json")
• load_hf_tokenizer_json("/path/to/model_dir")

Load tokenizer by built-in model name.

Load tokenizer from file system path.

Common format contracts:

• :hf_tokenizer_json -> tokenizer.json
• :bpe_gpt2 -> vocab.json + merges.txt
• :bpe_encoder -> encoder.json + vocab.bpe
• :wordpiece / :wordpiece_vocab -> vocab.txt
• :sentencepiece_model -> *.model / *.model.v3 / sentencepiece.bpe.model
• :tiktoken -> *.tiktoken or tiktoken-text tokenizer.model

Examples:

• load_tokenizer("/path/to/model_dir")
• load_tokenizer("/path/to/tokenizer.model"; format=:tiktoken)
• load_tokenizer("/path/to/tokenizer.json"; format=:hf_tokenizer_json)

Load tokenizer from explicit (vocab_path, merges_path) tuple.

This tuple form is for classic BPE/byte-level BPE (vocab.txt + merges.txt) or explicit JSON-pair loaders (vocab.json + merges.txt, encoder.json + vocab.bpe) when accompanied by format.

Load tokenizer from a named specification.

Examples:

• (format=:wordpiece, path="/.../vocab.txt")
• (format=:hf_tokenizer_json, path="/.../tokenizer.json")
• (format=:unigram, path="/.../unigram.tsv")
• (format=:bpe_gpt2, vocab_json="/.../vocab.json", merges_txt="/.../merges.txt")
• (format=:bpe_encoder, encoder_json="/.../encoder.json", vocab_bpe="/.../vocab.bpe")
• (format=:wordpiece, vocab_txt="/.../vocab.txt") (alias)
• (format=:sentencepiece_model, model_file="/.../tokenizer.model") (alias)
• (format=:tiktoken, encoding_file="/.../o200k_base.tiktoken") (alias)
• (format=:hf_tokenizer_json, tokenizer_json="/.../tokenizer.json") (alias)
• (format=:unigram, unigram_tsv="/.../unigram.tsv") (alias)

Load tokenizer from a FilesSpec.

Detect tokenizer format from a local file or directory.

Returns one of symbols such as :hf_tokenizer_json, :bpe_gpt2, :bpe_encoder, :sentencepiece_model, :tiktoken, :wordpiece, :bpe, or :unigram.

Examples:

• detect_tokenizer_format("/path/to/model_dir")
• detect_tokenizer_format("/path/to/tokenizer.model")

Inspect a tokenizer directory and return detected candidate files.

Example: detect_tokenizer_files("/path/to/model_dir")

quick_tokenize(tokenizer_or_source, input_text; kwargs...) -> NamedTuple

High-level one-call wrapper for common single-text tokenization workflows.

This helper applies the recommended offsets pipeline by default:

1. tokenization_text = tokenization_view(tokenizer, input_text)
2. encode_result(tokenizer, tokenization_text; assume_normalized=true, ...)

Supported inputs:

• quick_tokenize(tokenizer::AbstractSubwordTokenizer, input_text; ...)
• quick_tokenize(source::Symbol, input_text; format=nothing, prefetch=true, ...)
• quick_tokenize(source::AbstractString, input_text; format=nothing, prefetch=true, ...)

Keyword arguments:

• add_special_tokens::Bool=true
• apply_tokenization_view::Bool=true
• return_offsets::Bool=true
• return_masks::Bool=true

Returns a NamedTuple with keys:

• token_pieces
• token_ids
• decoded_text
• tokenization_text
• offsets
• attention_mask
• token_type_ids
• special_tokens_mask
• metadata

quick_encode_batch(tokenizer_or_source, input_texts; kwargs...) -> NamedTuple

High-level wrapper for batch structured encoding.

By default, each input text is first converted with tokenization_view so offsets and alignment metadata are anchored to tokenizer-coordinate text.

Keyword arguments:

• add_special_tokens::Bool=true
• apply_tokenization_view::Bool=true
• return_offsets::Bool=true
• return_masks::Bool=true
• format::Union{Nothing,Symbol}=nothing (source overloads only)
• prefetch::Bool=true (source overloads only)

Returns a NamedTuple with keys:

• tokenization_texts
• results
• sequence_lengths
• metadata

collate_padded_batch(results; tokenizer=nothing, pad_token_id=nothing, kwargs...) -> NamedTuple

Collate Vector{TokenizationResult} into dense (sequence_length, batch_size) matrices.

Returns:

• ids::Matrix{Int}
• attention_mask::Matrix{Int}
• token_type_ids::Matrix{Int}
• special_tokens_mask::Matrix{Int}
• sequence_lengths::Vector{Int}
• pad_token_id::Int
• pad_side::Symbol

Padding behavior:

• ids are filled with pad_token_id.
• attention_mask uses 1 for valid tokens and 0 for padding.
• token_type_ids defaults to 0 where missing.
• special_tokens_mask defaults to 0 on valid tokens where missing and uses 1 on padding positions.

Pad token selection:

• If pad_token_id is provided, it is used directly.
• Otherwise pad_id(tokenizer) is used when available.
• Otherwise eos_id(tokenizer) is used when available.
• If none are available, throws an ArgumentError.

Optional keyword arguments:

• pad_to_multiple_of::Union{Nothing,Int}=nothing
• pad_side::Symbol=:right (only right padding is currently supported)

causal_lm_labels(ids, attention_mask; ignore_index=-100, zero_based=false) -> Matrix{Int}

Build next-token labels for causal language modeling from padded ids and attention_mask matrices shaped (sequence_length, batch_size).

For each sequence column:

• valid non-final positions receive the next valid token id,
• the final valid position receives ignore_index,
• padding positions receive ignore_index.

When zero_based=true, subtracts 1 from all non-ignored labels to support consumers that expect 0-based ids.

quick_causal_lm_batch(tokenizer_or_source, input_texts; kwargs...) -> NamedTuple

One-call helper for training-ready causal LM tensors.

Pipeline:

1. quick_encode_batch(...; return_masks=true)
2. collate_padded_batch(...)
3. causal_lm_labels(...)

Keyword arguments:

• add_special_tokens::Bool=true
• apply_tokenization_view::Bool=true
• return_offsets::Bool=false
• pad_token_id::Union{Nothing,Int}=nothing
• pad_to_multiple_of::Union{Nothing,Int}=nothing
• pad_side::Symbol=:right
• ignore_index::Int=-100
• zero_based::Bool=false
• format::Union{Nothing,Symbol}=nothing (source overloads only)
• prefetch::Bool=true (source overloads only)

Returns a NamedTuple with keys:

• ids
• attention_mask
• labels
• token_type_ids
• special_tokens_mask
• tokenization_texts
• sequence_lengths
• pad_token_id
• ignore_index
• zero_based

quick_train_bundle(trainer, corpus; kwargs...) -> NamedTuple

High-level training round-trip helper:

1. Train with a selected trainer.
2. Save a training bundle with save_training_bundle.
3. Reload with load_training_bundle.
4. Run a sanity encode/decode pass.

Supported trainer symbols:

• :wordpiece
• :hf_bert_wordpiece
• :hf_roberta_bytebpe
• :hf_gpt2_bytebpe

Convenience overload:

• quick_train_bundle(corpus; kwargs...) defaults to trainer=:wordpiece.

Keyword arguments:

• bundle_directory::Union{Nothing,AbstractString}=nothing
• overwrite::Bool=true
• export_format::Symbol=:auto
• sanity_text::AbstractString="hello world"
• plus trainer-specific keywords forwarded to the selected train_*_result.

Returns a NamedTuple with keys:

• bundle_directory
• bundle_files
• training_summary
• tokenizer
• sanity_encoded_ids
• sanity_decoded_text

List available built-in model names.

Describe a built-in model.

Resolve built-in model name to on-disk path.

Ensure artifact-backed built-in models are present on disk.

Returns a dictionary of key => is_available.

Register a local tokenizer path under a symbolic key and persist it in the cache registry.

Register local model files by explicit specification.

Register local model files from a FilesSpec.

Install an installable-gated tokenizer into the user cache and register it by key.

Install the gated LLaMA 2 tokenizer files into local cache and register them.

This is a convenience wrapper over install_model!(:llama2_tokenizer; ...).

Install the gated LLaMA 3 8B tokenizer files into local cache and register them.

This is a convenience wrapper over install_model!(:llama3_8b_tokenizer; ...).

Download selected files from a Hugging Face repository revision into cache.

This helper is opt-in and useful for user-managed / gated tokenizers.

Recommended built-in keys for LLM-oriented default prefetching.

Abstract parent type for all subword tokenizers.

Tokenizers are callable and support: tokenizer(text::AbstractString) -> Vector{String}.

Structured file specification for local tokenizer loading/registration.

Use path for single-file formats and explicit pairs for multi-file formats.

Vocabulary container with forward/reverse lookup and special token IDs.

IDs are 1-based.

Structured tokenization output for downstream pipelines.

Offset contract:

• coordinate unit: UTF-8 codeunits.
• index base: 1.
• span style: half-open [start, stop).
• valid bounds for spanful tokens: 1 <= start <= stop <= ncodeunits(text) + 1.
• sentinel for tokens without source-text spans: (0, 0).
• inserted post-processor specials use sentinel offsets.
• present-in-text special added tokens keep real spans, and may still have special_tokens_mask[i] == 1.
• special_tokens_mask marks special-token identity; offsets determine span participation.

Common metadata for tokenizer instances.

Assert offsets satisfy the package offset contract.

Throws ArgumentError on first contract violation. With require_string_boundaries=true, non-empty spans must start/end on valid Julia string boundaries.

Return prefetch status for a single model key.

List available built-in model names.

BOS token ID if available.

BOS token ID if available.

BOS token ID if available.

BOS token ID if available.

Return beginning-of-sequence token ID if available.

BOS token ID if available.

BOS token ID if available.

BOS token ID if available.

List cache keys for in-session cached tokenizers.

causal_lm_labels(ids, attention_mask; ignore_index=-100, zero_based=false) -> Matrix{Int}

Build next-token labels for causal language modeling from padded ids and attention_mask matrices shaped (sequence_length, batch_size).

For each sequence column:

• valid non-final positions receive the next valid token id,
• the final valid position receives ignore_index,
• padding positions receive ignore_index.

When zero_based=true, subtracts 1 from all non-ignored labels to support consumers that expect 0-based ids.

Clear the in-session tokenizer cache used by one-call convenience APIs.

collate_padded_batch(results; tokenizer=nothing, pad_token_id=nothing, kwargs...) -> NamedTuple

Collate Vector{TokenizationResult} into dense (sequence_length, batch_size) matrices.

Returns:

• ids::Matrix{Int}
• attention_mask::Matrix{Int}
• token_type_ids::Matrix{Int}
• special_tokens_mask::Matrix{Int}
• sequence_lengths::Vector{Int}
• pad_token_id::Int
• pad_side::Symbol

Padding behavior:

• ids are filled with pad_token_id.
• attention_mask uses 1 for valid tokens and 0 for padding.
• token_type_ids defaults to 0 where missing.
• special_tokens_mask defaults to 0 on valid tokens where missing and uses 1 on padding positions.

Pad token selection:

• If pad_token_id is provided, it is used directly.
• Otherwise pad_id(tokenizer) is used when available.
• Otherwise eos_id(tokenizer) is used when available.
• If none are available, throws an ArgumentError.

Optional keyword arguments:

• pad_to_multiple_of::Union{Nothing,Int}=nothing
• pad_side::Symbol=:right (only right padding is currently supported)

One-call decode by tokenizer path/directory.

Decode token IDs into text.

Decode token IDs to text.

Decode byte-level BPE IDs back to text.

Decode SentencePiece IDs back to text.

One-call decode by model key.

Decode tiktoken rank IDs to text.

Decode unigram token IDs back to text.

Decode WordPiece token IDs back into text.

Describe a built-in model.

Inspect a tokenizer directory and return detected candidate files.

Example: detect_tokenizer_files("/path/to/model_dir")

Detect tokenizer format from a local file or directory.

Returns one of symbols such as :hf_tokenizer_json, :bpe_gpt2, :bpe_encoder, :sentencepiece_model, :tiktoken, :wordpiece, :bpe, or :unigram.

Examples:

• detect_tokenizer_format("/path/to/model_dir")
• detect_tokenizer_format("/path/to/tokenizer.model")

Download selected files from a Hugging Face repository revision into cache.

This helper is opt-in and useful for user-managed / gated tokenizers.

One-call encode by tokenizer path/directory.

Encode text into token IDs.

Encode text to token IDs.

Encode text to byte-level BPE IDs.

Encode text to SentencePiece IDs.

One-call encode by model key.

Encode text into tiktoken rank IDs (1-based in this package).

Encode text to unigram token IDs.

Encode text to WordPiece token IDs.

Batch variant of encode_result.

Encode text and return a structured TokenizationResult.

Key keyword arguments:

• assume_normalized::Bool=false: when true, tokenizer intrinsic normalization is skipped and offsets are computed against the exact provided text.
• return_offsets::Bool=false: include token-level offsets when available.
• return_masks::Bool=false: include attention/token-type/special-token masks.

Offset note:

• Offsets use the package-wide 1-based UTF-8 codeunit half-open convention.
• assume_normalized changes whether intrinsic normalization runs; it does not change the offset coordinate system.

One-call structured encode by tokenizer path/directory.

One-call structured encode by model key.

EOS token ID if available.

EOS token ID if available.

EOS token ID if available.

EOS token ID if available.

Return end-of-sequence token ID if available.

EOS token ID if available.

EOS token ID if available.

EOS token ID if available.

Export tokenizer to external formats.

Supported format values:

• :internal
• :bpe / :bpe_gpt2
• :wordpiece_vocab
• :unigram_tsv
• :sentencepiece_model
• :hf_tokenizer_json

Return a cached tokenizer for a model key or path, loading and caching on first use.

Return true when an offset carries a non-empty source-text span.

Return true when an offset carries a real source-text span.

Reverse token lookup.

Reverse token lookup.

Reverse token lookup.

Reverse token lookup.

Map ID to token string.

Reverse token lookup.

Reverse token lookup.

Reverse token lookup.

Install the gated LLaMA 2 tokenizer files into local cache and register them.

This is a convenience wrapper over install_model!(:llama2_tokenizer; ...).

Install the gated LLaMA 3 8B tokenizer files into local cache and register them.

This is a convenience wrapper over install_model!(:llama3_8b_tokenizer; ...).

Install an installable-gated tokenizer into the user cache and register it by key.

Return whether idx is a valid Julia string boundary for text.

This includes the exclusive end boundary ncodeunits(text) + 1.

Return a function compatible with KeemenaPreprocessing's callable tokenizer contract.

Level key used by KeemenaPreprocessing for callable tokenizers.

Load a BPE tokenizer from explicit vocab + merges paths.

Load a BPE tokenizer from either a directory (vocab.txt + merges.txt) or a vocab file path.

Load GPT-2 encoder variant from encoder.json + vocab.bpe.

Example: load_bpe_encoder("/path/to/encoder.json", "/path/to/vocab.bpe")

Load GPT-2 / RoBERTa style BPE from vocab.json + merges.txt.

Example: load_bpe_gpt2("/path/to/vocab.json", "/path/to/merges.txt")

Load a byte-level BPE tokenizer from explicit vocab + merges paths.

Load a byte-level BPE tokenizer from a directory (vocab.txt + merges.txt) or vocab path.

Load a Hugging Face tokenizer.json tokenizer in pure Julia.

Expected files:

• tokenizer.json directly, or
• a directory containing tokenizer.json.

Examples:

• load_hf_tokenizer_json("/path/to/tokenizer.json")
• load_hf_tokenizer_json("/path/to/model_dir")

Load a SentencePiece .model file.

Supported inputs:

• standard SentencePiece binary protobuf .model/.model.v3 payloads
• Keemena text-exported model files:
  • key/value lines (type=unigram|bpe, whitespace_marker=▁, unk_token=<unk>)
  • piece rows: piece<TAB>token<TAB>score[<TAB>special_symbol]
  • bpe merge rows (for type=bpe): merge<TAB>left<TAB>right

Examples:

• load_sentencepiece("/path/to/tokenizer.model"; kind=:auto)
• load_sentencepiece("/path/to/tokenizer.model.v3"; kind=:bpe)

Load a tiktoken encoding file (*.tiktoken).

The expected format is line-based: <base64_token_bytes><space><rank> where ranks are non-negative integers.

Examples:

• load_tiktoken("/path/to/o200k_base.tiktoken")
• load_tiktoken("/path/to/tokenizer.model") (when file contains tiktoken text lines)

Load tokenizer from file system path.

Common format contracts:

• :hf_tokenizer_json -> tokenizer.json
• :bpe_gpt2 -> vocab.json + merges.txt
• :bpe_encoder -> encoder.json + vocab.bpe
• :wordpiece / :wordpiece_vocab -> vocab.txt
• :sentencepiece_model -> *.model / *.model.v3 / sentencepiece.bpe.model
• :tiktoken -> *.tiktoken or tiktoken-text tokenizer.model

Examples:

• load_tokenizer("/path/to/model_dir")
• load_tokenizer("/path/to/tokenizer.model"; format=:tiktoken)
• load_tokenizer("/path/to/tokenizer.json"; format=:hf_tokenizer_json)

Load tokenizer from a FilesSpec.

Load tokenizer from a named specification.

Examples:

• (format=:wordpiece, path="/.../vocab.txt")
• (format=:hf_tokenizer_json, path="/.../tokenizer.json")
• (format=:unigram, path="/.../unigram.tsv")
• (format=:bpe_gpt2, vocab_json="/.../vocab.json", merges_txt="/.../merges.txt")
• (format=:bpe_encoder, encoder_json="/.../encoder.json", vocab_bpe="/.../vocab.bpe")
• (format=:wordpiece, vocab_txt="/.../vocab.txt") (alias)
• (format=:sentencepiece_model, model_file="/.../tokenizer.model") (alias)
• (format=:tiktoken, encoding_file="/.../o200k_base.tiktoken") (alias)
• (format=:hf_tokenizer_json, tokenizer_json="/.../tokenizer.json") (alias)
• (format=:unigram, unigram_tsv="/.../unigram.tsv") (alias)

Load tokenizer by built-in model name.

Load tokenizer from explicit (vocab_path, merges_path) tuple.

This tuple form is for classic BPE/byte-level BPE (vocab.txt + merges.txt) or explicit JSON-pair loaders (vocab.json + merges.txt, encoder.json + vocab.bpe) when accompanied by format.

Load a Unigram tokenizer from unigram.tsv (file or directory).

Expected format (tab-separated): token<TAB>score[<TAB>special_symbol]

Load a WordPiece tokenizer from a vocab file path or a directory containing vocab.txt.

Examples:

• load_wordpiece("/path/to/vocab.txt")
• load_wordpiece("/path/to/model_dir")

Return model metadata.

Tokenizer metadata.

Tokenizer metadata.

Tokenizer metadata.

Tokenizer metadata.

Tokenizer metadata.

Tokenizer metadata.

Resolve built-in model name to on-disk path.

Return tokenizer intrinsic normalization output.

This does not perform pipeline-level preprocessing. Tokenizers without intrinsic normalization return text unchanged.

Normalize text using an optional user-provided callable.

Return whether participating offsets are non-overlapping in sequence order.

Participating offsets satisfy:

• not sentinel when ignore_sentinel=true
• not empty when ignore_empty=true

For participating offsets, this enforces next.start >= prev.stop.

Offset coordinate system for TokenizationResult.offsets.

Offsets are UTF-8 codeunit indices with half-open span convention [start, stop).

Offset index base for TokenizationResult.offsets.

Offsets are 1-based codeunit indices.

Sentinel used for tokens without a source-text span.

Offset span style.

TokenizationResult.offsets use half-open spans [start, stop).

Padding token ID if available.

Padding token ID if available.

Padding token ID if available.

Padding token ID if available.

Return padding-token ID if available.

Padding token ID if available.

Padding token ID if available.

Padding token ID if available.

Ensure artifact-backed built-in models are present on disk.

Returns a dictionary of key => is_available.

Return detailed prefetch status for built-in model keys.

Each value includes:

• available::Bool
• method::Symbol (:artifact, :fallback_download, :already_present, or :failed)
• path::Union{Nothing,String}
• error::Union{Nothing,String}

Print a compact prefetch status line for one model key.

quick_causal_lm_batch(tokenizer_or_source, input_texts; kwargs...) -> NamedTuple

One-call helper for training-ready causal LM tensors.

Pipeline:

1. quick_encode_batch(...; return_masks=true)
2. collate_padded_batch(...)
3. causal_lm_labels(...)

Keyword arguments:

• add_special_tokens::Bool=true
• apply_tokenization_view::Bool=true
• return_offsets::Bool=false
• pad_token_id::Union{Nothing,Int}=nothing
• pad_to_multiple_of::Union{Nothing,Int}=nothing
• pad_side::Symbol=:right
• ignore_index::Int=-100
• zero_based::Bool=false
• format::Union{Nothing,Symbol}=nothing (source overloads only)
• prefetch::Bool=true (source overloads only)

Returns a NamedTuple with keys:

• ids
• attention_mask
• labels
• token_type_ids
• special_tokens_mask
• tokenization_texts
• sequence_lengths
• pad_token_id
• ignore_index
• zero_based

quick_encode_batch(tokenizer_or_source, input_texts; kwargs...) -> NamedTuple

High-level wrapper for batch structured encoding.

By default, each input text is first converted with tokenization_view so offsets and alignment metadata are anchored to tokenizer-coordinate text.

Keyword arguments:

• add_special_tokens::Bool=true
• apply_tokenization_view::Bool=true
• return_offsets::Bool=true
• return_masks::Bool=true
• format::Union{Nothing,Symbol}=nothing (source overloads only)
• prefetch::Bool=true (source overloads only)

Returns a NamedTuple with keys:

• tokenization_texts
• results
• sequence_lengths
• metadata

quick_tokenize(tokenizer_or_source, input_text; kwargs...) -> NamedTuple

High-level one-call wrapper for common single-text tokenization workflows.

This helper applies the recommended offsets pipeline by default:

1. tokenization_text = tokenization_view(tokenizer, input_text)
2. encode_result(tokenizer, tokenization_text; assume_normalized=true, ...)

Supported inputs:

• quick_tokenize(tokenizer::AbstractSubwordTokenizer, input_text; ...)
• quick_tokenize(source::Symbol, input_text; format=nothing, prefetch=true, ...)
• quick_tokenize(source::AbstractString, input_text; format=nothing, prefetch=true, ...)

Keyword arguments:

• add_special_tokens::Bool=true
• apply_tokenization_view::Bool=true
• return_offsets::Bool=true
• return_masks::Bool=true

Returns a NamedTuple with keys:

• token_pieces
• token_ids
• decoded_text
• tokenization_text
• offsets
• attention_mask
• token_type_ids
• special_tokens_mask
• metadata

quick_train_bundle(trainer, corpus; kwargs...) -> NamedTuple

High-level training round-trip helper:

1. Train with a selected trainer.
2. Save a training bundle with save_training_bundle.
3. Reload with load_training_bundle.
4. Run a sanity encode/decode pass.

Supported trainer symbols:

• :wordpiece
• :hf_bert_wordpiece
• :hf_roberta_bytebpe
• :hf_gpt2_bytebpe

Convenience overload:

• quick_train_bundle(corpus; kwargs...) defaults to trainer=:wordpiece.

Keyword arguments:

• bundle_directory::Union{Nothing,AbstractString}=nothing
• overwrite::Bool=true
• export_format::Symbol=:auto
• sanity_text::AbstractString="hello world"
• plus trainer-specific keywords forwarded to the selected train_*_result.

Returns a NamedTuple with keys:

• bundle_directory
• bundle_files
• training_summary
• tokenizer
• sanity_encoded_ids
• sanity_decoded_text

Recommended built-in keys for LLM-oriented default prefetching.

Deprecated alias kept for compatibility. Use register_local_model! instead.

Register a local tokenizer path under a symbolic key and persist it in the cache registry.

Register local model files from a FilesSpec.

Register local model files by explicit specification.

Whether this tokenizer defines intrinsic normalization that can change text.

Save tokenizer to a canonical on-disk format.

format=:internal chooses a tokenizer-family specific default:

• WordPieceTokenizer -> vocab.txt
• BPETokenizer / ByteBPETokenizer -> vocab.txt + merges.txt
• UnigramTokenizer -> unigram.tsv
• SentencePieceTokenizer -> spm.model

Return the offset span as UTF-8 codeunits.

Sentinel and empty spans return UInt8[]. Invalid or out-of-bounds spans also return UInt8[] to keep this helper non-throwing for downstream inspection.

Return span length measured in UTF-8 codeunits.

Sentinel and empty spans return 0.

Return special token IDs keyed by symbol.

Special token IDs.

Special token IDs.

Special token IDs.

Special token IDs.

Special token IDs.

Special token IDs.

Forward token lookup.

Forward token lookup.

Forward token lookup.

Forward token lookup.

Map token string to ID, falling back to :unk.

Forward token lookup.

Forward token lookup.

Forward token lookup.

Canonical tokenizer text view used for subword offsets/alignment.

One-call tokenize by tokenizer path/directory.

Tokenize text into subword pieces.

Tokenize with classic BPE merges.

Tokenize text by first mapping bytes to unicode symbols, then applying BPE merges.

Tokenize text with SentencePiece wrapper behavior.

One-call tokenize by model key.

Tokenize text into b64:<...> token pieces.

Tokenize text using deterministic Viterbi segmentation.

Greedy longest-match WordPiece tokenization.

Attempt to return a substring for a half-open codeunit span [start, stop).

Sentinel and empty spans return "". If span boundaries are not valid Julia string boundaries, this returns nothing. This helper never throws.

Unknown token ID.

Unknown token ID.

Unknown token ID.

Unknown token ID.

Return unknown-token ID.

Unknown token ID.

Unknown token ID.

Unknown token ID.

Validate offsets against the package offset contract.

Returns true when all offsets satisfy bounds/sentinel invariants. With require_string_boundaries=true, non-empty spans must also start/end on valid Julia string boundaries.

Vocabulary size.

Vocabulary size.

Vocabulary size.

Vocabulary size.

Vocabulary size.

Vocabulary size.

Vocabulary size.

Vocabulary size.