Tokenize

Split text into tokens, unigrams, and vgrams.

Low-level helpers used by Create(). Use them to debug or build a custom pipeline.

Pipeline

text
  → CreateTokens    (filter each word when normalize = true)
  → unigrams        (strip each token when normalize = true)
  → vgrams          (overlapping windows, strip each when normalize = true)

Functions

import {
  CreateTokens,
  CreateUnigrams,
  CreateVGrams,
  Tokenize,
} from "@koda.oss/glyph";

CreateTokens("Hello, world!");
CreateUnigrams("Hello, world!");
CreateVGrams("one two three four", 3);

Tokenize("alpha beta gamma", { vgramSize: 2, normalize: true });

Tokenize(text, options?)

Returns all three lists:

{
  "tokens": ["alpha,", "beta!", "gamma"],
  "unigrams": ["alpha", "beta", "gamma"],
  "vgrams": ["alphabeta", "betagamma"]
}

Option	Default in Tokenize()	Default in Create()
vgramSize	2	4
normalize	true	true

Pass the same vgramSize in Tokenize() and Create() when you need identical vgrams.

Tokens

Word split on whitespace. Each token passes through TextFilter when normalize is true.

Unigrams

One stripped token per word. See Text normalization.

Vgrams

Overlapping word windows of width vgramSize, joined with a space, then stripped.

CreateVGrams("one two three four five", 3, true);
// ["onetwothree", "twothreefour", "threefourfive"]

Condition	Vgram list
tokens.length < vgramSize	[] (empty)
Enough tokens	One stripped vgram per window

Short text still fingerprints via tokens and unigrams inside Create().