tokenizeText

const tokenizeText: (text, grammar) => TokenStream

Defined in: prism/core.d.ts:112

This is the heart of Prism, and the most low-level function you can use. It accepts a string of text as input and the language definitions to use, and returns an array with the tokenized code.

When the language definition includes nested tokens, the function is called recursively on each of these tokens.

This method could be useful in other contexts as well, as a very crude parser.

Parameters

text

string

A string with the code to be highlighted.

grammar

Grammar

An object containing the tokens to use.

Usually a language definition like languages.markup.

Returns

TokenStream

An array of strings and tokens, a token stream.

Example

let code = `var foo = 0;`;
let tokens = tokenizeText(code, languages.javascript);
tokens.forEach(token => {
  if (token instanceof Token && token.type == 'number') {
    console.log(`Found numeric literal: ${token.content}`);
  }
});