Function tokenizeText

This is the heart of Prism, and the most low-level function you can use. It accepts a string of text as input and the language definitions to use, and returns an array with the tokenized code.

When the language definition includes nested tokens, the function is called recursively on each of these tokens.

This method could be useful in other contexts as well, as a very crude parser.

Example

let code = `var foo = 0;`;
let tokens = tokenizeText(code, languages.javascript);
tokens.forEach(token => {
  if (token instanceof Token && token.type == 'number') {
    console.log(`Found numeric literal: ${token.content}`);
  }
});
Copy