A token stream is an array of strings and Token objects.
Token streams have to fulfill a few properties that are assumed by most functions (mostly internal ones) that process them.
No adjacent strings.
No empty strings.
The only exception here is the token stream that only contains the empty string and nothing else.
Generated using TypeDoc
A token stream is an array of strings and Token objects.
Token streams have to fulfill a few properties that are assumed by most functions (mostly internal ones) that process them.
No adjacent strings.
No empty strings.
The only exception here is the token stream that only contains the empty string and nothing else.