Tokenizer
in package
Table of Contents
Properties
- $endScopeTokens : array<int|string, int|string>
- A list of tokens that end the scope.
- $ignoredLines : array<string|int, mixed>
- A list of lines being ignored due to error suppression comments.
- $knownLengths : array<string|int, int>
- Known lengths of tokens.
- $scopeOpeners : array<string|int, mixed>
- A list of tokens that are allowed to open a scope.
- $config : Config
- The config data for the run.
- $eolChar : string
- The EOL char used in the content.
- $numTokens : int
- The number of tokens in the tokens array.
- $tokens : array<string|int, mixed>
- A token-based representation of the content.
Methods
- __construct() : void
- Initialise and run the tokenizer.
- getTokens() : array<string|int, mixed>
- Gets the array of tokens.
- replaceTabsInToken() : void
- Replaces tabs in original token content with spaces.
- isMinifiedContent() : bool
- Checks the content to see if it looks minified.
- processAdditional() : void
- Performs additional processing after main tokenizing.
- tokenize() : array<string|int, mixed>
- Creates an array of tokens when given some content.
- createLevelMap() : void
- Constructs the level map.
- createParenthesisNestingMap() : void
- Creates a map for the parenthesis tokens that surround other tokens.
- createPositionMap() : void
- Sets token position information.
- createScopeMap() : void
- Creates a scope map of tokens that open scopes.
- createTokenMap() : void
- Creates a map of brackets positions.
- recurseScopeMap() : int
- Recurses though the scope openers to build a scope map.
Properties
$endScopeTokens
A list of tokens that end the scope.
public
array<int|string, int|string>
$endScopeTokens
= []
$ignoredLines
A list of lines being ignored due to error suppression comments.
public
array<string|int, mixed>
$ignoredLines
= []
$knownLengths
Known lengths of tokens.
public
array<string|int, int>
$knownLengths
= []
$scopeOpeners
A list of tokens that are allowed to open a scope.
public
array<string|int, mixed>
$scopeOpeners
= []
$config
The config data for the run.
protected
Config
$config
= null
$eolChar
The EOL char used in the content.
protected
string
$eolChar
= ''
$numTokens
The number of tokens in the tokens array.
protected
int
$numTokens
= 0
$tokens
A token-based representation of the content.
protected
array<string|int, mixed>
$tokens
= []
Methods
__construct()
Initialise and run the tokenizer.
public
__construct(string $content, Config|null $config[, string $eolChar = '\n' ]) : void
Parameters
- $content : string
-
The content to tokenize.
- $config : Config|null
-
The config data for the run.
- $eolChar : string = '\n'
-
The EOL char used in the content.
Tags
getTokens()
Gets the array of tokens.
public
getTokens() : array<string|int, mixed>
Return values
array<string|int, mixed>replaceTabsInToken()
Replaces tabs in original token content with spaces.
public
replaceTabsInToken(array<string|int, mixed> &$token[, string $prefix = ' ' ][, string $padding = ' ' ][, int|null $tabWidth = null ]) : void
Each tab can represent between 1 and $config->tabWidth spaces, so this cannot be a straight string replace. The original content is placed into an orig_content index and the new token length is also set in the length index.
Parameters
- $token : array<string|int, mixed>
-
The token to replace tabs inside.
- $prefix : string = ' '
-
The character to use to represent the start of a tab.
- $padding : string = ' '
-
The character to use to represent the end of a tab.
- $tabWidth : int|null = null
-
The number of spaces each tab represents.
isMinifiedContent()
Checks the content to see if it looks minified.
protected
isMinifiedContent(string $content[, string $eolChar = '\n' ]) : bool
Parameters
- $content : string
-
The content to tokenize.
- $eolChar : string = '\n'
-
The EOL char used in the content.
Return values
boolprocessAdditional()
Performs additional processing after main tokenizing.
protected
abstract processAdditional() : void
tokenize()
Creates an array of tokens when given some content.
protected
abstract tokenize(string $string) : array<string|int, mixed>
Parameters
- $string : string
-
The string to tokenize.
Return values
array<string|int, mixed>createLevelMap()
Constructs the level map.
private
createLevelMap() : void
The level map adds a 'level' index to each token which indicates the depth that a token within a set of scope blocks. It also adds a 'conditions' index which is an array of the scope conditions that opened each of the scopes - position 0 being the first scope opener.
createParenthesisNestingMap()
Creates a map for the parenthesis tokens that surround other tokens.
private
createParenthesisNestingMap() : void
createPositionMap()
Sets token position information.
private
createPositionMap() : void
Can also convert tabs into spaces. Each tab can represent between 1 and $width spaces, so this cannot be a straight string replace.
createScopeMap()
Creates a scope map of tokens that open scopes.
private
createScopeMap() : void
Tags
createTokenMap()
Creates a map of brackets positions.
private
createTokenMap() : void
recurseScopeMap()
Recurses though the scope openers to build a scope map.
private
recurseScopeMap(int $stackPtr[, int $depth = 1 ][, int &$ignore = 0 ]) : int
Parameters
- $stackPtr : int
-
The position in the stack of the token that opened the scope (eg. an IF token or FOR token).
- $depth : int = 1
-
How many scope levels down we are.
- $ignore : int = 0
-
How many curly braces we are ignoring.
Tags
Return values
int —The position in the stack that closed the scope.