DummyTokenizer
extends Tokenizer
in package
Dummy tokenizer class to allow for accessing the replaceTabsInToken() method.
Tags
Table of Contents
Properties
- $endScopeTokens : array<string|int, mixed>
- A list of tokens that end the scope.
- $ignoredLines : array<string|int, mixed>
- A list of lines being ignored due to error suppression comments.
- $knownLengths : array<string|int, int>
- Known lengths of tokens.
- $scopeOpeners : array<string|int, mixed>
- A list of tokens that are allowed to open a scope.
- $config : Config
- The config data for the run.
- $eolChar : string
- The EOL char used in the content.
- $numTokens : int
- The number of tokens in the tokens array.
- $tokens : array<string|int, mixed>
- A token-based representation of the content.
Methods
- __construct() : void
- Initialise and (don't) run the tokenizer.
- getTokens() : array<string|int, mixed>
- Gets the array of tokens.
- replaceTabsInToken() : void
- Replaces tabs in original token content with spaces.
- isMinifiedContent() : bool
- Checks the content to see if it looks minified.
- processAdditional() : void
- Performs additional processing after main tokenizing.
- tokenize() : array<int, array<string, mixed>>
- Creates an array of tokens when given some content.
Properties
$endScopeTokens
A list of tokens that end the scope.
public
array<string|int, mixed>
$endScopeTokens
= []
$ignoredLines
A list of lines being ignored due to error suppression comments.
public
array<string|int, mixed>
$ignoredLines
= []
$knownLengths
Known lengths of tokens.
public
array<string|int, int>
$knownLengths
= []
$scopeOpeners
A list of tokens that are allowed to open a scope.
public
array<string|int, mixed>
$scopeOpeners
= []
$config
The config data for the run.
protected
Config
$config
= null
$eolChar
The EOL char used in the content.
protected
string
$eolChar
= ''
$numTokens
The number of tokens in the tokens array.
protected
int
$numTokens
= 0
$tokens
A token-based representation of the content.
protected
array<string|int, mixed>
$tokens
= []
Methods
__construct()
Initialise and (don't) run the tokenizer.
public
__construct(string $content, Config|null $config[, string $eolChar = '
' ]) : void
Parameters
- $content : string
-
The content to tokenize,
- $config : Config|null
-
The config data for the run.
- $eolChar : string = ' '
-
The EOL char used in the content.
getTokens()
Gets the array of tokens.
public
getTokens() : array<string|int, mixed>
Return values
array<string|int, mixed>replaceTabsInToken()
Replaces tabs in original token content with spaces.
public
replaceTabsInToken(array<string|int, mixed> &$token[, string $prefix = ' ' ][, string $padding = ' ' ][, int $tabWidth = null ]) : void
Each tab can represent between 1 and $config->tabWidth spaces, so this cannot be a straight string replace. The original content is placed into an orig_content index and the new token length is also set in the length index.
Parameters
- $token : array<string|int, mixed>
-
The token to replace tabs inside.
- $prefix : string = ' '
-
The character to use to represent the start of a tab.
- $padding : string = ' '
-
The character to use to represent the end of a tab.
- $tabWidth : int = null
-
The number of spaces each tab represents.
isMinifiedContent()
Checks the content to see if it looks minified.
protected
isMinifiedContent(string $content[, string $eolChar = '
' ]) : bool
Parameters
- $content : string
-
The content to tokenize.
- $eolChar : string = ' '
-
The EOL char used in the content.
Return values
boolprocessAdditional()
Performs additional processing after main tokenizing.
protected
processAdditional() : void
tokenize()
Creates an array of tokens when given some content.
protected
tokenize(string $string) : array<int, array<string, mixed>>
Parameters
- $string : string
-
The string to tokenize.