A single token extracted by a tokenizer. It is simply the lexeme itself, decorated with a ‘group’ attribute to identify the type of the lexeme.
Methods
Attributes
| [R] | group | the type of the lexeme that was extracted. |
| [R] | instruction | the instruction associated with this token (:none, :region_open, or :region_close) |
Public Class methods
Create a new Token representing the given text, and belonging to the given group.
[ show source ]
# File lib/syntax/common.rb, line 19
19: def initialize( text, group, instruction = :none )
20: super text
21: @group = group
22: @instruction = instruction
23: end