A simple implementation of an YAML lexer. It handles most cases. It is not a validating lexer.
Methods
Public Instance methods
Step through a single iteration of the tokenization process. This will yield (potentially) many tokens, and possibly zero tokens.
[ show source ]
# File lib/syntax/lang/yaml.rb, line 11
11: def step
12: if bol?
13: case
14: when scan(/---(\s*.+)?$/)
15: start_group :document, matched
16: when scan(/(\s*)([a-zA-Z][-\w]*)(\s*):/)
17: start_group :normal, subgroup(1)
18: start_group :key, subgroup(2)
19: start_group :normal, subgroup(3)
20: start_group :punct, ":"
21: when scan(/(\s*)-/)
22: start_group :normal, subgroup(1)
23: start_group :punct, "-"
24: when scan(/\s*$/)
25: start_group :normal, matched
26: when scan(/#.*$/)
27: start_group :comment, matched
28: else
29: append getch
30: end
31: else
32: case
33: when scan(/[\n\r]+/)
34: start_group :normal, matched
35: when scan(/[ \t]+/)
36: start_group :normal, matched
37: when scan(/!+(.*?^)?\S+/)
38: start_group :type, matched
39: when scan(/&\S+/)
40: start_group :anchor, matched
41: when scan(/\*\S+/)
42: start_group :ref, matched
43: when scan(/\d\d:\d\d:\d\d/)
44: start_group :time, matched
45: when scan(/\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d(\.\d+)? [-+]\d\d:\d\d/)
46: start_group :date, matched
47: when scan(/['"]/)
48: start_group :punct, matched
49: scan_string matched
50: when scan(/:\w+/)
51: start_group :symbol, matched
52: when scan(/[:]/)
53: start_group :punct, matched
54: when scan(/#.*$/)
55: start_group :comment, matched
56: when scan(/>-?/)
57: start_group :punct, matched
58: start_group :normal, scan(/.*$/)
59: append getch until eos? || bol?
60: return if eos?
61: indent = check(/ */)
62: start_group :string
63: loop do
64: line = check_until(/[\n\r]|\Z/)
65: break if line.nil?
66: if line.chomp.length > 0
67: this_indent = line.chomp.match( /^\s*/ )[0]
68: break if this_indent.length < indent.length
69: end
70: append scan_until(/[\n\r]|\Z/)
71: end
72: else
73: start_group :normal, scan_until(/(?=$|#)/)
74: end
75: end
76: end