Class: Parser::Base

Inherits:
Racc::Parser
  • Object
show all
Defined in:
lib/parser/base.rb

Overview

Base class for version-specific parsers.

Instance Attribute Summary (collapse)

Class Method Summary (collapse)

Instance Method Summary (collapse)

Constructor Details

- (Base) initialize(builder = Parser::Builders::Default.new)

Create a new parser instance.

Note that on JRuby and Rubinius the diagnostics.all_errors_are_fatal switch is always turned on as a workaround for a standard library bug. Do not set it to false.

Parameters:

  • builder (Parser::Builders::Default) (defaults to: Parser::Builders::Default.new) —

    The AST builder to use.



126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
 
# File 'lib/parser/base.rb', line 126

def initialize(builder=Parser::Builders::Default.new)
  @diagnostics = Diagnostic::Engine.new

  if RUBY_PLATFORM != 'ruby'
    # This is a workaround for a Racc bug. In the pure Ruby Racc runtime,
    # which gets activated on JRuby/RBX, there is a bug in error token
    # popping, which results in an infinite loop.
    @diagnostics.all_errors_are_fatal = true
  end

  @static_env  = StaticEnvironment.new

  @lexer = Lexer.new(version)
  @lexer.diagnostics = @diagnostics
  @lexer.static_env  = @static_env

  @builder = builder
  @builder.parser = self

  if self.class::Racc_debug_parser && ENV['RACC_DEBUG']
    @yydebug = true
  end

  reset
end

Instance Attribute Details

- (Object) builder (readonly) 



113
114
115
 
# File 'lib/parser/base.rb', line 113

def builder
  @builder
end

- (Parser::Diagnostic::Engine) diagnostics (readonly)

Returns:



14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
 
# File 'lib/parser/base.rb', line 14

class Base < Racc::Parser
  ##
  # Parses a string of Ruby code and returns the AST. If the source
  # cannot be parsed, {SyntaxError} is raised and a diagnostic is
  # printed to `stderr`.
  #
  # @example
  #  Parser::Base.parse('puts "hello"')
  #
  # @param [String] string The block of code to parse.
  # @param [String] file The name of the file the code originated from.
  # @param [Numeric] line The initial line number.
  # @return [Parser::AST::Node]
  #
  def self.parse(string, file='(string)', line=1)
    parser = default_parser
    source_buffer = setup_source_buffer(file, line, string, parser.default_encoding)
    parser.parse(source_buffer)
  end

  ##
  # Parses a string of Ruby code and returns the AST and comments. If the
  # source cannot be parsed, {SyntaxError} is raised and a diagnostic is
  # printed to `stderr`.
  #
  # @example
  #  Parser::Base.parse_with_comments('puts "hello"')
  #
  # @param [String] string The block of code to parse.
  # @param [String] file The name of the file the code originated from.
  # @param [Numeric] line The initial line number.
  # @return [Array]
  #
  def self.parse_with_comments(string, file='(string)', line=1)
    parser = default_parser
    source_buffer = setup_source_buffer(file, line, string, parser.default_encoding)
    parser.parse_with_comments(source_buffer)
  end

  ##
  # Parses Ruby source code by reading it from a file. If the source
  # cannot be parsed, {SyntaxError} is raised and a diagnostic is
  # printed to `stderr`.
  #
  # @param [String] filename Path to the file to parse.
  # @return [Parser::AST::Node]
  # @see #parse
  #
  def self.parse_file(filename)
    parse(File.read(filename), filename)
  end

  ##
  # Parses Ruby source code by reading it from a file and returns the AST and
  # comments. If the source cannot be parsed, {SyntaxError} is raised and a
  # diagnostic is printed to `stderr`.
  #
  # @param [String] filename Path to the file to parse.
  # @return [Array]
  # @see #parse
  #
  def self.parse_file_with_comments(filename)
    parse_with_comments(File.read(filename), filename)
  end

  ##
  # @return [Parser::Base] parser with the default options set.
  #
  def self.default_parser
    parser = new

    parser.diagnostics.all_errors_are_fatal = true
    parser.diagnostics.ignore_warnings      = true

    parser.diagnostics.consumer = lambda do |diagnostic|
      $stderr.puts(diagnostic.render)
    end

    parser
  end

  def self.setup_source_buffer(file, line, string, encoding)
    if string.respond_to? :force_encoding
      string = string.dup.force_encoding(encoding)
    end

    source_buffer = Source::Buffer.new(file, line)

    if name == 'Parser::Ruby18'
      source_buffer.raw_source = string
    else
      source_buffer.source     = string
    end

    source_buffer
  end
  private_class_method :setup_source_buffer

  attr_reader :diagnostics
  attr_reader :builder
  attr_reader :static_env
  attr_reader :source_buffer

  ##
  # Create a new parser instance.
  #
  # Note that on JRuby and Rubinius the `diagnostics.all_errors_are_fatal`
  # switch is always turned on as a workaround for a standard library bug.
  # Do not set it to `false`.
  #
  # @param [Parser::Builders::Default] builder The AST builder to use.
  #
  def initialize(builder=Parser::Builders::Default.new)
    @diagnostics = Diagnostic::Engine.new

    if RUBY_PLATFORM != 'ruby'
      # This is a workaround for a Racc bug. In the pure Ruby Racc runtime,
      # which gets activated on JRuby/RBX, there is a bug in error token
      # popping, which results in an infinite loop.
      @diagnostics.all_errors_are_fatal = true
    end

    @static_env  = StaticEnvironment.new

    @lexer = Lexer.new(version)
    @lexer.diagnostics = @diagnostics
    @lexer.static_env  = @static_env

    @builder = builder
    @builder.parser = self

    if self.class::Racc_debug_parser && ENV['RACC_DEBUG']
      @yydebug = true
    end

    reset
  end

  ##
  # Resets the state of the parser.
  #
  def reset
    @source_buffer = nil
    @def_level     = 0 # count of nested def's.

    @lexer.reset
    @static_env.reset

    self
  end

  ##
  # Parses a source buffer and returns the AST.
  #
  # @param [Parser::Source::Buffer] source_buffer The source buffer to parse.
  # @return [Parser::AST::Node]
  #
  def parse(source_buffer)
    @lexer.source_buffer = source_buffer
    @source_buffer       = source_buffer

    do_parse
  ensure
    # Don't keep references to the source file.
    @source_buffer       = nil
    @lexer.source_buffer = nil
  end

  ##
  # Parses a source buffer and returns the AST along with the comments of the
  # Ruby source code.
  #
  # @see #parse
  # @see Parser::Source::Comment#associate
  # @return [Array]
  #
  def parse_with_comments(source_buffer)
    @lexer.comments = []

    [ parse(source_buffer), @lexer.comments ]
  ensure
    @lexer.comments = nil
  end

  ##
  # Currently, token stream format returned by #tokenize is not documented,
  # but is considered part of a public API and only changed according
  # to Semantic Versioning.
  #
  # However, note that the exact token composition of various constructs
  # might vary. For example, a string `"foo"` is represented equally well
  # by `:tSTRING_BEG " :tSTRING_CONTENT foo :tSTRING_END "` and
  # `:tSTRING "foo"`; such details must not be relied upon.
  #
  # @param [Parser::Source::Buffer] source_buffer
  # @return [Array]
  #
  def tokenize(source_buffer)
    @lexer.tokens = []

    ast, comments = parse_with_comments(source_buffer)

    [ ast, comments, @lexer.tokens ]
  ensure
    @lexer.tokens = nil
  end

  ##
  # @api private
  # @return [Boolean]
  #
  def in_def?
    @def_level > 0
  end

  private

  def next_token
    @lexer.advance
  end

  def check_kwarg_name(name_t)
    case name_t[0]
    when /^[a-z_]/
      # OK
    when /^[A-Z]/
      diagnostic :error, :argument_const, nil, name_t
    end
  end

  def diagnostic(level, reason, arguments, location_t, highlights_ts=[])
    _, location = location_t

    highlights = highlights_ts.map do |token|
      _, range = token
      range
    end

    @diagnostics.process(
        Diagnostic.new(level, reason, arguments, location, highlights))

    if level == :error
      yyerror
    end
  end

  def on_error(error_token_id, error_value, value_stack)
    token_name = token_to_str(error_token_id)
    _, location = error_value

    @diagnostics.process(Diagnostic.new(
        :error, :unexpected_token, { :token => token_name }, location))
  end
end

- (Object) source_buffer (readonly) 



115
116
117
 
# File 'lib/parser/base.rb', line 115

def source_buffer
  @source_buffer
end

- (Parser::StaticEnvironment) static_env (readonly)

Returns:

  • (Parser::StaticEnvironment) 


14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
 
# File 'lib/parser/base.rb', line 14

class Base < Racc::Parser
  ##
  # Parses a string of Ruby code and returns the AST. If the source
  # cannot be parsed, {SyntaxError} is raised and a diagnostic is
  # printed to `stderr`.
  #
  # @example
  #  Parser::Base.parse('puts "hello"')
  #
  # @param [String] string The block of code to parse.
  # @param [String] file The name of the file the code originated from.
  # @param [Numeric] line The initial line number.
  # @return [Parser::AST::Node]
  #
  def self.parse(string, file='(string)', line=1)
    parser = default_parser
    source_buffer = setup_source_buffer(file, line, string, parser.default_encoding)
    parser.parse(source_buffer)
  end

  ##
  # Parses a string of Ruby code and returns the AST and comments. If the
  # source cannot be parsed, {SyntaxError} is raised and a diagnostic is
  # printed to `stderr`.
  #
  # @example
  #  Parser::Base.parse_with_comments('puts "hello"')
  #
  # @param [String] string The block of code to parse.
  # @param [String] file The name of the file the code originated from.
  # @param [Numeric] line The initial line number.
  # @return [Array]
  #
  def self.parse_with_comments(string, file='(string)', line=1)
    parser = default_parser
    source_buffer = setup_source_buffer(file, line, string, parser.default_encoding)
    parser.parse_with_comments(source_buffer)
  end

  ##
  # Parses Ruby source code by reading it from a file. If the source
  # cannot be parsed, {SyntaxError} is raised and a diagnostic is
  # printed to `stderr`.
  #
  # @param [String] filename Path to the file to parse.
  # @return [Parser::AST::Node]
  # @see #parse
  #
  def self.parse_file(filename)
    parse(File.read(filename), filename)
  end

  ##
  # Parses Ruby source code by reading it from a file and returns the AST and
  # comments. If the source cannot be parsed, {SyntaxError} is raised and a
  # diagnostic is printed to `stderr`.
  #
  # @param [String] filename Path to the file to parse.
  # @return [Array]
  # @see #parse
  #
  def self.parse_file_with_comments(filename)
    parse_with_comments(File.read(filename), filename)
  end

  ##
  # @return [Parser::Base] parser with the default options set.
  #
  def self.default_parser
    parser = new

    parser.diagnostics.all_errors_are_fatal = true
    parser.diagnostics.ignore_warnings      = true

    parser.diagnostics.consumer = lambda do |diagnostic|
      $stderr.puts(diagnostic.render)
    end

    parser
  end

  def self.setup_source_buffer(file, line, string, encoding)
    if string.respond_to? :force_encoding
      string = string.dup.force_encoding(encoding)
    end

    source_buffer = Source::Buffer.new(file, line)

    if name == 'Parser::Ruby18'
      source_buffer.raw_source = string
    else
      source_buffer.source     = string
    end

    source_buffer
  end
  private_class_method :setup_source_buffer

  attr_reader :diagnostics
  attr_reader :builder
  attr_reader :static_env
  attr_reader :source_buffer

  ##
  # Create a new parser instance.
  #
  # Note that on JRuby and Rubinius the `diagnostics.all_errors_are_fatal`
  # switch is always turned on as a workaround for a standard library bug.
  # Do not set it to `false`.
  #
  # @param [Parser::Builders::Default] builder The AST builder to use.
  #
  def initialize(builder=Parser::Builders::Default.new)
    @diagnostics = Diagnostic::Engine.new

    if RUBY_PLATFORM != 'ruby'
      # This is a workaround for a Racc bug. In the pure Ruby Racc runtime,
      # which gets activated on JRuby/RBX, there is a bug in error token
      # popping, which results in an infinite loop.
      @diagnostics.all_errors_are_fatal = true
    end

    @static_env  = StaticEnvironment.new

    @lexer = Lexer.new(version)
    @lexer.diagnostics = @diagnostics
    @lexer.static_env  = @static_env

    @builder = builder
    @builder.parser = self

    if self.class::Racc_debug_parser && ENV['RACC_DEBUG']
      @yydebug = true
    end

    reset
  end

  ##
  # Resets the state of the parser.
  #
  def reset
    @source_buffer = nil
    @def_level     = 0 # count of nested def's.

    @lexer.reset
    @static_env.reset

    self
  end

  ##
  # Parses a source buffer and returns the AST.
  #
  # @param [Parser::Source::Buffer] source_buffer The source buffer to parse.
  # @return [Parser::AST::Node]
  #
  def parse(source_buffer)
    @lexer.source_buffer = source_buffer
    @source_buffer       = source_buffer

    do_parse
  ensure
    # Don't keep references to the source file.
    @source_buffer       = nil
    @lexer.source_buffer = nil
  end

  ##
  # Parses a source buffer and returns the AST along with the comments of the
  # Ruby source code.
  #
  # @see #parse
  # @see Parser::Source::Comment#associate
  # @return [Array]
  #
  def parse_with_comments(source_buffer)
    @lexer.comments = []

    [ parse(source_buffer), @lexer.comments ]
  ensure
    @lexer.comments = nil
  end

  ##
  # Currently, token stream format returned by #tokenize is not documented,
  # but is considered part of a public API and only changed according
  # to Semantic Versioning.
  #
  # However, note that the exact token composition of various constructs
  # might vary. For example, a string `"foo"` is represented equally well
  # by `:tSTRING_BEG " :tSTRING_CONTENT foo :tSTRING_END "` and
  # `:tSTRING "foo"`; such details must not be relied upon.
  #
  # @param [Parser::Source::Buffer] source_buffer
  # @return [Array]
  #
  def tokenize(source_buffer)
    @lexer.tokens = []

    ast, comments = parse_with_comments(source_buffer)

    [ ast, comments, @lexer.tokens ]
  ensure
    @lexer.tokens = nil
  end

  ##
  # @api private
  # @return [Boolean]
  #
  def in_def?
    @def_level > 0
  end

  private

  def next_token
    @lexer.advance
  end

  def check_kwarg_name(name_t)
    case name_t[0]
    when /^[a-z_]/
      # OK
    when /^[A-Z]/
      diagnostic :error, :argument_const, nil, name_t
    end
  end

  def diagnostic(level, reason, arguments, location_t, highlights_ts=[])
    _, location = location_t

    highlights = highlights_ts.map do |token|
      _, range = token
      range
    end

    @diagnostics.process(
        Diagnostic.new(level, reason, arguments, location, highlights))

    if level == :error
      yyerror
    end
  end

  def on_error(error_token_id, error_value, value_stack)
    token_name = token_to_str(error_token_id)
    _, location = error_value

    @diagnostics.process(Diagnostic.new(
        :error, :unexpected_token, { :token => token_name }, location))
  end
end

Class Method Details

+ (Parser::Base) default_parser

Returns parser with the default options set.

Returns:



82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/parser/base.rb', line 82

def self.default_parser
  parser = new

  parser.diagnostics.all_errors_are_fatal = true
  parser.diagnostics.ignore_warnings      = true

  parser.diagnostics.consumer = lambda do |diagnostic|
    $stderr.puts(diagnostic.render)
  end

  parser
end

+ (Parser::AST::Node) parse(string, file = '(string)', line = 1)

Parses a string of Ruby code and returns the AST. If the source cannot be parsed, SyntaxError is raised and a diagnostic is printed to stderr.

Examples:

Parser::Base.parse('puts "hello"')

Parameters:

  • string (String) —

    The block of code to parse.

  • file (String) (defaults to: '(string)') —

    The name of the file the code originated from.

  • line (Numeric) (defaults to: 1) —

    The initial line number.

Returns:



28
29
30
31
32
 
# File 'lib/parser/base.rb', line 28

def self.parse(string, file='(string)', line=1)
  parser = default_parser
  source_buffer = setup_source_buffer(file, line, string, parser.default_encoding)
  parser.parse(source_buffer)
end

+ (Parser::AST::Node) parse_file(filename)

Parses Ruby source code by reading it from a file. If the source cannot be parsed, SyntaxError is raised and a diagnostic is printed to stderr.

Parameters:

  • filename (String) —

    Path to the file to parse.

Returns:

See Also:



62
63
64
 
# File 'lib/parser/base.rb', line 62

def self.parse_file(filename)
  parse(File.read(filename), filename)
end

+ (Array) parse_file_with_comments(filename)

Parses Ruby source code by reading it from a file and returns the AST and comments. If the source cannot be parsed, SyntaxError is raised and a diagnostic is printed to stderr.

Parameters:

  • filename (String) —

    Path to the file to parse.

Returns:

  • (Array)

See Also:



75
76
77
 
# File 'lib/parser/base.rb', line 75

def self.parse_file_with_comments(filename)
  parse_with_comments(File.read(filename), filename)
end

+ (Array) parse_with_comments(string, file = '(string)', line = 1)

Parses a string of Ruby code and returns the AST and comments. If the source cannot be parsed, SyntaxError is raised and a diagnostic is printed to stderr.

Examples:

Parser::Base.parse_with_comments('puts "hello"')

Parameters:

  • string (String) —

    The block of code to parse.

  • file (String) (defaults to: '(string)') —

    The name of the file the code originated from.

  • line (Numeric) (defaults to: 1) —

    The initial line number.

Returns:

  • (Array) 


47
48
49
50
51
 
# File 'lib/parser/base.rb', line 47

def self.parse_with_comments(string, file='(string)', line=1)
  parser = default_parser
  source_buffer = setup_source_buffer(file, line, string, parser.default_encoding)
  parser.parse_with_comments(source_buffer)
end

Instance Method Details

- (Parser::AST::Node) parse(source_buffer)

Parses a source buffer and returns the AST.

Parameters:

Returns:



171
172
173
174
175
176
177
178
179
180
 
# File 'lib/parser/base.rb', line 171

def parse(source_buffer)
  @lexer.source_buffer = source_buffer
  @source_buffer       = source_buffer

  do_parse
ensure
  # Don't keep references to the source file.
  @source_buffer       = nil
  @lexer.source_buffer = nil
end

- (Array) parse_with_comments(source_buffer)

Parses a source buffer and returns the AST along with the comments of the Ruby source code.

Returns:

  • (Array)

See Also:

  • #parse
  • Source::Comment#associate


190
191
192
193
194
195
196
 
# File 'lib/parser/base.rb', line 190

def parse_with_comments(source_buffer)
  @lexer.comments = []

  [ parse(source_buffer), @lexer.comments ]
ensure
  @lexer.comments = nil
end

- (Object) reset

Resets the state of the parser.



155
156
157
158
159
160
161
162
163
 
# File 'lib/parser/base.rb', line 155

def reset
  @source_buffer = nil
  @def_level     = 0 # count of nested def's.

  @lexer.reset
  @static_env.reset

  self
end

- (Array) tokenize(source_buffer)

Currently, token stream format returned by #tokenize is not documented, but is considered part of a public API and only changed according to Semantic Versioning.

However, note that the exact token composition of various constructs might vary. For example, a string "foo" is represented equally well by :tSTRING_BEG " :tSTRING_CONTENT foo :tSTRING_END " and :tSTRING "foo"; such details must not be relied upon.

Parameters:

Returns:

  • (Array) 


211
212
213
214
215
216
217
218
219
 
# File 'lib/parser/base.rb', line 211

def tokenize(source_buffer)
  @lexer.tokens = []

  ast, comments = parse_with_comments(source_buffer)

  [ ast, comments, @lexer.tokens ]
ensure
  @lexer.tokens = nil
end