The possible token classes with examples:
   ---------------------------------------
   | Input Example  Token    Class        |
   |------------------------------------  |
   | X              "X"      var          |
   | _              "_"      anonymous    |
   | abc            'abc'    atom         |
   | 'a-b'          'a-b'    quoted_atom  |
   | 123            123      integer      |
   | 1.2            1.2      float        |
   | 1_3            1_3      rational     |
   | 0.9__1.1       0.9__1.1 breal        |
   | "abc"          "abc"    string       |
   | |              "|"      solo         |
   | )              ")"      solo         |
   | (              ")"      solo         |
   | <SPACE>(       "("      open_par     |
   | ,              ','      comma        |
   | .<NL>          '.'      fullstop     |
   | 1e789<NL>      "1e789"  error        |
   ---------------------------------------|
   Note that round, square and curly brackets are solo tokens whose
   value is returned as a string.  Opening brackets preceded by space
   are treated specially as open_par tokens.  Comma and fullstop have
   their own token class.  All syntax errors are reported as class
   error, with the input string up to the error as Token.  The default
   error handler for the event 190 (reading EOF) returns end_of_file
   in both Class and Token.
Note about signed numbers: the tokenizer returns a sign followed by a number as two separate tokens. For instance, the input "-5" is read as two tokens, the atom '-' and the integer 5. In the case of bounded reals, this leads to "-1.1__-0.9" being returned as the atom '-' and the bounded real 1.1__-0.9. This is a non-canonical breal, since the upper bound is lower than the lower bound. read_token/2,3 is the only predicate that can produce such objects, and it is the programmer's responsibility to construct a valid breal using breal_bounds/3 and breal_from_bounds/3, taking the sign into account. Note that all other arithmetic operations are undefined on non-canonical breals.
Success:
      [eclipse 1]: read_token(input,T,C).
              []
      T = []
      C = atom
      [eclipse 2]: read_token(input,T,C).
              [
      T = "["
      C = solo
      [eclipse 3]: read_token(input, "k",C).
              "k"
      C = string
      [eclipse 4]: read_token(input,T,C).
              X
      T = "X"
      C = var
      [eclipse 5]: read_token(input,T,C).
              1.6e-5.
      T = 1.6e-05
      C = float
Fail:
      [eclipse 6]: read_token(input, "[", C).
              &
      no.
Error:
      [eclipse 7]: read_token(input, T, C).
              ^D
      T = end_of_file
      C = end_of_file
      yes. (Error 190, default handler)
      read_token(S, a(b,c), C).         (Error 4).
      read_token("string", a(b,c), C).  (Error 5).
      read_token(9, X + 2, C).          (Error 192). % stream not open
      read_token(atom, X + 2, C).       (Error 193).