17.5 tokenize -- Tokenizer for Python source
Keywords: lib

# 17.5 `tokenize` -- Tokenizer for Python source

The `tokenize` module provides a lexical scanner for Python source code, implemented in Python. The scanner in this module returns comments as tokens as well, making it useful for implementing ``pretty-printers,'' including colorizers for on-screen displays.

The scanner is exposed by a single function:

`tokenize` (readline[, tokeneater])
  The `tokenize()` function accepts two parameters: one representing the input stream, and one providing an output mechanism for `tokenize()`.

  The first parameter, readline, must be a callable object which provides the same interface as the `readline()` method of built-in file objects (see section ). Each call to the function should return one line of input as a string.

  The second parameter, tokeneater, must also be a callable object. It is called with five parameters: the token type, the token string, a tuple `(srow, scol)` specifying the row and column where the token begins in the source, a tuple `(erow, ecol)` giving the ending position of the token, and the line on which the token was found. The line passed is the logical line; continuation lines are included.

All constants from the `token` module are also exported from `tokenize`, as is one additional token type value that might be passed to the tokeneater function by `tokenize()`:

`COMMENT`
  Token value used to indicate a comment.