RussianLetterTokenizer (Lucene 2.4.1 API)
Keywords: org.apache.lucene.analysis.ru.RussianLetterTokenizer class

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.lucene.analysis.ru
Class RussianLetterTokenizer

    java.lang.Object
      org.apache.lucene.analysis.TokenStream
          org.apache.lucene.analysis.Tokenizer
              org.apache.lucene.analysis.CharTokenizer
                  org.apache.lucene.analysis.ru.RussianLetterTokenizer

    public class RussianLetterTokenizer

extends CharTokenizer

A RussianLetterTokenizer is a tokenizer that extends LetterTokenizer by additionally looking up letters in a given "russian charset". The problem with LeterTokenizer is that it uses Character.isLetter() method, which doesn't know how to detect letters in encodings like CP1252 and KOI8 (well-known problems with 0xD7 and 0xF7 chars)

Version:
  $Id: RussianLetterTokenizer.java 564236 2007-08-09 15:21:19Z gsingers $

Field Summary

Fields inherited from class org.apache.lucene.analysis.Tokenizer
`input`

Constructor Summary
`RussianLetterTokenizer(Reader in, char[] charset)`

Method Summary
`protected boolean` | `isTokenChar(char c)` Collects only characters which satisfy `Character.isLetter(char)`.

Methods inherited from class org.apache.lucene.analysis.CharTokenizer
`next, normalize, reset`

Methods inherited from class org.apache.lucene.analysis.Tokenizer
`close`

Methods inherited from class org.apache.lucene.analysis.TokenStream
`next, reset`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### RussianLetterTokenizer

    public RussianLetterTokenizer(Reader in,
                                  char[] charset)

Method Detail

### isTokenChar

    protected boolean isTokenChar(char c)

  Collects only characters which satisfy `Character.isLetter(char)`.

  Specified by:
    `isTokenChar` in class `CharTokenizer`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2000-2009 Apache Software Foundation. All Rights Reserved.