Uses of Class org.apache.lucene.analysis.Token (Lucene 2.4.1 API)

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV NEXT | FRAMES NO FRAMES

## Uses of Class
org.apache.lucene.analysis.Token

Packages that use Token
org.apache.lucene.analysis | API and code to convert text into indexable/searchable tokens.
org.apache.lucene.analysis.br | Analyzer for Brazilian.
org.apache.lucene.analysis.cjk | Analyzer for Chinese, Japanese and Korean.
org.apache.lucene.analysis.cn | Analyzer for Chinese.
org.apache.lucene.analysis.compound | A filter that decomposes compound words you find in many Germanic languages to the word parts.
org.apache.lucene.analysis.de | Analyzer for German.
org.apache.lucene.analysis.el | Analyzer for Greek.
org.apache.lucene.analysis.fr | Analyzer for French.
org.apache.lucene.analysis.miscellaneous |
org.apache.lucene.analysis.ngram |
org.apache.lucene.analysis.nl | Analyzer for Dutch.
org.apache.lucene.analysis.payloads | Provides various convenience classes for creating payloads on Tokens.
org.apache.lucene.analysis.ru | Analyzer for Russian.
org.apache.lucene.analysis.shingle |
org.apache.lucene.analysis.sinks | Implementations of the SinkTokenizer that might be useful.
org.apache.lucene.analysis.snowball | `TokenFilter` and `Analyzer` implementations that use Snowball stemmers.
org.apache.lucene.analysis.standard | A fast grammar-based tokenizer constructed with JFlex.
org.apache.lucene.analysis.th |
org.apache.lucene.index.memory | High-performance single-document main memory Apache Lucene fulltext search index.
org.apache.lucene.search.highlight | The highlight package contains classes to provide "keyword in context" features typically used to highlight search terms in the text of results pages.
org.apache.lucene.wikipedia.analysis |

Uses of Token in org.apache.lucene.analysis

Methods in org.apache.lucene.analysis that return Token
`Token` | `Token.clone(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)` Makes a clone, but replaces the term buffer & start/end offset in the process.
`Token` | `TokenStream.next()` Deprecated. The returned Token is a "full private copy" (not re-used across calls to next()) but will be slower than calling `TokenStream.next(Token)` instead..
`Token` | `LowerCaseFilter.next(Token reusableToken)`
`Token` | `SinkTokenizer.next(Token reusableToken)` Returns the next token out of the list of cached tokens
`Token` | `CharTokenizer.next(Token reusableToken)`
`Token` | `CachingTokenFilter.next(Token reusableToken)`
`Token` | `LengthFilter.next(Token reusableToken)` Returns the next input Token whose term() is the right len
`Token` | `TeeTokenFilter.next(Token reusableToken)`
`Token` | `PorterStemFilter.next(Token reusableToken)`
`Token` | `ISOLatin1AccentFilter.next(Token reusableToken)`
`Token` | `TokenStream.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.
`Token` | `StopFilter.next(Token reusableToken)` Returns the next input Token whose term() is not a stop word.
`Token` | `KeywordTokenizer.next(Token reusableToken)`
`Token` | `Token.reinit(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)` Shorthand for calling `clear()`, `setTermBuffer(char[], int, int)`, `setStartOffset(int)`, `setEndOffset(int)` `setType(java.lang.String)` on Token.DEFAULT_TYPE
`Token` | `Token.reinit(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset, String newType)` Shorthand for calling `clear()`, `setTermBuffer(char[], int, int)`, `setStartOffset(int)`, `setEndOffset(int)`, `setType(java.lang.String)`
`Token` | `Token.reinit(String newTerm, int newStartOffset, int newEndOffset)` Shorthand for calling `clear()`, `setTermBuffer(String)`, `setStartOffset(int)`, `setEndOffset(int)` `setType(java.lang.String)` on Token.DEFAULT_TYPE
`Token` | `Token.reinit(String newTerm, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)` Shorthand for calling `clear()`, `setTermBuffer(String, int, int)`, `setStartOffset(int)`, `setEndOffset(int)` `setType(java.lang.String)` on Token.DEFAULT_TYPE
`Token` | `Token.reinit(String newTerm, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset, String newType)` Shorthand for calling `clear()`, `setTermBuffer(String, int, int)`, `setStartOffset(int)`, `setEndOffset(int)` `setType(java.lang.String)`
`Token` | `Token.reinit(String newTerm, int newStartOffset, int newEndOffset, String newType)` Shorthand for calling `clear()`, `setTermBuffer(String)`, `setStartOffset(int)`, `setEndOffset(int)` `setType(java.lang.String)`

Methods in org.apache.lucene.analysis with parameters of type Token
`void` | `SinkTokenizer.add(Token t)` Override this method to cache only certain tokens, or new tokens based on the old tokens.
`Token` | `LowerCaseFilter.next(Token reusableToken)`
`Token` | `SinkTokenizer.next(Token reusableToken)` Returns the next token out of the list of cached tokens
`Token` | `CharTokenizer.next(Token reusableToken)`
`Token` | `CachingTokenFilter.next(Token reusableToken)`
`Token` | `LengthFilter.next(Token reusableToken)` Returns the next input Token whose term() is the right len
`Token` | `TeeTokenFilter.next(Token reusableToken)`
`Token` | `PorterStemFilter.next(Token reusableToken)`
`Token` | `ISOLatin1AccentFilter.next(Token reusableToken)`
`Token` | `TokenStream.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.
`Token` | `StopFilter.next(Token reusableToken)` Returns the next input Token whose term() is not a stop word.
`Token` | `KeywordTokenizer.next(Token reusableToken)`
`void` | `Token.reinit(Token prototype)` Copy the prototype token's fields into this one.
`void` | `Token.reinit(Token prototype, char[] newTermBuffer, int offset, int length)` Copy the prototype token's fields into this one, with a different term.
`void` | `Token.reinit(Token prototype, String newTerm)` Copy the prototype token's fields into this one, with a different term.

Uses of Token in org.apache.lucene.analysis.br

Methods in org.apache.lucene.analysis.br that return Token
`Token` | `BrazilianStemFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.br with parameters of type Token
`Token` | `BrazilianStemFilter.next(Token reusableToken)`

Uses of Token in org.apache.lucene.analysis.cjk

Methods in org.apache.lucene.analysis.cjk that return Token
`Token` | `CJKTokenizer.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.

Methods in org.apache.lucene.analysis.cjk with parameters of type Token
`Token` | `CJKTokenizer.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.

Uses of Token in org.apache.lucene.analysis.cn

Methods in org.apache.lucene.analysis.cn that return Token
`Token` | `ChineseTokenizer.next(Token reusableToken)`
`Token` | `ChineseFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.cn with parameters of type Token
`Token` | `ChineseTokenizer.next(Token reusableToken)`
`Token` | `ChineseFilter.next(Token reusableToken)`

Uses of Token in org.apache.lucene.analysis.compound

Methods in org.apache.lucene.analysis.compound that return Token
`protected Token` | `CompoundWordTokenFilterBase.createToken(int offset, int length, Token prototype)`
`Token` | `CompoundWordTokenFilterBase.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.compound with parameters of type Token
`protected Token` | `CompoundWordTokenFilterBase.createToken(int offset, int length, Token prototype)`
`protected void` | `CompoundWordTokenFilterBase.decompose(Token token)`
`protected abstract void` | `CompoundWordTokenFilterBase.decomposeInternal(Token token)`
`protected void` | `DictionaryCompoundWordTokenFilter.decomposeInternal(Token token)`
`protected void` | `HyphenationCompoundWordTokenFilter.decomposeInternal(Token token)`
`Token` | `CompoundWordTokenFilterBase.next(Token reusableToken)`

Uses of Token in org.apache.lucene.analysis.de

Methods in org.apache.lucene.analysis.de that return Token
`Token` | `GermanStemFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.de with parameters of type Token
`Token` | `GermanStemFilter.next(Token reusableToken)`

Uses of Token in org.apache.lucene.analysis.el

Methods in org.apache.lucene.analysis.el that return Token
`Token` | `GreekLowerCaseFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.el with parameters of type Token
`Token` | `GreekLowerCaseFilter.next(Token reusableToken)`

Uses of Token in org.apache.lucene.analysis.fr

Methods in org.apache.lucene.analysis.fr that return Token
`Token` | `FrenchStemFilter.next(Token reusableToken)`
`Token` | `ElisionFilter.next(Token reusableToken)` Returns the next input Token with term() without elisioned start

Methods in org.apache.lucene.analysis.fr with parameters of type Token
`Token` | `FrenchStemFilter.next(Token reusableToken)`
`Token` | `ElisionFilter.next(Token reusableToken)` Returns the next input Token with term() without elisioned start

Uses of Token in org.apache.lucene.analysis.miscellaneous

Methods in org.apache.lucene.analysis.miscellaneous that return Token
`Token` | `SingleTokenTokenStream.getToken()`
`Token` | `EmptyTokenStream.next(Token reusableToken)`
`Token` | `SingleTokenTokenStream.next(Token reusableToken)`
`Token` | `PrefixAndSuffixAwareTokenFilter.next(Token reusableToken)`
`Token` | `PrefixAwareTokenFilter.next(Token reusableToken)`
`Token` | `PrefixAndSuffixAwareTokenFilter.updateInputToken(Token inputToken, Token lastPrefixToken)`
`Token` | `PrefixAndSuffixAwareTokenFilter.updateSuffixToken(Token suffixToken, Token lastInputToken)`
`Token` | `PrefixAwareTokenFilter.updateSuffixToken(Token suffixToken, Token lastPrefixToken)` The default implementation adds last prefix token end offset to the suffix token start and end offsets.

Methods in org.apache.lucene.analysis.miscellaneous with parameters of type Token
`Token` | `EmptyTokenStream.next(Token reusableToken)`
`Token` | `SingleTokenTokenStream.next(Token reusableToken)`
`Token` | `PrefixAndSuffixAwareTokenFilter.next(Token reusableToken)`
`Token` | `PrefixAwareTokenFilter.next(Token reusableToken)`
`void` | `SingleTokenTokenStream.setToken(Token token)`
`Token` | `PrefixAndSuffixAwareTokenFilter.updateInputToken(Token inputToken, Token lastPrefixToken)`
`Token` | `PrefixAndSuffixAwareTokenFilter.updateSuffixToken(Token suffixToken, Token lastInputToken)`
`Token` | `PrefixAwareTokenFilter.updateSuffixToken(Token suffixToken, Token lastPrefixToken)` The default implementation adds last prefix token end offset to the suffix token start and end offsets.

Constructors in org.apache.lucene.analysis.miscellaneous with parameters of type Token
`SingleTokenTokenStream(Token token)`

Uses of Token in org.apache.lucene.analysis.ngram

Methods in org.apache.lucene.analysis.ngram that return Token
`Token` | `EdgeNGramTokenFilter.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.
`Token` | `EdgeNGramTokenizer.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.
`Token` | `NGramTokenizer.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.
`Token` | `NGramTokenFilter.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.

Methods in org.apache.lucene.analysis.ngram with parameters of type Token
`Token` | `EdgeNGramTokenFilter.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.
`Token` | `EdgeNGramTokenizer.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.
`Token` | `NGramTokenizer.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.
`Token` | `NGramTokenFilter.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.

Uses of Token in org.apache.lucene.analysis.nl

Methods in org.apache.lucene.analysis.nl that return Token
`Token` | `DutchStemFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.nl with parameters of type Token
`Token` | `DutchStemFilter.next(Token reusableToken)`

Uses of Token in org.apache.lucene.analysis.payloads

Methods in org.apache.lucene.analysis.payloads that return Token
`Token` | `TokenOffsetPayloadTokenFilter.next(Token reusableToken)`
`Token` | `TypeAsPayloadTokenFilter.next(Token reusableToken)`
`Token` | `NumericPayloadTokenFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.payloads with parameters of type Token
`Token` | `TokenOffsetPayloadTokenFilter.next(Token reusableToken)`
`Token` | `TypeAsPayloadTokenFilter.next(Token reusableToken)`
`Token` | `NumericPayloadTokenFilter.next(Token reusableToken)`

Uses of Token in org.apache.lucene.analysis.ru

Methods in org.apache.lucene.analysis.ru that return Token
`Token` | `RussianStemFilter.next(Token reusableToken)`
`Token` | `RussianLowerCaseFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.ru with parameters of type Token
`Token` | `RussianStemFilter.next(Token reusableToken)`
`Token` | `RussianLowerCaseFilter.next(Token reusableToken)`

Uses of Token in org.apache.lucene.analysis.shingle

Methods in org.apache.lucene.analysis.shingle that return Token
`Token` | `ShingleMatrixFilter.next(Token reusableToken)`
`Token` | `ShingleFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.shingle with parameters of type Token
`float` | `ShingleMatrixFilter.calculateShingleWeight(Token shingleToken, List shingle, int currentPermutationStartOffset, List currentPermutationRows, List currentPermuationTokens)` Evaluates the new shingle token weight.
`abstract ShingleMatrixFilter.TokenPositioner` | `ShingleMatrixFilter.TokenSettingsCodec.getTokenPositioner(Token token)` Retrieves information on how a `Token` is to be inserted to a `ShingleMatrixFilter.Matrix`.
`ShingleMatrixFilter.TokenPositioner` | `ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec.getTokenPositioner(Token token)`
`ShingleMatrixFilter.TokenPositioner` | `ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec.getTokenPositioner(Token token)`
`ShingleMatrixFilter.TokenPositioner` | `ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec.getTokenPositioner(Token token)`
`abstract float` | `ShingleMatrixFilter.TokenSettingsCodec.getWeight(Token token)` Have this method return 1f in order to 'disable' weights.
`float` | `ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec.getWeight(Token token)`
`float` | `ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec.getWeight(Token token)`
`float` | `ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec.getWeight(Token token)` Returns a 32 bit float from the payload, or 1f it null.
`Token` | `ShingleMatrixFilter.next(Token reusableToken)`
`Token` | `ShingleFilter.next(Token reusableToken)`
`abstract void` | `ShingleMatrixFilter.TokenSettingsCodec.setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)` Sets information on how a `Token` is to be inserted to a `ShingleMatrixFilter.Matrix`.
`void` | `ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec.setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)`
`void` | `ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec.setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)`
`void` | `ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec.setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)` Sets the TokenPositioner as token flags int value.
`abstract void` | `ShingleMatrixFilter.TokenSettingsCodec.setWeight(Token token, float weight)` Have this method do nothing in order to 'disable' weights.
`void` | `ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec.setWeight(Token token, float weight)`
`void` | `ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec.setWeight(Token token, float weight)`
`void` | `ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec.setWeight(Token token, float weight)` Stores a 32 bit float in the payload, or set it to null if 1f;
`void` | `ShingleMatrixFilter.updateToken(Token token, List shingle, int currentPermutationStartOffset, List currentPermutationRows, List currentPermuationTokens)` Final touch of a shingle token before it is passed on to the consumer from method `ShingleMatrixFilter.next(org.apache.lucene.analysis.Token)`.

Constructors in org.apache.lucene.analysis.shingle with parameters of type Token
`ShingleMatrixFilter.Matrix.Column(Token token)`

Uses of Token in org.apache.lucene.analysis.sinks

Methods in org.apache.lucene.analysis.sinks with parameters of type Token
`void` | `DateRecognizerSinkTokenizer.add(Token t)`
`void` | `TokenTypeSinkTokenizer.add(Token t)`
`void` | `TokenRangeSinkTokenizer.add(Token t)`

Uses of Token in org.apache.lucene.analysis.snowball

Methods in org.apache.lucene.analysis.snowball that return Token
`Token` | `SnowballFilter.next(Token reusableToken)` Returns the next input Token, after being stemmed

Methods in org.apache.lucene.analysis.snowball with parameters of type Token
`Token` | `SnowballFilter.next(Token reusableToken)` Returns the next input Token, after being stemmed

Uses of Token in org.apache.lucene.analysis.standard

Methods in org.apache.lucene.analysis.standard that return Token
`Token` | `StandardTokenizer.next(Token reusableToken)`
`Token` | `StandardFilter.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.

Methods in org.apache.lucene.analysis.standard with parameters of type Token
`Token` | `StandardTokenizer.next(Token reusableToken)`
`Token` | `StandardFilter.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.

Uses of Token in org.apache.lucene.analysis.th

Methods in org.apache.lucene.analysis.th that return Token
`Token` | `ThaiWordFilter.next(Token reusableToken)`

Methods in org.apache.lucene.analysis.th with parameters of type Token
`Token` | `ThaiWordFilter.next(Token reusableToken)`

Uses of Token in org.apache.lucene.index.memory

Methods in org.apache.lucene.index.memory that return Token
`protected Token` | `SynonymTokenFilter.createToken(String synonym, Token current, Token reusableToken)` Creates and returns a token for the given synonym of the current input token; Override for custom (stateless or stateful) behavior, if desired.
`Token` | `SynonymTokenFilter.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.

Methods in org.apache.lucene.index.memory with parameters of type Token
`protected Token` | `SynonymTokenFilter.createToken(String synonym, Token current, Token reusableToken)` Creates and returns a token for the given synonym of the current input token; Override for custom (stateless or stateful) behavior, if desired.
`Token` | `SynonymTokenFilter.next(Token reusableToken)` Returns the next token in the stream, or null at EOS.

Uses of Token in org.apache.lucene.search.highlight

Methods in org.apache.lucene.search.highlight that return Token
`Token` | `TokenGroup.getToken(int index)`

Methods in org.apache.lucene.search.highlight with parameters of type Token
`float` | `SpanScorer.getTokenScore(Token token)`
`float` | `QueryScorer.getTokenScore(Token token)`
`float` | `Scorer.getTokenScore(Token token)` Called for each token in the current fragment
`boolean` | `Fragmenter.isNewFragment(Token nextToken)` Test to see if this token from the stream should be held in a new TextFragment
`boolean` | `SimpleFragmenter.isNewFragment(Token token)`
`boolean` | `NullFragmenter.isNewFragment(Token token)`
`boolean` | `SimpleSpanFragmenter.isNewFragment(Token token)`

Uses of Token in org.apache.lucene.wikipedia.analysis

Methods in org.apache.lucene.wikipedia.analysis that return Token
`Token` | `WikipediaTokenizer.next(Token reusableToken)`

Methods in org.apache.lucene.wikipedia.analysis with parameters of type Token
`Token` | `WikipediaTokenizer.next(Token reusableToken)`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV NEXT | FRAMES NO FRAMES

Copyright © 2000-2009 Apache Software Foundation. All Rights Reserved.