ANTLR3_COMMON_TOKEN_STREAM_struct - Online Linux Manual PageSection : 3
Updated : Tue Jan 26 2021
Source : Version 3.3.1
Note : ANTLR3C
NAMEANTLR3_COMMON_TOKEN_STREAM_struct − Common token stream is an implementation of ANTLR_TOKEN_STREAM for the default parsers and recognizers.
SYNOPSIS
#include <antlr3tokenstream.h>
Data FieldsANTLR3_UINT32 channel
pANTLR3_LIST channelOverrides
Override map of tokens. ANTLR3_BOOLEAN discardOffChannel
If this flag is set to ANTLR3_TRUE, then tokens that the stream sees that are not in the channel that this stream is tuned to, are not tracked in the tokens table. void(* discardOffChannelToks )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_BOOLEAN discard)
Signal to discard off channel tokens from here on in. pANTLR3_LIST discardSet
Discared set. void(* discardTokenType )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 ttype)
Add a particular token type to the discard set. void(* free )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)
Function that knows how to free an ANTLR3_COMMON_TOKEN_STREAM. pANTLR3_LIST(* getTokenRange )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)
Function that returns all the tokens between a start and a stop index. pANTLR3_VECTOR(* getTokens )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)
Function that returns a pointer to the ANTLR3_LIST of all tokens in the stream (this causes the buffer to fill if we have not get any yet) pANTLR3_LIST(* getTokensList )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_LIST list)
Function that returns all the tokens indicated by being a member of the supplied List. pANTLR3_LIST(* getTokensSet )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_BITSET types)
Function that returns all the tokens indicated by the specified bitset, within a range of tokens. pANTLR3_LIST(* getTokensType )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, ANTLR3_UINT32 type)
Function that returns all tokens of a certain type within a range. ANTLR3_INT32 p
The index into the tokens list of the current token (the next one that will be consumed. void(* reset )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)
Function that resets the token stream so that it can be reused, but but that does not free up any resources, such as the token factory the factory pool and so on. void(* setTokenTypeChannel )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 ttype, ANTLR3_UINT32 channel)
A simple filter mechanism whereby you can tell this token stream to force all tokens of type ttype to be on channel. void * super
Whatever is supplying the COMMON_TOKEN_STREAM needs a pointer to itself so that this can be accessed by any of the API functions which it implements. pANTLR3_VECTOR tokens
Records every single token pulled from the source indexed by the token index. pANTLR3_TOKEN_STREAM tstream
The ANTLR3_TOKEN_STREAM interface implementation, which also includes the intstream implementation.
Detailed DescriptionCommon token stream is an implementation of ANTLR_TOKEN_STREAM for the default parsers and recognizers. You may of course build your own implementation if you are so inclined.
Field Documentation
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_STREAM_struct::channelReferenced by antlr3CommonTokenStreamSourceNew(), fillBuffer(), reset(), skipOffTokenChannels(), and skipOffTokenChannelsReverse().
pANTLR3_LIST ANTLR3_COMMON_TOKEN_STREAM_struct::channelOverridesOverride map of tokens. If a token type has an entry in here, then the pointer in the table points to an int, being the override channel number that should always be used for this token type. Referenced by antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), fillBuffer(), reset(), and setTokenTypeChannel().
ANTLR3_BOOLEAN ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannelIf this flag is set to ANTLR3_TRUE, then tokens that the stream sees that are not in the channel that this stream is tuned to, are not tracked in the tokens table. When set to false, ALL tokens are added to the tracking. Referenced by antlr3CommonTokenStreamSourceNew(), discardOffChannel(), fillBuffer(), and reset().
void(* ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannelToks) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_BOOLEAN discard)Signal to discard off channel tokens from here on in. Referenced by antlr3CommonTokenStreamNew().
pANTLR3_LIST ANTLR3_COMMON_TOKEN_STREAM_struct::discardSetDiscared set. If a token has an entry in this table, then it is thrown away (data pointer is always NULL). Referenced by antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), discardTokenType(), fillBuffer(), and reset().
void(* ANTLR3_COMMON_TOKEN_STREAM_struct::discardTokenType) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 ttype)Add a particular token type to the discard set. If a token is found to belong to this set, then it is skipped/thrown away Referenced by antlr3CommonTokenStreamNew().
void(* ANTLR3_COMMON_TOKEN_STREAM_struct::free) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)Function that knows how to free an ANTLR3_COMMON_TOKEN_STREAM. Referenced by antlr3CommonTokenStreamSourceNew().
pANTLR3_LIST(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokenRange) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)Function that returns all the tokens between a start and a stop index. TODO: This is a new list (Ack! Maybe this is a reason to have factories for LISTS and HASHTABLES etc :-( come back to this) Referenced by antlr3CommonTokenStreamNew().
pANTLR3_VECTOR(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokens) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)Function that returns a pointer to the ANTLR3_LIST of all tokens in the stream (this causes the buffer to fill if we have not get any yet) Referenced by antlr3CommonTokenStreamNew().
pANTLR3_LIST(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensList) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_LIST list)Function that returns all the tokens indicated by being a member of the supplied List. Referenced by antlr3CommonTokenStreamNew().
pANTLR3_LIST(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensSet) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_BITSET types)Function that returns all the tokens indicated by the specified bitset, within a range of tokens. Referenced by antlr3CommonTokenStreamNew(), getTokenRange(), getTokensList(), and getTokensType().
pANTLR3_LIST(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensType) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, ANTLR3_UINT32 type)Function that returns all tokens of a certain type within a range. Referenced by antlr3CommonTokenStreamNew().
ANTLR3_INT32 ANTLR3_COMMON_TOKEN_STREAM_struct::pThe index into the tokens list of the current token (the next one that will be consumed. p = -1 indicates that the token list is empty. Referenced by antlr3CommonTokenStreamNew(), consume(), fillBuffer(), getTokens(), getTokensSet(), LB(), reset(), seek(), tindex(), tokLT(), toString(), and toStringSS().
void(* ANTLR3_COMMON_TOKEN_STREAM_struct::reset) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)Function that resets the token stream so that it can be reused, but but that does not free up any resources, such as the token factory the factory pool and so on. This prevents the need to keep freeing and reallocating the token pools if the thing you are building is a multi-shot dameon or somethign like that. It is much faster to just reuse all the vectors. Referenced by antlr3CommonTokenStreamNew().
void(* ANTLR3_COMMON_TOKEN_STREAM_struct::setTokenTypeChannel) (struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 ttype, ANTLR3_UINT32 channel)A simple filter mechanism whereby you can tell this token stream to force all tokens of type ttype to be on channel. For example, when interpreting, we cannot exec actions so we need to tell the stream to force all WS and NEWLINE to be a different, ignored channel. Referenced by antlr3CommonTokenStreamNew().
void* ANTLR3_COMMON_TOKEN_STREAM_struct::superWhatever is supplying the COMMON_TOKEN_STREAM needs a pointer to itself so that this can be accessed by any of the API functions which it implements.
pANTLR3_VECTOR ANTLR3_COMMON_TOKEN_STREAM_struct::tokensRecords every single token pulled from the source indexed by the token index. There might be more efficient ways to do this, such as referencing directly in to the token factory pools, but for now this is convenient and the ANTLR3_LIST is not a huge overhead as it only stores pointers anyway, but allows for iterations and so on. Referenced by antlr3CommonTokenStreamNew(), antlr3CTSFree(), consume(), fillBuffer(), get(), getTokens(), LB(), reset(), size(), skipOffTokenChannels(), skipOffTokenChannelsReverse(), tokLT(), and toStringSS().
pANTLR3_TOKEN_STREAM ANTLR3_COMMON_TOKEN_STREAM_struct::tstreamThe ANTLR3_TOKEN_STREAM interface implementation, which also includes the intstream implementation. We could duplicate the pANTLR_INT_STREAM in this interface and initialize it to a copy, but this could be confusing it just results in one more level of indirection and I think that with judicial use of 'const' later, the optimizer will do decent job. Referenced by antlr3CommonTokenDebugStreamSourceNew(), antlr3CommonTokenStreamNew(), antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), antlr3RecognitionExceptionNew(), fillBuffer(), getTokensSet(), and skipOffTokenChannels().
AuthorGenerated automatically by Doxygen for ANTLR3C from the source code. 0
Johanes Gumabo
Data Size : 37,475 byte
man-antlr3-discardOffChannelToks.3Build : 2024-12-05, 20:55 :
Visitor Screen : x
Visitor Counter ( page / site ) : 2 / 188,067
Visitor ID : :
Visitor IP : 18.227.140.100 :
Visitor Provider : AMAZON-02 :
Provider Position ( lat x lon ) : 39.962500 x -83.006100 : x
Provider Accuracy Radius ( km ) : 1000 :
Provider City : Columbus :
Provider Province : Ohio , : ,
Provider Country : United States :
Provider Continent : North America :
Visitor Recorder : Version :
Visitor Recorder : Library :
Online Linux Manual Page : Version : Online Linux Manual Page - Fedora.40 - march=x86-64 - mtune=generic - 24.12.05
Online Linux Manual Page : Library : lib_c - 24.10.03 - march=x86-64 - mtune=generic - Fedora.40
Online Linux Manual Page : Library : lib_m - 24.10.03 - march=x86-64 - mtune=generic - Fedora.40
Data Base : Version : Online Linux Manual Page Database - 24.04.13 - march=x86-64 - mtune=generic - fedora-38
Data Base : Library : lib_c - 23.02.07 - march=x86-64 - mtune=generic - fedora.36
Very long time ago, I have the best tutor, Wenzel Svojanovsky . If someone knows the email address of Wenzel Svojanovsky , please send an email to johanes_gumabo@yahoo.co.id .
If error, please print screen and send to johanes_gumabo@yahoo.co.id
Under development. Support me via PayPal.