ANTLR3_COMMON_TOKEN_struct - Online Linux Manual PageSection : 3
Updated : Tue Jan 26 2021
Source : Version 3.3.1
Note : ANTLR3C
NAMEANTLR3_COMMON_TOKEN_struct − The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure.
SYNOPSIS
#include <antlr3commontoken.h>
Data FieldsANTLR3_UINT32 channel
The virtual channel that this token exists in. ANTLR3_INT32 charPosition
The character position in the line that this token was derived from. void * custom
Pointer to a custom element that the ANTLR3 programmer may define and install. ANTLR3_BOOLEAN factoryMade
Indicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it. void(* freeCustom )(void *custom)
Pointer to a function that knows how to free the custom structure when the token is destroyed. ANTLR3_UINT32(* getChannel )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels. ANTLR3_INT32(* getCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the offset in the line where this token exists. ANTLR3_UINT32(* getLine )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the 'line' number where this token resides. ANTLR3_MARKER(* getStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the start index in the input stream for this token. ANTLR3_MARKER(* getStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the stop index in the input stream for this token. pANTLR3_STRING(* getText )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token. ANTLR3_MARKER(* getTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that returns an index 0...n-1 of the token in the token input stream. ANTLR3_UINT32(* getType )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that returns the token type of this token. ANTLR3_MARKER index
What the index of this token is, 0, 1, .., n-2, n-1 tokens. pANTLR3_INPUT_STREAM input
Pointer to the input stream that this token originated in. ANTLR3_UINT32 line
The line number in the input stream where this token was derived from. void * lineStart
The offset into the input stream that the line in which this token resides starts. void(* setChannel )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel)
Pointer to a function that sets the channel that this token should belong to. void(* setCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos)
Pointer to a function that sets the offset in the line where this token exists. void(* setLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line)
Pointer to a function that sets the 'line' number where this token reside. void(* setStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
Pointer to a function that sets the start index in the input stream for this token. void(* setStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
Pointer to a function that sets the stop index in the input stream for this token. void(* setText )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text)
Pointer to a function that 'might' be able to set the text associated with a token. void(* setText8 )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text)
Pointer to a function that 'might' be able to set the text associated with a token. void(* setTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER)
Pointer to a function that can set the token index of this token in the token input stream. void(* setType )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype)
Pointer to a function that sets the type of this token. ANTLR3_MARKER start
The character offset in the input stream where the text for this token starts. ANTLR3_MARKER stop
The character offset in the input stream where the text for this token stops. pANTLR3_STRING_FACTORY strFactory
A string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING. ANTLR3_UINT32 textState
Indicates whether this token is carrying: union {
pANTLR3_UCHAR chars
Pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar. pANTLR3_STRING text
Some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL. } tokText
pANTLR3_STRING(* toString )(struct ANTLR3_COMMON_TOKEN_struct *token)
"Pointertoafunctionthatreturnsthistokenasatextrepresentationthatcanbeprintedwithembeddedcontrolcodessuchas
replaced with the printable sequence '\n' This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations\&. " ANTLR3_UINT32 type
The actual type of this token. ANTLR3_UINT32 user1
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. ANTLR3_UINT32 user2
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. ANTLR3_UINT32 user3
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
Detailed DescriptionThe definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure. Remarks Token streams are in essence provided by lexers or other programs that serve as lexers.
Field Documentation
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::channelThe virtual channel that this token exists in. Referenced by emit(), getChannel(), setChannel(), skipOffTokenChannels(), and skipOffTokenChannelsReverse().
ANTLR3_INT32 ANTLR3_COMMON_TOKEN_struct::charPositionThe character position in the line that this token was derived from. Referenced by emit(), getCharPositionInLine(), and setCharPositionInLine().
pANTLR3_UCHAR ANTLR3_COMMON_TOKEN_struct::charsPointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar. No sense in constructing a pANTLR3_STRING just for that, as mostly the text will not be accessed - if it is, then we will build a pANTLR3_STRING for it a that point. Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), getText(), newDownNode(), newUpNode(), and setText8().
void* ANTLR3_COMMON_TOKEN_struct::customPointer to a custom element that the ANTLR3 programmer may define and install. Referenced by antlr3LexerNew(), emit(), factoryClose(), and getMissingSymbol().
ANTLR3_BOOLEAN ANTLR3_COMMON_TOKEN_struct::factoryMadeIndicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it. Referenced by antlr3LexerNew(), antlr3TokenFactoryNew(), newPoolToken(), newToken(), and nextTokenStr().
void(* ANTLR3_COMMON_TOKEN_struct::freeCustom) (void *custom)Pointer to a function that knows how to free the custom structure when the token is destroyed. Referenced by factoryClose().
ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getChannel) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), fillBuffer(), serializeToken(), and toString().
ANTLR3_INT32(* ANTLR3_COMMON_TOKEN_struct::getCharPositionInLine) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to a function that gets the offset in the line where this token exists. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getCharPositionInLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().
ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getLine) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to a function that gets the 'line' number where this token resides. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().
ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getStartIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to a function that gets the start index in the input stream for this token. Referenced by antlr3SetTokenAPI(), getText(), and toString().
ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getStopIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to a function that gets the stop index in the input stream for this token. Referenced by antlr3SetTokenAPI(), getText(), and toString().
pANTLR3_STRING(* ANTLR3_COMMON_TOKEN_struct::getText) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), serializeToken(), toString(), and toStringSS().
ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getTokenIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to a function that returns an index 0...n-1 of the token in the token input stream. Referenced by antlr3SetTokenAPI(), createNodeTok(), createTokenFromToken(), dbgSetTokenBoundaries(), serializeToken(), setTokenBoundaries(), toString(), and toStringTT().
ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getType) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to a function that returns the token type of this token. Referenced by _LA(), antlr3SetTokenAPI(), createTokenFromToken(), fillBuffer(), getMissingSymbol(), getTokensSet(), getType(), and serializeToken().
ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::indexWhat the index of this token is, 0, 1, .., n-2, n-1 tokens. Referenced by getTokenIndex(), and setTokenIndex().
pANTLR3_INPUT_STREAM ANTLR3_COMMON_TOKEN_struct::inputPointer to the input stream that this token originated in. Referenced by antlr3RecognitionExceptionNew(), createToken(), getMissingSymbol(), getStartIndex(), getText(), newPoolToken(), and setInputStream().
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::lineThe line number in the input stream where this token was derived from. Referenced by emit(), getLine(), and setLine().
void* ANTLR3_COMMON_TOKEN_struct::lineStartThe offset into the input stream that the line in which this token resides starts. Referenced by emit(), and getMissingSymbol().
void(* ANTLR3_COMMON_TOKEN_struct::setChannel) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel)Pointer to a function that sets the channel that this token should belong to. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), fillBuffer(), and getMissingSymbol().
void(* ANTLR3_COMMON_TOKEN_struct::setCharPositionInLine) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos)Pointer to a function that sets the offset in the line where this token exists. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and getMissingSymbol().
void(* ANTLR3_COMMON_TOKEN_struct::setLine) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line)Pointer to a function that sets the 'line' number where this token reside. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), and nextTokenStr().
void(* ANTLR3_COMMON_TOKEN_struct::setStartIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)Pointer to a function that sets the start index in the input stream for this token. Referenced by antlr3SetTokenAPI(), newPoolToken(), nextTokenStr(), and tokLT().
void(* ANTLR3_COMMON_TOKEN_struct::setStopIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)Pointer to a function that sets the stop index in the input stream for this token. Referenced by antlr3SetTokenAPI(), nextTokenStr(), and tokLT().
void(* ANTLR3_COMMON_TOKEN_struct::setText) (struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text)Pointer to a function that 'might' be able to set the text associated with a token. Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actaully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically. Referenced by antlr3SetTokenAPI().
void(* ANTLR3_COMMON_TOKEN_struct::setText8) (struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text)Pointer to a function that 'might' be able to set the text associated with a token. Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically. Referenced by antlr3SetTokenAPI(), createTypeTokenText(), and getMissingSymbol().
void(* ANTLR3_COMMON_TOKEN_struct::setTokenIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER)Pointer to a function that can set the token index of this token in the token input stream. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and fillBuffer().
void(* ANTLR3_COMMON_TOKEN_struct::setType) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype)Pointer to a function that sets the type of this token. Referenced by antlr3CommonTokenNew(), antlr3LexerNew(), antlr3SetTokenAPI(), createToken(), createTokenFromToken(), createTypeToken(), createTypeTokenText(), and getMissingSymbol().
ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::startThe character offset in the input stream where the text for this token starts. Referenced by emit(), getStartIndex(), and setStartIndex().
ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::stopThe character offset in the input stream where the text for this token stops. Referenced by emit(), getStopIndex(), and setStopIndex().
pANTLR3_STRING_FACTORY ANTLR3_COMMON_TOKEN_struct::strFactoryA string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING. Referenced by ANTLR3_TREE_ADAPTORNew(), antlr3CommonTreeNodeStreamNew(), antlr3CommonTreeNodeStreamNewStream(), antlr3LexerNew(), createToken(), getText(), newPoolToken(), setCharStream(), and setInputStream().
pANTLR3_STRING ANTLR3_COMMON_TOKEN_struct::textSome token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL. Referenced by createTokenFromToken(), emit(), getText(), setText(), and setText8().
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::textStateIndicates whether this token is carrying: State | Meaning ------+-------------------------------------- 0 | Nothing (neither rewrite text, nor setText) 1 | char * to user supplied rewrite text 2 | pANTLR3_STRING because of setText or similar action Affects the union structure tokText below (uses 32 bit so alignment is always good) Referenced by antlr3CommonTreeNodeStreamNew(), antlr3LexerNew(), createToken(), createTokenFromToken(), emit(), getText(), newDownNode(), newUpNode(), setText(), and setText8().
union { ... } ANTLR3_COMMON_TOKEN_struct::tokTextReferenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), emit(), getText(), newDownNode(), newUpNode(), setText(), and setText8().
pANTLR3_STRING(* ANTLR3_COMMON_TOKEN_struct::toString) (struct ANTLR3_COMMON_TOKEN_struct *token)Pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as
replaced with the printable sequence '\n' This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations\&. Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and displayRecognitionError().
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::typeThe actual type of this token. Referenced by displayRecognitionError(), emit(), fillBuffer(), getText(), getType(), nextToken(), setType(), and toString().
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user1Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them. Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user2Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them. Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user3Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them. Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().
AuthorGenerated automatically by Doxygen for ANTLR3C from the source code. 0
Johanes Gumabo
Data Size : 71,941 byte
man-antlr3-text.3Build : 2024-12-05, 20:55 :
Visitor Screen : x
Visitor Counter ( page / site ) : 3 / 200,526
Visitor ID : :
Visitor IP : 3.144.18.59 :
Visitor Provider : AMAZON-02 :
Provider Position ( lat x lon ) : 39.962500 x -83.006100 : x
Provider Accuracy Radius ( km ) : 1000 :
Provider City : Columbus :
Provider Province : Ohio , : ,
Provider Country : United States :
Provider Continent : North America :
Visitor Recorder : Version :
Visitor Recorder : Library :
Online Linux Manual Page : Version : Online Linux Manual Page - Fedora.40 - march=x86-64 - mtune=generic - 24.12.05
Online Linux Manual Page : Library : lib_c - 24.10.03 - march=x86-64 - mtune=generic - Fedora.40
Online Linux Manual Page : Library : lib_m - 24.10.03 - march=x86-64 - mtune=generic - Fedora.40
Data Base : Version : Online Linux Manual Page Database - 24.04.13 - march=x86-64 - mtune=generic - fedora-38
Data Base : Library : lib_c - 23.02.07 - march=x86-64 - mtune=generic - fedora.36
Very long time ago, I have the best tutor, Wenzel Svojanovsky . If someone knows the email address of Wenzel Svojanovsky , please send an email to johanes_gumabo@yahoo.co.id .
If error, please print screen and send to johanes_gumabo@yahoo.co.id
Under development. Support me via PayPal.
ERROR : Need New Coding : (rof_nr_x|149|ANTLR3_COMMON_TOKEN_struct.3|154|\n' This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations\&. " | replaced with the printable sequence '\\\\n' This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations\&. "
) (rof_nr_x|149|ANTLR3_COMMON_TOKEN_struct.3|389|\n' This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations\&. | replaced with the printable sequence '\\\\n' This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations\&.
)