# tokenizer.h File Reference

`#include "error.h"`
`#include "transcode.h"`
`#include "token.h"`

Include dependency graph for tokenizer.h:

Go to the source code of this file.

## Typedefs

typedef struct FAXPP_TokenizerEnv_s | FAXPP_Tokenizer
| The tokenizer structure. Details of the structure are private.

## Functions

FAXPP_Tokenizer * | FAXPP_create_tokenizer (FAXPP_Transcoder encode)
| Creates a tokenizer object.
void | FAXPP_free_tokenizer (FAXPP_Tokenizer *tokenizer)
| Frees a tokenizer object.
FAXPP_DecodeFunction | FAXPP_get_tokenizer_decode (const FAXPP_Tokenizer *tokenizer)
| Returns the current FAXPP_DecodeFunction that the tokenizer is using.
void | FAXPP_set_tokenizer_decode (FAXPP_Tokenizer *tokenizer, FAXPP_DecodeFunction decode)
| Sets the FAXPP_DecodeFunction that the tokenizer uses to decode the XML document.
FAXPP_Error | FAXPP_init_tokenize (FAXPP_Tokenizer *tokenizer, void *buffer, unsigned int length, unsigned int done)
| Initialize the tokenizer to tokenize the given buffer, returning strings encoded using the given encoding function.
FAXPP_Error | FAXPP_tokenizer_release_buffer (FAXPP_Tokenizer *tokenizer, void **buffer_position)
| Instructs the tokenizer to release any dependencies it has on it's current buffer.
FAXPP_Error | FAXPP_continue_tokenize (FAXPP_Tokenizer *tokenizer, void *buffer, unsigned int length, unsigned int done)
| Provides a new buffer for the tokenizer to continue tokenizing.
FAXPP_Error | FAXPP_next_token (FAXPP_Tokenizer *tokenizer)
| Reads the next token from the buffer, placing the information for it into the current token.
const FAXPP_Token * | FAXPP_get_current_token (const FAXPP_Tokenizer *tokenizer)
| Returns the current token produced by the tokenizer when FAXPP_next_token() was called.
unsigned int | FAXPP_get_tokenizer_nesting_level (const FAXPP_Tokenizer *tokenizer)
| Returns the current element nesting level in the XML document.
unsigned int | FAXPP_get_tokenizer_error_line (const FAXPP_Tokenizer *tokenizer)
| Returns the line that the current error occured on.
unsigned int | FAXPP_get_tokenizer_error_column (const FAXPP_Tokenizer *tokenizer)
| Returns the column that the current error occured on.

## Typedef Documentation

typedef struct FAXPP_TokenizerEnv_s FAXPP_Tokenizer

The tokenizer structure. Details of the structure are private.

Definition at line 32 of file tokenizer.h.

## Function Documentation

FAXPP_Error FAXPP_continue_tokenize | ( | FAXPP_Tokenizer * | tokenizer,
|  | void * | buffer,
|  | unsigned int | length,
|  | unsigned int | done |
| ) |  |  |

Provides a new buffer for the tokenizer to continue tokenizing.

FAXPP_tokenizer_release_buffer() should have been called before this, and the remaining data in the old buffer transferred to the new one.

Parameters:

  | tokenizer |
  | buffer | A pointer to the start of the buffer to tokenize
  | length | The length of the given buffer
  | done | Set to non-zero if this is the last buffer from the input

Return values:

  | NO_ERROR |

FAXPP_Tokenizer * FAXPP_create_tokenizer | ( | FAXPP_Transcoder | encode | ) |

Creates a tokenizer object.

Parameters:

  | encode | The transcoder to use when encoding token values

Returns:
  A pointer to the tokenizer object, or 0 if out of memory.

void FAXPP_free_tokenizer | ( | FAXPP_Tokenizer * | tokenizer | ) |

Frees a tokenizer object.

Parameters:

  | tokenizer | The tokenizer to free

const FAXPP_Token * FAXPP_get_current_token | ( | const FAXPP_Tokenizer * | tokenizer | ) |

Returns the current token produced by the tokenizer when FAXPP_next_token() was called.

Parameters:

  | tokenizer |

Returns:
  The current token

FAXPP_DecodeFunction FAXPP_get_tokenizer_decode | ( | const FAXPP_Tokenizer * | tokenizer | ) |

Returns the current FAXPP_DecodeFunction that the tokenizer is using.

Parameters:

  | tokenizer |

Returns:
  The decode function

unsigned int FAXPP_get_tokenizer_error_column | ( | const FAXPP_Tokenizer * | tokenizer | ) |

Returns the column that the current error occured on.

Parameters:

  | tokenizer |

Returns:
  The column number

unsigned int FAXPP_get_tokenizer_error_line | ( | const FAXPP_Tokenizer * | tokenizer | ) |

Returns the line that the current error occured on.

Parameters:

  | tokenizer |

Returns:
  The line number

unsigned int FAXPP_get_tokenizer_nesting_level | ( | const FAXPP_Tokenizer * | tokenizer | ) |

Returns the current element nesting level in the XML document.

Parameters:

  | tokenizer |

Returns:
  The current nesting level

FAXPP_Error FAXPP_init_tokenize | ( | FAXPP_Tokenizer * | tokenizer,
|  | void * | buffer,
|  | unsigned int | length,
|  | unsigned int | done |
| ) |  |  |

Initialize the tokenizer to tokenize the given buffer, returning strings encoded using the given encoding function.

Parameters:

  | tokenizer | The tokenizer to initialize
  | buffer | A pointer to the start of the buffer to tokenize
  | length | The length of the given buffer
  | done | Set to non-zero if this is the last buffer from the input

Return values:

  | UNSUPPORTED_ENCODING | If the encoding sniffing algorithm cannot recognize the encoding of the buffer
  | NO_ERROR |

FAXPP_Error FAXPP_next_token | ( | FAXPP_Tokenizer * | tokenizer | ) |

Reads the next token from the buffer, placing the information for it into the current token.

Parameters:

  | tokenizer |

Return values:

  | DOUBLE_DASH_IN_COMMENT |
  | PREMATURE_END_OF_BUFFER |
  | INVALID_START_OF_COMMENT |
  | INVALID_CHAR_IN_START_ELEMENT |
  | INVALID_CHAR_IN_ATTRIBUTE |
  | INVALID_CHAR_IN_END_ELEMENT |
  | NON_WHITESPACE_OUTSIDE_DOC_ELEMENT |
  | BAD_ENCODING |
  | UNSUPPORTED_ENCODING |
  | ADDITIONAL_DOCUMENT_ELEMENT |
  | INVALID_CHAR_IN_PI_NAME |
  | INVALID_PI_NAME_OF_XML |
  | INVALID_CHAR_IN_ELEMENT_NAME |
  | INVALID_CHAR_IN_ATTRIBUTE_NAME |
  | RESTRICTED_CHAR |
  | INVALID_CHAR_IN_ENTITY_REFERENCE |
  | INVALID_CHAR_IN_CHAR_REFERENCE |
  | INVALID_CHAR_IN_XML_DECL |
  | EXPECTING_EQUALS |
  | EXPECTING_WHITESPACE |
  | UNKNOWN_XML_VERSION |
  | INVALID_ENCODING_VALUE |
  | OUT_OF_MEMORY |
  | NO_ERROR |

void FAXPP_set_tokenizer_decode | ( | FAXPP_Tokenizer * | tokenizer,
|  | FAXPP_DecodeFunction | decode |
| ) |  |  |

Sets the FAXPP_DecodeFunction that the tokenizer uses to decode the XML document.

This will typically be called when an encoding declaration is read, to switch to the correct decode function.

Parameters:

  | tokenizer |
  | decode | The decode function

FAXPP_Error FAXPP_tokenizer_release_buffer | ( | FAXPP_Tokenizer * | tokenizer,
|  | void ** | buffer_position |
| ) |  |  |

Instructs the tokenizer to release any dependencies it has on it's current buffer.

This is typically called on recieving a PREMATURE_END_OF_BUFFER error, before using FAXPP_continue_tokenize() to provide a new buffer. In this case, the buffer data between *buffer_position and the end of the buffer need to be copied into the start of the new buffer.

Parameters:

  | tokenizer |
  `[out]` | buffer_position | Set to a pointer in the current buffer that the tokenizer has tokenized up to

Return values:

  | OUT_OF_MEMORY |
  | NO_ERROR |