Function GenieTokenizer_decode

Function Documentation

Genie_Status_t GenieTokenizer_decode(const GenieTokenizer_Handle_t tokenizerHandle, const int32_t *tokenIds, const uint32_t numTokenIds, const Genie_AllocCallback_t callback, const char **outputString)

A function to decode input token ids into text.

Parameters

  • tokenizerHandle – [in] A handle to the tokenizer. Must not be NULL.

  • tokenIds – [in] Input token ids. Must not be NULL.

  • numTokenIds – [in] The number of input token ids. Must be non-zero.

  • callback – [in] A callback function to allocate outputString. Must not be NULL.

  • outputString – [out] The decoded null-terminated string.The associated buffer was allocated in the client defined allocation callback and the memory needs to be managed by the client.

Returns

Status code:

  • GENIE_STATUS_SUCCESS: API call was successful.

  • GENIE_STATUS_ERROR_INVALID_HANDLE: Tokenizer handle is invalid.

  • GENIE_STATUS_ERROR_INVALID_ARGUMENT: At least one argument is invalid.

  • GENIE_STATUS_ERROR_MEM_ALLOC: Memory allocation failure.

  • GENIE_STATUS_ERROR_GENERAL: Tokenizer Decode failure.