community[patch]: invoke callback prior to yielding token (llama.cpp) (#19392)

**Description:** Invoke callback prior to yielding token for llama.cpp
**Issue:** [Callback for on_llm_new_token should be invoked before the
token is yielded by the model
#16913](https://github.com/langchain-ai/langchain/issues/16913)
**Dependencies:** None
pull/19308/head^2
aditya thomas 3 months ago committed by GitHub
parent c4599444ee
commit 4856a87261
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

@ -344,11 +344,11 @@ class LlamaCpp(LLM):
text=part["choices"][0]["text"],
generation_info={"logprobs": logprobs},
)
yield chunk
if run_manager:
run_manager.on_llm_new_token(
token=chunk.text, verbose=self.verbose, log_probs=logprobs
)
yield chunk
def get_num_tokens(self, text: str) -> int:
tokenized_text = self.client.tokenize(text.encode("utf-8"))

Loading…
Cancel
Save