Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-16 06:13:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

community[patch]: invoke callback prior to yielding token (llama.cpp) (#19392)

Browse Source 
**Description:** Invoke callback prior to yielding token for llama.cpp
**Issue:** [Callback for on_llm_new_token should be invoked before the
token is yielded by the model
#16913](https://github.com/langchain-ai/langchain/issues/16913)
**Dependencies:** None
This commit is contained in:
aditya thomas 2024-03-23 01:47:56 +05:30 committed by GitHub
parent c4599444ee
commit 4856a87261
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
libs/community/langchain_community/llms/llamacpp.py
View File

@ -344,11 +344,11 @@ class LlamaCpp(LLM):
text=part["choices"][0]["text"], text=part["choices"][0]["text"],
generation_info={"logprobs": logprobs}, generation_info={"logprobs": logprobs},
) )
yield chunk
if run_manager: if run_manager:
run_manager.on_llm_new_token( run_manager.on_llm_new_token(
token=chunk.text, verbose=self.verbose, log_probs=logprobs token=chunk.text, verbose=self.verbose, log_probs=logprobs
) )
yield chunk
def get_num_tokens(self, text: str) -> int: def get_num_tokens(self, text: str) -> int:
tokenized_text = self.client.tokenize(text.encode("utf-8")) tokenized_text = self.client.tokenize(text.encode("utf-8"))