You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
langchain/libs/partners/nvidia-trt
Mikhail Khludnev 14ff1438e6
nvidia-trt[patch]: propagate InferenceClientException to the caller. (#16936)
- **Description:**  
 
before the change I've got

1. propagate InferenceClientException to the caller.
2. stop grpc receiver thread on exception 

```
        for token in result_queue:
>           result_str += token
E           TypeError: can only concatenate str (not "InferenceServerException") to str

../../langchain_nvidia_trt/llms.py:207: TypeError
```
And stream thread keeps running. 

after the change request thread stops correctly and caller got a root
cause exception:

```
E                   tritonclient.utils.InferenceServerException: [request id: 4529729] expected number of inputs between 2 and 3 but got 10 inputs for model 'vllm_model'

../../langchain_nvidia_trt/llms.py:205: InferenceServerException
```

  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
  - **Twitter handle:** [t.me/mkhl_spb](https://t.me/mkhl_spb)
 
I'm not sure about test coverage. Should I setup deep mocks or there's a
kind of triton stub via testcontainers or so.
8 months ago
..
docs [Partner] NVIDIA TRT Package (#14733) 9 months ago
langchain_nvidia_trt nvidia-trt[patch]: propagate InferenceClientException to the caller. (#16936) 8 months ago
scripts [Partner] NVIDIA TRT Package (#14733) 9 months ago
tests Fix: fix partners name typo in tests (#15066) 9 months ago
.gitignore [Partner] NVIDIA TRT Package (#14733) 9 months ago
LICENSE [Partner] NVIDIA TRT Package (#14733) 9 months ago
Makefile [Partner] NVIDIA TRT Package (#14733) 9 months ago
README.md [Partner] NVIDIA TRT Package (#14733) 9 months ago
mypy.ini [Partner] NVIDIA TRT Package (#14733) 9 months ago
poetry.lock nvidia-trt: remove tritonclient all extra dep (#16749) 8 months ago
pyproject.toml nvidia-trt: remove tritonclient all extra dep (#16749) 8 months ago

README.md

langchain-nvidia-trt