mirror of https://github.com/hwchase17/langchain
Add OpenAIEmbeddings special token params for tiktoken (#2682)
#2681
Original type hints
```python
allowed_special: Union[Literal["all"], AbstractSet[str]] = set(), # noqa: B006
disallowed_special: Union[Literal["all"], Collection[str]] = "all",
```
from
46287bfa49/tiktoken/core.py (L79-L80)
are not compatible with pydantic
<img width="718" alt="image"
src="https://user-images.githubusercontent.com/5096640/230993236-c744940e-85fb-4baa-b9da-8b00fb60a2a8.png">
I think we could use
```python
allowed_special: Union[Literal["all"], Set[str]] = set()
disallowed_special: Union[Literal["all"], Set[str], Tuple[()]] = "all"
```
Please let me know if you would like to implement it differently.
pull/2696/head
parent
1c979e320d
commit
023de9a70b
Loading…
Reference in New Issue