6 Commits (50535a843504938d6062b1fcd5f7a35f50974b8e)

Author	SHA1	Message	Date
justheuristic	d271b75dd4	Let users specify sequence length instead of assuming 2048 (#52) ... - Maximum length is now provided in `.inference_session(max_length=100)` - previously, we would always assume max length = 2048 - added a generic way to forward **kwargs to inference session - for compatibility with #47 - Note to @borzunov : it does *not* pass them arbitrarily, but instead checks for kwarg names at the bottom level - run_server can be started with a custom max_length for inference - renamed --cache_size_bytes to --attention_cache_bytes (to avoid collision with --cache_dir) - --attn_cache_bytes can now support humane file sizes (e.g. 300MB instead of 314572800) - made some server-side errors more human-readable to user (e.g. when max length is exceeded) Co-authored-by: Aleksandr Borzunov <borzunov.alexander@gmail.com> Co-authored-by: Alexander Borzunov <hxrussia@gmail.com>	2 years ago
Dmitry Baranchuk	11a424837f	integrate mixed-8bit model (#39) ... * integrate mixed-8bit model * Fix bug with model duplication in RAM * set throughput=1.0 to fix zero throughput problem * add revision support * update hivemind and bitsandbytes * update deploy scripts * update installation instructions	2 years ago
Dmitry Baranchuk	04a2b6f5e3	Support various backend dtypes & async serialization (#38)	2 years ago
justheuristic	e2711a033b	Add automated tests (#23) ... This PR will run basic tests automatically on each subsequent PR - convert a small model on every PR - run existing tests on every PR - enforce black / isort - require checks on merge - make sure tests are not flappy Co-authored-by: Alexander Borzunov <hxrussia@gmail.com> Co-authored-by: Dmitry Baranchuk <dmitrybaranchuk@gmail.com>	2 years ago
justheuristic	99059ae667	install script	2 years ago
justheuristic	b370b43110	freeze hivemind and transformers versions	2 years ago