petals

Commit Graph

Author	SHA1	Message	Date
justheuristic	2ad0b2b936	Fix p2p pushing in rpc_inference (by @miaoqijun ) , support transformers 4.38.2 (#563 ) This pull request solves #560 using a solution proposed by @miaoqijun . It also bumps transformers to the latest version to test with the latest code. --------- Co-authored-by: Yingtong Dou <ytongdou@gmail.com>	3 months ago
Alexander Borzunov	de2475f31c	Make client compatible with transformers' GenerationMixin (#464 ) This PR drops custom generation codes and introduces compatibility with `transformers.GenerationMixin` instead. This includes support for more sampling options (`top_p`, `top_k`, `repetition_penalty` requested in #460) and beam search - all that is now identical to running model with transformers locally. Most features (excluding beam search and other rarely used stuff) are also compatible with resuming existing sessions. ### Breaking changes If `.generate()` or forward passes are being run inside an `.inference_session()` context, they now use the opened session by default. So, these snippets are now equivalent: ```python # Using default session with model.inference_session(max_length=100): output_ids = model.generate(input_ids, max_new_tokens=3) # Explicitly specifying a session with model.inference_session(max_length=100) as sess: output_ids = model.generate(input_ids, max_new_tokens=3, session=sess) ``` Earlier, the 1st snippet was creating a new session, which is not what most people expected (= such code was most likely to introduce a bug, which is now fixed).	10 months ago
Alexander Borzunov	063e94b4c8	Move SequenceManagerConfig -> ClientConfig, petals.dht_utils -> petals.utils.dht (#463 )	10 months ago
Artem Chumachenko	568f21dc3b	Add customizable input tensors (#445 )	10 months ago
Alexander Borzunov	056f22515a	Prioritize short inference, unmerge pools for long inference (#458 ) Right now, long inference requests may occupy Runtime for a few seconds without giving it away to process short (most latency-sensitive requests). This PR fixes it by disallowing the merged pool for long requests and prioritizing the short ones.	10 months ago
justheuristic	ac9b546706	[Refactor] extract block forward, backward and inference into a separate file (#435 ) This PR does not change any functionality. It merely moves stuff around. List of changes: handler.py/_rpc_forward became block_methods/rpc_forward handler.py/_rpc_backward became block_methods/rpc_backward the math bits of rpc_inference were extracted into block_methods/iterate_rpc_inference --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: artek0chumak <artek.chumak@gmail.com> Co-authored-by: Aleksandr Borzunov <borzunov.alexander@gmail.com>	10 months ago

6 Commits (2ad0b2b936629da2076d639da2c1200468ea568e)