This PR:
1. Makes inference/forward/backward calls on client remember the dtype and device of source tensors, then move/cast the outputs to the same dtype/device. This way:
- Users don't need to make changes in the code launching `RemoteSequential` to make it run on a different device.
- `model.generate()` also starts to support both CPU and GPU.
2. Sets default `low_cpu_mem_usage=True`, client's request timeout to 20 sec.
3. Removes excess casts to float32 left in Dmitry's code.
4. (minor) Improves error messages.
The goals of these changes are:
- Make Petals work in Colab right after just doing `pip install -r requirements.txt`
- Make tests work independently of the protobuf package version chosen while installing dependencies
- Before this PR, `ServerState.JOINING` was announced only once. This announcement quickly expires in case of the full-size BLOOM, since loading blocks takes several minutes. This PR fixes it, so `ServerState.JOINING` is announced periodically in a thread until blocks are loaded.
- This PR also makes the `Server` class a non-thread, so it runs in the main thread and can catch `KeyboardInterrupt`. This is important, since if we are downloading blocks right now, we need to stop it and send the `ServerState.OFFLINE` message. Note that `ModuleContainer` is still a thread.
- (minor) For the sake of readability, I moved the `ModuleContainer.create()` definition, so it is now defined before `Server.__init__()` (this is because `.create()` is invoked first).
This PR makes servers and clients use public swarm's bootstrap peers if no other initial peers are specified.
If you'd like a server to start a new swarm, provide the `--new_swarm` CLI argument.
- run_server now accepts model name as both positional and keyword argument
- changed names in README to account for interface updates
- moved model conversion from README to a separate wiki page
- updated requirements.txt
* update dependency versions
* install bitsandbytes cpuonly from pip
* remove deprecated API from task pool
* clearer startup logs
Co-authored-by: Tim Dettmers <dettmers@cs.washington.edu>
* priority in handlers and backend pools
* simple points system on server side
* priortize task in handler before submit task
* fix tests
* s/expert/block/g
Co-authored-by: justheuristic <justheuristic@gmail.com>
Aleksandr Borzunov
Alexander Borzunov
Artem Chumachenko
justheuristic
Pavel Samygin