Commit Graph

321 Commits

Author SHA1 Message Date
Alexander Borzunov
ef29dacc4c
Increase startup_timeout on server 2023-01-11 20:41:21 +04:00
justheuristic
c2cb6d19ae
Increase tolerances in test_tp_block (#196)
deflapify tests
2023-01-11 17:54:24 +03:00
Alexander Borzunov
b4f3224cda
Make client ignore blacklist if all servers holding a block are blacklisted (#197)
If all servers holding a certain block are blacklisted, we should display errors from them instead of raising `No peers holding blocks`.

Indeed, if the error is client-caused, the client should learn its reason from the latest error messages. In turn, if the error is server/network-caused and we only have a few servers, we'd better know the error instead of banning all the servers and making the user think that no servers are available.
2023-01-11 16:50:24 +04:00
Alexander Borzunov
127cf66bee
Ignore network RPS if we failed to measure it (#198) 2023-01-11 16:37:30 +04:00
Alexander Borzunov
487411e87e
Fix fine-tuning notebooks intros (#194)
The notebook intros were outdated and mentioned the 6B model, while the actual code already runs the 176B model. This led to confusion among our users in Discord.
2023-01-11 02:28:49 +04:00
Alexander Borzunov
82c9f93ce6
Bump version to 1.1.0 (#190) 2023-01-10 15:47:58 +04:00
Alexander Borzunov
a617ce3cfa
Fix psutil-related AccessDenied crash, disable --load_in_8bit by default in case of TP (#188)
* Don't count open fds since it leads to AccessDenied crashes on some machines
* Use --load_in_8bit=False by default in case of tensor parallelism
* Install petals from PyPI in fine-tuning tutorials
2023-01-10 13:04:52 +04:00
Egiazarian Vage
93bed7da5a
Support libp2p relays for NAT traversal (#186)
- Added relay options to servers
- Enabled relay options by default
- Changed hivemind version to 1.1.5
- Moved reachability check to be performed after blocks are loaded

Co-authored-by: Alexander Borzunov <borzunov.alexander@gmail.com>
2023-01-09 20:41:23 +04:00
Alexander Borzunov
16b69d6050
Fix GiBs in the "insufficient disk space" message (#187) 2023-01-09 09:54:44 +04:00
Alexander Borzunov
391c855208
Add readme subsections (#185) 2023-01-08 09:59:07 +04:00
Alexander Borzunov
f344c7801b
Add link to health.petals.ml to readme (#184) 2023-01-08 08:41:47 +04:00
Alexander Borzunov
27406a9377
Add more links to BLOOM to readme (#183) 2023-01-07 10:48:51 +04:00
Alexander Borzunov
0f6464103d
Remove protobuf from requirements (#182)
A correct protobuf version should be already installed by hivemind.

This also resolves version conflict on Colab, where protobuf versions required by Petals were different from the ones required by pre-installed tensorflow and tensorboard packages.

Co-authored-by: Max Ryabinin <mryabinin0@gmail.com>
2023-01-07 01:55:40 +04:00
Alexander Borzunov
712f5a330f
Remove backup bootstrap peer 2023-01-06 10:51:12 +04:00
justheuristic
d1fa5eb260
hotfix: add initial peer that did not crash :) (#181)
add hotfix initial peer (@borzunov's peers are down)
2023-01-06 04:57:49 +03:00
Alexander Borzunov
6dd9a938bd
Import bitsandbytes only if it's going to be used (#180) 2023-01-05 10:34:52 +04:00
Alexander Borzunov
e27706358c
Use slightly less memory in .generate() (#177) 2023-01-05 09:34:03 +04:00
Alexander Borzunov
55698381d0
Disable chunked_forward() on AVX512 CPUs (#179) 2023-01-04 23:28:16 +04:00
Alexander Borzunov
6948a0c5ee
Allow to disable chunked forward (#176) 2023-01-04 06:42:03 +04:00
Alexander Borzunov
356e099c3d
Make Docker command more visible (#175) 2023-01-04 02:18:45 +04:00
justheuristic
ae9e71fe8e
Add local tensor-parallel fwd/bwd (#143)
This pull request adds an option to run Petals server on multiple local GPUs. It uses https://github.com/BlackSamorez/tensor_parallel

- 8bit approximation error same as in main (mean~=2% q0.9~=5%)
    - TP=1, 2, 3 (see screenshots above)
- forward, grad w.r.t. input and inference exact match with main with TP=1
- `>=`80% GPU utilization with 3x 1080ti, batch = 8 tokens
- throughput measured with and without TP
- TP on 1080Tis has near-linear speedup comparable to the benchmarks (see first message)


Co-authored-by: Iaroslav Lisniak <yalisnyak@nes.ru>
Co-authored-by: Andrei Panferov <andrei@blacksamorez.ru>
Co-authored-by: Alexander Borzunov <borzunov.alexander@gmail.com>
2023-01-03 18:35:51 +03:00
Alexander Borzunov
779959bc70
Add link to PyPI (#173) 2022-12-31 02:51:52 +04:00
Alexander Borzunov
cdc3b6a25a
Add PyPI badge, update instructions and links in readme (#172) 2022-12-31 02:22:40 +04:00
Aleksandr Borzunov
ff8ade8d3b Bump version to 1.0.0 2022-12-30 21:52:57 +00:00
justheuristic
4014442a0f
Fix instruction for developers (#170) 2022-12-30 23:42:07 +04:00
Alexander Borzunov
26e6120288
Fix code example in readme (#169)
Makes it closer to runnable code, except for imports and defining tokenizer & data loader.
2022-12-30 19:50:38 +04:00
Alexander Borzunov
0b0277ed6f
Add link to chat.petals.ml (#168) 2022-12-30 07:58:33 +04:00
Vadim Peretokin
50fb8205de
Correct grammar in readme (#166) 2022-12-29 16:46:23 +03:00
Alexander Borzunov
714da529e6
Update wording in readme (#165) 2022-12-27 02:40:36 +04:00
Alexander Borzunov
9997ada3bb
Shield alloc & free from cancellation (#163)
A handler's RPC code may be cancelled due to a request timeout or a client closing the connection. Before this PR:

- If `.cancel()` happens while waiting for `hivemind.utils.enter_asynchronously()`, the lock will never be released.
- If `.cancel()` happens while doing that before freeing memory, the memory will never be freed.

This PR fixes it by deferring the cancellation with [asyncio.shield()](https://docs.python.org/3/library/asyncio-task.html#asyncio.shield). Now, the cancellation will happen only when all locks are released and alloc/free has completed.
2022-12-22 20:05:57 +04:00
Alexander Borzunov
d6992fca63
Hot fix: Increase hivemind.P2P's startup_timeout for Colab, remove absent initial peer (#162) 2022-12-19 19:36:20 +04:00
Artem Chumachenko
0a6b5f31aa
Fix misstypos in the example notebooks. (#161)
Fix misstypos
2022-12-16 19:42:18 +04:00
Alexander Borzunov
7cdc57a04b
Alloc inference cache as one contiguous buffer (#160) 2022-12-16 10:38:20 +04:00
Alexander Borzunov
523a7cad33
Fix issues related to petals as a module (#159)
1. Added `from petals.client import *` to `petals/__init__.py`, so you can write just that:

    ```python
    from petals import DistributedBloomForCausalLM
    ```

    I didn't do the same with server, since its classes are supposed to by used by `petals.cli.run_server`, not end-users. Though it's still possible to do `from petals.server.smth import smth` if necessary.

2. Fixed one more logging issue: log lines from hivemind were shown twice due to a bug in #156.

3. Removed unused `runtime.py`, since the server actually uses `hivemind.moe.Runtime`, and `runtime.py` has no significant changes comparing to it.
2022-12-16 09:09:06 +04:00
justheuristic
91898c3c90
Switch to speedtest-cli (#157)
This pullrequest removes custom speed_test code in favour of speedtest-cli module.
This is necessary to ensure that random warnings / print-outs do not mess with our outputs.

Co-authored-by: Max Ryabinin <mryabinin0@gmail.com>
2022-12-15 15:21:33 +03:00
Max Ryabinin
34644f13e1
Downgrade CUDA in Docker image to 11.0.3 (#145)
* Downgrade CUDA in Docker image to 11.0.3

* Remove development deps from the image
2022-12-15 10:14:29 +03:00
Artem Chumachenko
7911c2641d
Update advanced notebooks (#148)
Update examples

Co-authored-by: Alexander Borzunov <borzunov.alexander@gmail.com>
2022-12-15 09:28:25 +04:00
Alexander Borzunov
668b736031
Fix logging: do not duplicate lines, enable colors in Colab (#156) 2022-12-15 09:12:18 +04:00
Alexander Borzunov
041ad20891
Check reachability automatically and give advice how to fix it (#155)
1. If we connect to the **public swarm**, the server now **automatically checks its DHT's reachability** from the outside world using API at http://health.petals.ml This is important to disallow unreachable servers to proceed (they create issues for the clients, such as repetitive retries).

    If http://health.petals.ml is down, the server proceeds without the check (so we don't depend on it). However, if health.petals.ml is up and explicitly tells us that we are unrechable, the server shows the reason of that and how to solve it.

    The check may be disabled with the `--skip_reachability_check` option (though I can't imagine cases where someone needs to use it).

2. Added `--port` and `--public_ip` as **simplified convenience options** for users not familiar with `--host_maddrs` and `--announce_maddrs`.
2022-12-15 05:04:09 +04:00
Alexander Borzunov
73df69a117
Reset MemoryCache during rebalancings (#154)
Before this PR, if there were open inference sessions right when rebalancing is triggered, their cache was never properly destroyed.
2022-12-15 00:11:46 +04:00
Max Ryabinin
bd91be27ea
Add missing methods for SamplingAlgorithm, fix docstrings (#107)
* Add missing methods for SamplingAlgorithm, fix docstrings

* Add SamplingAlgorithm to _choose_sample_algorithm

* Add test_sampling

* Add a warning if sampling options were passed, but do_sample=False

* Skip the sampling test for now

Co-authored-by: Alexander Borzunov <borzunov.alexander@gmail.com>
2022-12-13 20:09:15 +03:00
Max Ryabinin
a0e8bbd28d
Fix arguments in remove_old_models.py (#153)
* Fix arguments in remove_old_models.py

* Remove unnecessary args.author

* Fix the GitHub Action as well
2022-12-13 19:01:12 +03:00
Alexander Borzunov
701ec7e53e
Clean up disk space (#152) 2022-12-13 18:50:43 +04:00
justheuristic
b04982c1a2
Bump transformers to 4.25.1 (#151)
- latest accelerate, transformers, huggingface_hub
- rearrange attention caches to support https://github.com/huggingface/transformers/pull/18344
- remove unused code
- fix edge case where session crashes when receiving seq length 0
- assert transformer version when importing WrappedBloomBlock

Co-authored-by: Alexander Borzunov <borzunov.alexander@gmail.com>
Co-authored-by: Max Ryabinin <mryabinin0@gmail.com>
2022-12-13 11:03:49 +03:00
Alexander Borzunov
e4dc938dfe
Fix OOMs during server rebalancing (#150)
The cause of OOMs were the cyclic references `TransformerBackend <-> PrioritizedTaskPool` that could not have been garbage collected properly. Still, I've added explicit tensor removal just in case.
2022-12-13 00:44:40 +04:00
Alexander Borzunov
83d9493b6c
Improve block size calculations (#149) 2022-12-12 13:15:23 +04:00
Aleksandr Borzunov
f42e559c77 Update README.md 2022-12-09 17:54:00 +00:00
Alexander Borzunov
6beb686909
Add link to privacy & security Wiki (#144) 2022-12-09 21:40:29 +04:00
Alexander Borzunov
84fec81543
Suppress asyncio error logs by default (#142) 2022-12-09 04:32:26 +04:00
Alexander Borzunov
e99bf36647
Use common folder for all caches, make it a volume in Dockerfile (#141) 2022-12-09 03:54:57 +04:00