Archives
/
petals
mirror of https://github.com/bigscience-workshop/petals
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
508 Commits
130 Branches
15 Tags
9.2 MiB
main
peft_update
forward_backward
fix-docker
forward_kwargs
bump
test_main
fix-inference-retry
lora_from_hub
payload-size
partial_rollback
qkv_merge
no_qkv_merge
wip_triton
hivemind-dht-fork-process
repetition-penalty
amd-gpus
bnb-0-41-1
lru
beat-docker-into-submission
measurements
debug-leak
fix-nf4-and-dtypes
declare_adapters
empty-weights
download_8bit_weights
no-cpufeature
versions
test_opt_serving
borzunov-patch-2
borzunov-patch-1
processing_attention
yozh-dev-branch
server-increase-startup-timeout
vectorized_beam_search
friendly-timeout-errors
hivemind-1.1.4
fix3
hotfix_bnb
fix-ptune
server-dtypes
pip-installable-v2
pip-installable
diff-compression
client-convenience
server-timeouts
server-logging
beamsearch
fix-protobuf
fix-requirements
fix-joining-announce
bootstrap-peers
fault-tolerant-inference
examples_fix_hivemind
forward-backward-timeouts
fix-rebalancing-issues
add-sst2-example
enable-rebalancing
update_example_1
fix-too-many-open-files
update-hivemind
extract-module-container
instruction-readability-style
readme-clarifications
justheuristic-patch-5
fix-readme
ptune-example-personachat
rtfd
fix-pb2
investigate-segfault
upd-deps
priority-tasks
justheuristic-patch-4
cache
justheuristic-patch-3
generation-inference
deep_prompt_inference
warn-about-6b-instructions
update-readme-disclaimers-faq
justheuristic-patch-2
update-bullet-points
update-readme-pics
readme-release
remove-remote-block
prompt-inference
fix-cache
optimize_seq
fix-seq-backward-recovery
fix-distr-seq-cls
justheuristic-patch-1
fix-convert-8bit
memory_savings
distributed-deep-ptune
ptune-wip
pytest-verbose
rename-test-model
8bit_backward
8bit-model
8bit_model_inference
petals-readme-title
support-backend-dtypes
deep-prompt-tuning
mockup
efficient-forward-backward
fix-branch-name
dbaranchuk-patch-1
get_sequence
generation
fix-ci
fix-master-ci
test-push
facelift
CI
prompt-tuning
client-attempt2
measure-throughput
lm_head
load-balancing
sequence
demo-1
standardize
diff
rpc
update-model
client
fix-auth-token
multiple-experts
8bit_blocks
inference_chain
main_fix
v1.0.0
v1.1.0
v1.1.1
v1.1.2
v1.1.3
v1.1.4
v1.1.5
v2.0.0.post1
v2.0.0.post2
v2.0.0.post3
v2.0.1
v2.0.1.post1
v2.0.1.post2
v2.1.0
v2.2.0
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'main'
${ noResults }
Commit Graph

6 Commits (main)

Author SHA1 Message Date
Alexander Borzunov de2475f31c
Make client compatible with transformers' GenerationMixin (#464) 
This PR drops custom generation codes and introduces compatibility with `transformers.GenerationMixin` instead. This includes support for more sampling options (`top_p`, `top_k`, `repetition_penalty` requested in #460) and beam search - all that is now identical to running model with transformers locally.

Most features (excluding beam search and other rarely used stuff) are also compatible with resuming existing sessions.

### Breaking changes

If `.generate()` or forward passes are being run inside an `.inference_session()` context, they now use the opened session by default. So, these snippets are now equivalent:

```python
# Using default session
with model.inference_session(max_length=100):
    output_ids = model.generate(input_ids, max_new_tokens=3)

# Explicitly specifying a session
with model.inference_session(max_length=100) as sess:
    output_ids = model.generate(input_ids, max_new_tokens=3, session=sess)
```

Earlier, the 1st snippet was creating a new session, which is not what most people expected (= such code was most likely to introduce a bug, which is now fixed).
 9 months ago
Alexander Borzunov 056f22515a
Prioritize short inference, unmerge pools for long inference (#458) 
Right now, long inference requests may occupy Runtime for a few seconds without giving it away to process short (most latency-sensitive requests). This PR fixes it by disallowing the merged pool for long requests and prioritizing the short ones.
 10 months ago
justheuristic c4938bc23e
Merge inference pools into one to increase inference speed (#225) 
It turns out using a separate pool for each block has led to significant slowdown, see #224 for details.
 1 year ago
Max Ryabinin 3ca8b4f082
Fix typos with codespell (#126) 1 year ago
Max Ryabinin 9faf08b898
Remove unused imports, add missing arguments to docstrings (#108) 
* Remove unused imports, add missing arguments to docstrings
 1 year ago
Alexander Borzunov 7bd5916744
Make Petals a pip-installable package (attempt 2) (#102) 
1. Petals can be now installed using `pip install git+https://github.com/bigscience-workshop/petals`
    - In case if you already cloned the repo, you can do `pip install .` or `pip install .[dev]`
2. Moved `src` => `src/petals`
    - Replaced `from src.smth import smth` with `from petals.smth import smth`
3. Moved `cli` => `src/petals/cli`
    - Replaced `python -m cli.run_smth` with `python -m petals.cli.run_smth` (all utilities are now available right after pip installation)
4. Moved the `requirements*.txt` contents to `setup.cfg` (`requirements.txt` for packages is not supported well by modern packaging utils)
5. Increased the package version from `0.2` to `1.0alpha1`
 2 years ago