petals

Commit Graph

Author	SHA1	Message	Date
justheuristic	617d70f7dc	Support --load_in_8bit on pre-Turing GPUs (#113 ) - Linear8bitLt now supports for pre-turing GPUs by temporarily upcasting quantized weights. - added a test for linear8bitlt accuracy with the new fallback, the accuracy is similar than the real thing, (slightly better due to non-quantized A) - performance is roughly halfway between the default mode and memory_efficient_backward Alternatives considered: - cupy - slow, casting to float internally - triton - fast but unstable af. every 3rd attempt to matmul is a segfault - bnb.functional.igemm (no lt) - "CuBLAS Error 8" on old GPUs Co-authored-by: Aleksandr Borzunov <borzunov.alexander@gmail.com>	2 years ago
justheuristic	01838f9a99	Fix Linear8bitlt state config, update tests (#112 ) * fix state initializer * update tests to actually use new code * keep bias during quantization	2 years ago
justheuristic	088713912d	Patch Linear8bit to enable CxB backward (#111 ) A patch to bitsandbytes 0.34.0 that introduces an option to run backward pass in default (fast) matrix layout. Authors: cxb inversion by @borzunov, original 8bit code by @timdettmers * optimized layout inversion code by @borzunov ([original code](https://colab.research.google.com/drive/1EJ0MKifajXSSVq7O2_QGwtb0l6gRAGrh?usp=sharing)) to use less forward calls * implemented CustomLinear8bitLt, a child of Linear8bitLt that can do backward without CB * added exact match tests for layouts and linear layers: see tests/test_linear8bitlt.py * switched petals to the new layer type Core idea: layouts apply the same permutation to every tile in the matrix. We can treat this as (batched) gather ops. Reshape input tensor so that ij-th gather operation op will apply to ij-th elements in each tile. Prototype: Layout info: https://github.com/TimDettmers/bitsandbytes/blob/main/csrc/kernels.cu#L2130-L2136 Co-authored-by: Alexander Borzunov <hxrussia@gmail.com> Co-authored-by: Aleksandr Borzunov <borzunov.alexander@gmail.com> Co-authored-by: Tim Dettmers <tim.dettmers@gmail.com>	2 years ago

Author

SHA1

Message

Date

justheuristic

617d70f7dc

Support --load_in_8bit on pre-Turing GPUs (#113 )

- Linear8bitLt now supports for pre-turing GPUs by temporarily upcasting quantized weights.
- added a test for linear8bitlt accuracy with the new fallback, the accuracy is similar than the real thing, (slightly better due to non-quantized A)
- performance is roughly halfway between the default mode and memory_efficient_backward

Alternatives considered:
- cupy - slow, casting to float internally
- triton - fast but unstable af. every 3rd attempt to matmul is a segfault
- bnb.functional.igemm (no lt) - "CuBLAS Error 8" on old GPUs

Co-authored-by: Aleksandr Borzunov <borzunov.alexander@gmail.com>

justheuristic

01838f9a99

Fix Linear8bitlt state config, update tests (#112 )

* fix state initializer
* update tests to actually use new code
* keep bias during quantization

justheuristic

088713912d

Patch Linear8bit to enable CxB backward (#111 )

A patch to bitsandbytes 0.34.0 that introduces an option to run backward pass in default (fast) matrix layout.
Authors: cxb inversion by @borzunov, original 8bit code by @timdettmers

* optimized layout inversion code by @borzunov ([original code](https://colab.research.google.com/drive/1EJ0MKifajXSSVq7O2_QGwtb0l6gRAGrh?usp=sharing)) to use less forward calls
* implemented CustomLinear8bitLt, a child of Linear8bitLt that can do backward without CB
* added exact match tests for layouts and linear layers: see tests/test_linear8bitlt.py
* switched petals to the new layer type

Core idea: layouts apply the same permutation to every tile in the matrix. We can treat this as (batched) gather ops.
  Reshape input tensor so that ij-th gather operation op will apply to ij-th elements in each tile.

Prototype: 
Layout info: https://github.com/TimDettmers/bitsandbytes/blob/main/csrc/kernels.cu#L2130-L2136


Co-authored-by: Alexander Borzunov <hxrussia@gmail.com>
Co-authored-by: Aleksandr Borzunov <borzunov.alexander@gmail.com>
Co-authored-by: Tim Dettmers <tim.dettmers@gmail.com>

3 Commits (e191ce2f4e866def841477c57d22241e51d9c855)