4 Commits (dd677d9e76bf9b21f48c9562e9267b6abc56d86a)

Author	SHA1	Message	Date
Alexander Borzunov	a1f7791d5e	Fix petals.utils.ping for servers with client-mode DHT (#430) ... Fix #429.	10 months ago
Alexander Borzunov	62d9ed5ce7	Implement shortest-path routing for inference (#362) ... This PR: 1. **Adds shortest path routing for inference.** We build a graph with client-server and server-server latencies and compute costs, as well as empirically measured overheads. For client-server latencies, we ping possible first and last servers in a sequence in `SequenceManager.update()`. We penalize servers who may not have enough cache for our request. This uses info added to DHT in #355, #356, #358. 2. **Makes a server ping neighboring servers in addition to next ones.** This is to get an opportunity to change the server even before we use all its blocks (e.g., because a neighboring server is faster). This feature is not enabled though, since it increases graph size for N servers to O(N^2) - but we may enable it if needed. 3. **Fixes a `SequenceManager` bug with the first `update()`.** Previously, this update was likely to produce incorrect information and cause to `MissingBlocksErrors` until the next update happens.	11 months ago
Alexander Borzunov	11f0d992d7	Report inference, forward, and network RPS separately (#358) ... Inference RPS may be very different from forward RPS. E.g., currently bnb uses a completely different algorithm for NF4 inference. We report detailed RPS info that can be then used for shortest-path routing for inference.	11 months ago
Alexander Borzunov	81c4a45ca2	Make a server ping next servers (#356) ... This PR makes a server ping potential next servers in a chain and report the RTTs to DHT. This will be used for shortest-path routing.	11 months ago