OpenTTD-patches/docs/jgrpp-low-level-changes.md

## JGR's Patchpack: Low level changes

This document describes low-level changes to the codebase which are not generally visible when actually running/playing the game, this is a non-exhaustive list.

This document does not describe the player-visible changes/additions described in the main readme.

### Crash logger and diagnostics

* Additional logged items: current company ID, map size, configure invocation, thread name, recently executed commands, static NewGRFs.
* Additional logged platform-specific items: detailed OS version (Unix), signal details (Unix, Mac), exception record data (Windows).
* Better handling of crashes which occur in a non-main thread (ask the main thread to do the crash screenshot and savegame).
* Support logging register values on Unix and Mac.
* Support using libbfd for symbol lookup and line numbers (gcc/clang).
* Support using gdb/lldb if available to add further detail to the crashlog (Unix, Mac).
* Support using sigaction and sigaltstack for more information and correct handling of stack overflow crashes (Unix).
* Attempt to log stack overflow and heap corruption exceptions (Windows).
* Demangle C++ symbols (Unix).
* Attempt to handle crashes which occur within the crashlog handler, by skipping or only partially writing the faulting section.
* Emit a "crash" log, savegame and screenshot on multiplayer desync.
* Add crash/desync information to output screenshot and savegame files.
* Multiplayer server and client exchange desync logs after a desync occurs.
* Multiplayer clients send state hashes and random values since the last sync to the server after a desync, to identify which frame first diverged.
* Decrease sync frame period when desync occurs.

#### Assertions

* Various assertions are extended to log further information on failure.
* Various assertions which check the state of a tile are extended to dump the tile state (m1 - m8, etc.) on failure.

#### Scope annotations

* Scopes (in the main thread) can be annotated with a functor/lambda which is called in the event of a crash to provide further information to add to the crash log.

#### NewGRF debug window

* Add various supplementary non-GRF information, e.g. vehicle variables and flags.
* Add NewGRF sprite group dumping and related functionality.

#### Logging

* Add yapfdesync, linkgraph, sound, and command log levels.
* Extend desync and random logging.

### Map

* Store tunnel start/end pairs in a pool, indexed in the start/end tiles.
* Set bit in map if level crossing is possibly occupied by a road vehicle.
* Cache inferred one-way state of road tiles.
* De-virtualise calls to AnimateTile().
* Cache animated tile speed.
* Cache whether water tiles have water for all neighbouring tiles.
* Improve performance of arctic snow line checks.

### Viewport

* Cache bridge/tunnel start and ends.
* Cache station sign bounds.
* Split sprite sort regions when more than 60 sprites present.
* Reduce unnecessary region redraws when scrolling viewports.
* Reduce viewport invalidation region size of track reservation and signal state changes.
* Cache landscape background in map mode.
* Partial parallelisation of non-map mode viewport rendering.

### Rendering

* Track dirty viewport areas seperately from general screen redraws, using a zoom-level dependant sized grid.
* Use a rectangle array for general screen redraws instead of a block grid.
* Add a dirty bit to windows and widgets, for redrawing entire windows or widgets.
* Clip drawing of window widgets which are not in the redraw area.
* Reduce unnecessary status bar and vehicle list window redraws.
* Filter out tile parts which are entirely outside the drawing area, within DrawTileProc handlers.
* Improve performance of drawing rail catenary.
* Cache which window types are currently shown.

### Data structures

* Various data structures have been replaced with B-tree maps/sets (cpp-btree library), or robin-hood hash maps.
* Various lists have been replaced with vectors, or other data structures.
* Various deques and queues have been replaced with ring buffers.
* Remove mutexes from SmallStack, only used from the main thread.
* Add a third parameter p3, and an auxiliary data mechanism to DoCommand/CommandContainer.
* Add a free bitmap for pool slots.
* Maintain free list for text effect entries.
* Many fields have been widened.
* Change underlying data structures for ScriptList, create reverse mapping on demand instead of unconditionally.
* Split GoodsEntry structure.

### Vehicles

* Cache the sprite_seq bounds.
* Index the order list in a vector.
* Observe the operation of the NewGRF when getting the vehicle image/sprite, and elide further calls to the NewGRF if it can be determined that the result will be the same.
* Update train/road vehicle image/sprite on demand (i.e. when on screen) when image is continuously updated by GRF.
* Add consist flag for the case where no vehicles in consist are on a slope.
* Add vehicle flag to mark the last vehicle in a consist with a visual effect.
* Index the vehicle list in per type arrays for use by CallVehicleTicks.
* Cache whether the vehicle should be drawn.
* Pre-compute engine refit capacity callbacks if possible.

### Network/multiplayer

* Add supplementary information to find server UDP packets and reply in an extended format with more info/wider fields if detected.
* Paginate UDP packets longer than the MTU across multiple packets.
* Use larger "packets" where useful in TCP connections.
* Send vehicle caches from network server to clients to avoid desyncs caused by non-deterministic NewGRFs.
* Change network protocol to send server/join and rcon passwords in an encrypted form (key exchange) instead of in clear text.
* Encrypt the contents of rcon messages to the server and any responses.

### Sprites/blitter

* Add a fast path to Blitter_32bppAnim::Draw.
* Replace sprite cache implementation.
* Only cache sprites at the currently required zoom levels instead of all of them.
* Add brightness adjusting modes to non-8bpp blitters.

### Link graph

* Completely change link graph job scheduling to make the duration of a job and the number of jobs per thread instance variable according to the estimated size of the job.
* Various use of custom allocators, etc.
* Early abort link graph threads if abandoning/quitting the game.
* Various forms of caching and incremental updates to the link graph overlay.
* Change FlowStat from an RB-tree to a flat map with small-object optimisation.
* Change FlowStatMap from an RB-tree to a B-tree indexed vector.
* Change LinkGraph::EdgeMatrix to a sparse storage format.
* Replace MCF Dijkstra RB-tree with B-tree.
* Reduce performance issues when deleting stale links with refit to any cargo.
* Dynamically adjust accuracy parameters in MCF 1st pass to avoid computing large numbers of excessively small flows.

### Pathfinder

* YAPF: Reduce need to scan open list queue when moving best node to closed list

### Save and load

* Feature versioning, see readme and code.
* Extend gamelog to not truncate version strings.
* Save/load the map in a single chunk, such that it can be saved/loaded in one pass.
* Various other changes to savegame format and settings handling, see readme and code for details.
* Replace read/write accessors and buffering.
* Perform savegame decompression in a separate thread.
* Pre-filter SaveLoad descriptor arrays for current version/mode, for chunks with many objects.
* Support zstd compression for autosaves and network joins.

### AI/GS

* [AI/GS script additions](docs/script-additions.html).
* Add AI/GS method to get current day length.
* Add GS method to create river tiles.
* Add AI/GS methods related to road and tram types.
* Add workaround for performance issues when attempting to create a town when no town names are left.
* Fixup a GS otherwise inconsistent with day length.

### NewGRF

* [NewGRF specification additions](docs/newgrf-additions.html).
* Add workaround for a known buggy NewGRF to avoid desync issues.
* Apply many optimisations to VarAction2 deterministic sprite groups.
* Avoid making callbacks which can be pre-determined to be unhandled, or which can be statically determined ahead of time.
* Avoid animating industry tiles which are not actually animated in the current layout.
* Setting the animation frame to its current value no longer triggers a redraw.
* Animation is not started if it can be determined that it would stop immediately.
* Avoid unnecessarily triggering or redrawing NewGRF houses.

### SDL2
* Update whole window surface if >= 80% needs updating.
* Only pass a single rectangle to SDL_UpdateWindowSurfaceRects to prevent screen tearing.
* Allow using the hash key (#) as a hotkey.

### Other performance improvements

* Use multiple threads for NewGRF scan MD5 calculations, on multi-CPU machines.
* Avoid redundant re-scans for AI and game script files.
* Avoid iterating vehicle list to release disaster vehicles if there are none.
* Avoid quadratic behaviour in updating station nearby lists in RecomputeCatchmentForAll.

### Command line

* Add switch: -J, quit after N days.
* Add savegame feature versions to output of -q.

### Configure/build

* Changes to gcc/clang detection and flags
* Changes to version detection and the format of the version string.
* Minor CMake changes.

### Misc

* Use of __builtin_expect, byte-swap builtins, overflow builtins, and various bitmath builtins.
* Add various debug console commands.
* Cache font heights.
* Cache resolved names for stations, towns and industries.
* Change inheritance model of class Window to keep UndefinedBehaviorSanitizer happy.
* Various other misc changes and fixes to reduce UndefinedBehaviorSanitizer and ThreadSanitizer spam.
* Add a chicken bits setting, just in case.