## JGR's Patchpack: Low level changes This document describes low-level changes to the codebase which are not generally visible when actually running/playing the game, this is a non-exhaustive list. This document does not describe the player-visible changes/additions described in the main readme. ### Crash logger and diagnostics * Additional logged items: current company ID, map size, configure invocation, thread name, recently executed commands, static NewGRFs. * Additional logged platform-specific items: detailed OS version (Unix), signal details (Unix, Mac), exception record data (Windows). * Better handling of crashes which occur in a non-main thread (ask the main thread to do the crash screenshot and savegame). * Support logging register values on Unix and Mac. * Support using libbfd for symbol lookup and line numbers (gcc/clang). * Support using gdb/lldb if available to add further detail to the crashlog (Unix, Mac). * Support using sigaction and sigaltstack for more information and correct handling of stack overflow crashes (Unix). * Attempt to log stack overflow and heap corruption exceptions (Windows). * Demangle C++ symbols (Unix). * Attempt to handle crashes which occur within the crashlog handler, by skipping or only partially writing the faulting section. * Emit a "crash" log, savegame and screenshot on multiplayer desync. * Add crash/desync information to output screenshot and savegame files. * Multiplayer server and client exchange desync logs after a desync occurs. * Multiplayer clients send state hashes and random values since the last sync to the server after a desync, to identify which frame first diverged. * Decrease sync frame period when desync occurs. #### Assertions * Various assertions are extended to log further information on failure. * Various assertions which check the state of a tile are extended to dump the tile state (m1 - m8, etc.) on failure. #### Scope annotations * Scopes (in the main thread) can be annotated with a functor/lambda which is called in the event of a crash to provide further information to add to the crash log. #### NewGRF debug window * Add various supplementary non-GRF information, e.g. vehicle variables and flags. * Add NewGRF sprite group dumping and related functionality. #### Logging * Add yapfdesync, linkgraph, sound, and command log levels. * Extend desync and random logging. ### Map * Store tunnel start/end pairs in a pool, indexed in the start/end tiles. * Set bit in map if level crossing is possibly occupied by a road vehicle. * Cache inferred one-way state of road tiles. * De-virtualise calls to AnimateTile(). * Cache animated tile speed. * Cache whether water tiles have water for all neighbouring tiles. * Improve performance of arctic snow line checks. ### Viewport * Cache bridge/tunnel start and ends. * Cache station sign bounds. * Split sprite sort regions when more than 60 sprites present. * Reduce unnecessary region redraws when scrolling viewports. * Reduce viewport invalidation region size of track reservation and signal state changes. * Cache landscape background in map mode. * Partial parallelisation of non-map mode viewport rendering. ### Rendering * Track dirty viewport areas seperately from general screen redraws, using a zoom-level dependant sized grid. * Use a rectangle array for general screen redraws instead of a block grid. * Add a dirty bit to windows and widgets, for redrawing entire windows or widgets. * Clip drawing of window widgets which are not in the redraw area. * Reduce unnecessary status bar and vehicle list window redraws. * Filter out tile parts which are entirely outside the drawing area, within DrawTileProc handlers. * Improve performance of drawing rail catenary. * Cache which window types are currently shown. ### Data structures * Various data structures have been replaced with B-tree maps/sets (cpp-btree library), or robin-hood hash maps. * Various lists have been replaced with vectors, or other data structures. * Various deques and queues have been replaced with ring buffers. * Remove mutexes from SmallStack, only used from the main thread. * Add a third parameter p3, and an auxiliary data mechanism to DoCommand/CommandContainer. * Add a free bitmap for pool slots. * Maintain free list for text effect entries. * Many fields have been widened. * Change underlying data structures for ScriptList, create reverse mapping on demand instead of unconditionally. * Split GoodsEntry structure. ### Vehicles * Cache the sprite_seq bounds. * Index the order list in a vector. * Observe the operation of the NewGRF when getting the vehicle image/sprite, and elide further calls to the NewGRF if it can be determined that the result will be the same. * Update train/road vehicle image/sprite on demand (i.e. when on screen) when image is continuously updated by GRF. * Add consist flag for the case where no vehicles in consist are on a slope. * Add vehicle flag to mark the last vehicle in a consist with a visual effect. * Index the vehicle list in per type arrays for use by CallVehicleTicks. * Cache whether the vehicle should be drawn. * Pre-compute engine refit capacity callbacks if possible. ### Network/multiplayer * Add supplementary information to find server UDP packets and reply in an extended format with more info/wider fields if detected. * Paginate UDP packets longer than the MTU across multiple packets. * Use larger "packets" where useful in TCP connections. * Send vehicle caches from network server to clients to avoid desyncs caused by non-deterministic NewGRFs. * Change network protocol to send server/join and rcon passwords in an encrypted form (key exchange) instead of in clear text. * Encrypt the contents of rcon messages to the server and any responses. ### Sprites/blitter * Add a fast path to Blitter_32bppAnim::Draw. * Replace sprite cache implementation. * Only cache sprites at the currently required zoom levels instead of all of them. * Add brightness adjusting modes to non-8bpp blitters. ### Link graph * Completely change link graph job scheduling to make the duration of a job and the number of jobs per thread instance variable according to the estimated size of the job. * Various use of custom allocators, etc. * Early abort link graph threads if abandoning/quitting the game. * Various forms of caching and incremental updates to the link graph overlay. * Change FlowStat from an RB-tree to a flat map with small-object optimisation. * Change FlowStatMap from an RB-tree to a B-tree indexed vector. * Change LinkGraph::EdgeMatrix to a sparse storage format. * Replace MCF Dijkstra RB-tree with B-tree. * Reduce performance issues when deleting stale links with refit to any cargo. * Dynamically adjust accuracy parameters in MCF 1st pass to avoid computing large numbers of excessively small flows. ### Pathfinder * YAPF: Reduce need to scan open list queue when moving best node to closed list ### Save and load * Feature versioning, see readme and code. * Extend gamelog to not truncate version strings. * Save/load the map in a single chunk, such that it can be saved/loaded in one pass. * Various other changes to savegame format and settings handling, see readme and code for details. * Replace read/write accessors and buffering. * Perform savegame decompression in a separate thread. * Pre-filter SaveLoad descriptor arrays for current version/mode, for chunks with many objects. * Support zstd compression for autosaves and network joins. ### AI/GS * [AI/GS script additions](docs/script-additions.html). * Add AI/GS method to get current day length. * Add GS method to create river tiles. * Add AI/GS methods related to road and tram types. * Add workaround for performance issues when attempting to create a town when no town names are left. * Fixup a GS otherwise inconsistent with day length. ### NewGRF * [NewGRF specification additions](docs/newgrf-additions.html). * Add workaround for a known buggy NewGRF to avoid desync issues. * Apply many optimisations to VarAction2 deterministic sprite groups. * Avoid making callbacks which can be pre-determined to be unhandled, or which can be statically determined ahead of time. * Avoid animating industry tiles which are not actually animated in the current layout. * Setting the animation frame to its current value no longer triggers a redraw. * Animation is not started if it can be determined that it would stop immediately. * Avoid unnecessarily triggering or redrawing NewGRF houses. ### SDL2 * Update whole window surface if >= 80% needs updating. * Only pass a single rectangle to SDL_UpdateWindowSurfaceRects to prevent screen tearing. * Allow using the hash key (#) as a hotkey. ### Other performance improvements * Use multiple threads for NewGRF scan MD5 calculations, on multi-CPU machines. * Avoid redundant re-scans for AI and game script files. * Avoid iterating vehicle list to release disaster vehicles if there are none. * Avoid quadratic behaviour in updating station nearby lists in RecomputeCatchmentForAll. ### Command line * Add switch: -J, quit after N days. * Add savegame feature versions to output of -q. ### Configure/build * Changes to gcc/clang detection and flags * Changes to version detection and the format of the version string. * Minor CMake changes. ### Misc * Use of __builtin_expect, byte-swap builtins, overflow builtins, and various bitmath builtins. * Add various debug console commands. * Cache font heights. * Cache resolved names for stations, towns and industries. * Change inheritance model of class Window to keep UndefinedBehaviorSanitizer happy. * Various other misc changes and fixes to reduce UndefinedBehaviorSanitizer and ThreadSanitizer spam. * Add a chicken bits setting, just in case.