Previously the CJK character detection defined only characters in the
range U+4000..U+AFFF as "CJK characters". This excludes an incredibly
large number of CJK characters within the BMP, let alone the whole two
planes dedicated to rarer CJK characters (the SIP and TIP). As a result,
a very large number of Chinese, Japanese, and Korean characters were not
detected as being CJK characters.
While slightly less elegant-looking, it is far more accurate to compute
the codepoint from the utf8 character and then see if it falls within
one of the defined CJK blocks. This is not future-proof against future
CJK ideograph extensions in future Unicode versions, but there is no
real way to accurately predict such changes so this is the best we can
do without accidentally treating characters explicitily defined as being
non-CJK in Unicode as CJK.
While we're at it, copy Lua 5.3's utf8.charpattern constant definition
so that we can more easily write utf8 iterators with string.gmatch (at
least in the interim until there is a rework of utf8 handling in
KOReader and everything is rebuilt on top of utf8proc).
Some unit tests are added for Korean and Japanese text, and the existing
unit tests needed a minor adjustment to handle the fact that
isSplittable now correctly detects CJK punctuation as a character to
compare against the forbidden split rules.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
* Geom:transformByScale:
* Apply the right scaling factor to the y axis
* Round in a more sensible fashion (àla fz_round_rect, since we pretty much exclusively use it in a similar fashion).
* Bump base (https://github.com/koreader/koreader-base/pull/1407)
Make 'em match backward & forward.
Now that we have working overrides and the gesture manager, trying to fit them in a weird superset of the top corner tapzones in a vain attempt to avoid bad interactions doesn't make much sense anymore, and just makes the Gesture Manager UI confusing.
Also make sure the corner zones override the L/R ones for double taps, like it's the case with other gestures.
Fix#7710
* Tear down FM instances properly
* Don't manhandle ReaderUI too much, and document when the tests do
actively broken shit, like bypassing safeties to open two // ReaderUI
instances.
We've managed to trip a few of those on dimen fields post-init but
pre-paintTo in a few weird coner-cases, a point at which dimen is often
nil.
ConfigDialog: Deal with that very thing in update()
Fix#7656
Ought to be faster than our naive array-based approach.
Especially for the glyph cache, which has a solid amount of elements,
and is mostly cache hits.
(There are few things worse for performance in Lua than
table.remove @ !tail and table.insert @ !tail, which this was full of :/).
DocCache: New module that's now an actual Cache instance instead of a
weird hack. Replaces "Cache" (the instance) as used across Document &
co.
Only Cache instance with on-disk persistence.
ImageCache: Update to new Cache.
GlyphCache: Update to new Cache.
Also, actually free glyph bbs on eviction.
* Ensure that going from one to the other tears down the former and
its plugins before instantiating the latter and its plugins.
UIManager: Unify Event sending & broadcasting
* Make the two behave the same way (walk the widget stack from top to
bottom), and properly handle the window stack shrinking shrinking
*and* growing.
Previously, broadcasting happened bottom-to-top and didn't really
handle the list shrinking/growing, while sending only handled the list
shrinking by a single element, and hopefully that element being the one
the event was just sent to.
These two items combined allowed us to optimize suboptimal
refresh behavior with Menu and other Menu classes when
opening/closing a document.
e.g., the "opening document" Notification is now properly regional,
and the "open last doc" option no longer flashes like a crazy person
anymore.
Plugins: Allow optimizing Menu refresh with custom menus, too.
Requires moving Menu's close_callback *after* onMenuSelect, which, eh,
probably makes sense, and is probably harmless in the grand scheme of
things.
* TimeVal: Log the results of the COARSE probes in debug logs
* GestureDetector: Print details of failed clock probes in debug logs
* GestureDetector: Skip the BOOTTIME probe when CLOCK_BOOTTIME is unsupported.
* Input: Decode ABS_DISTANCE events in debug logs
* Get rid of duplicated <linux/input.h> constants, use the FFI module everywhere (re #7536)
* Kobo: Get rid of the `touch_alyssum_protocol` quirk. Replace it by setting `main_finger_slot` to `1`, like on the H2O.
Requires https://github.com/koreader/koreader-base/pull/1344 & https://github.com/koreader/koreader-base/pull/1346 (fix#7485)
Assorted input fixes:
* Actually handle errors in the "there's a callback timer" input polling loop.
* Don't break timerfd when the clock probe was inconclusive.
Not directly related, but noticed because of duplicate onInputEvent handlers:
* HookContainer: Fix deregistration to actually deregister properly. "Regression" extant since its inception in #2933 (!).
* Made sure the three plugins (basically the trio of AutoThingies ;p) that were using HookContainer actually unschedule their task on teardown.
* ReaderDictionary: Port delay computations to TimeVal
* ReaderHighlight: Port delay computations to TimeVal
* ReaderView: Port delay computations to TimeVal
* Android: Reset gesture detection state on APP_CMD_TERM_WINDOW.
This prevents potentially being stuck in bogus gesture states when switching apps.
* GestureDetector:
* Port delay computations to TimeVal
* Fixed delay computations to handle time warps (large and negative deltas).
* Simplified timed callback handling to invalidate timers much earlier, preventing accumulating useless timers that no longer have any chance of ever detecting a gesture.
* Fixed state clearing to handle the actual effective slots, instead of hard-coding slot 0 & slot 1.
* Simplified timed callback handling in general, and added support for a timerfd backend for better performance and accuracy.
* The improved timed callback handling allows us to detect and honor (as much as possible) the three possible clock sources usable by Linux evdev events.
The only case where synthetic timestamps are used (and that only to handle timed callbacks) is limited to non-timerfd platforms where input events use
a clock source that is *NOT* MONOTONIC.
AFAICT, that's pretty much... PocketBook, and that's it?
* Input:
* Use the <linux/input.h> FFI module instead of re-declaring every constant
* Fixed (verbose) debug logging of input events to actually translate said constants properly.
* Completely reset gesture detection state on suspend. This should prevent bogus gesture detection on resume.
* Refactored the waitEvent loop to make it easier to comprehend (hopefully) and much more efficient.
Of specific note, it no longer does a crazy select spam every 100µs, instead computing and relying on sane timeouts,
as afforded by switching the UI event/input loop to the MONOTONIC time base, and the refactored timed callbacks in GestureDetector.
* reMarkable: Stopped enforcing synthetic timestamps on input events, as it should no longer be necessary.
* TimeVal:
* Refactored and simplified, especially as far as metamethods are concerned (based on <bsd/sys/time.h>).
* Added a host of new methods to query the various POSIX clock sources, and made :now default to MONOTONIC.
* Removed the debug guard in __sub, as time going backwards can be a perfectly normal occurrence.
* New methods:
* Clock sources: :realtime, :monotonic, :monotonic_coarse, :realtime_coarse, :boottime
* Utility: :tonumber, :tousecs, :tomsecs, :fromnumber, :isPositive, :isZero
* UIManager:
* Ported event loop & scheduling to TimeVal, and switched to the MONOTONIC time base.
This ensures reliable and consistent scheduling, as time is ensured never to go backwards.
* Added a :getTime() method, that returns a cached TimeVal:now(), updated at the top of every UI frame.
It's used throughout the codebase to cadge a syscall in circumstances where we are guaranteed that a syscall would return a mostly identical value,
because very few time has passed.
The only code left that does live syscalls does it because it's actually necessary for accuracy,
and the only code left that does that in a REALTIME time base is code that *actually* deals with calendar time (e.g., Statistics).
* DictQuickLookup: Port delay computations to TimeVal
* FootNoteWidget: Port delay computations to TimeVal
* HTMLBoxWidget: Port delay computations to TimeVal
* Notification: Port delay computations to TimeVal
* TextBoxWidget: Port delay computations to TimeVal
* AutoSuspend: Port to TimeVal
* AutoTurn:
* Fix it so that settings are actually honored.
* Port to TimeVal
* BackgroundRunner: Port to TimeVal
* Calibre: Port benchmarking code to TimeVal
* BookInfoManager: Removed unnecessary yield in the metadata extraction subprocess now that subprocesses get scheduled properly.
* All in all, these changes reduced the CPU cost of a single tap by a factor of ten (!), and got rid of an insane amount of weird poll/wakeup cycles that must have been hell on CPU schedulers and batteries..
* LuaSettings/DocSettings: Updated readSetting API to allow proper initialization to default.
Use it to initialize tables, e.g., fixing corner-cases in readerFooter that could prevent settings from being saved.
(Fixes an issue reported on Gitter).
* LuaSettings/DocSettings: Add simpler API than the the flip* ones to toggle boolean settings.
* Update LuaSettings/DocSettigns usage throughout the codebase to use the dedicated boolean methods wher appropriate, and clean up some of the more mind-bending uses.
* FileChooser: Implement an extended default exclusion list (fix#2360)
* ScreenSaver: Refactor to avoid the pile of kludges this was threatening to become. Code should be easier to follow and use, and fallbacks now behave as expected (fix#4418).
* flash_ui: Yield to the kernel between the HL and the UNHL/CB to let the EPDC do its thing in peace.
* UIManager: Handle nils in task scheduling arguments.
* SkimTo: Use the same, thicker chapter nav icons as ReaderSearch (fix#7326).
* SkimTo: The bookmark toggle button doesn't require a vsync flag.
* Fix typo in dropbox
Reported by @lescheck
* Rephrase text justification explanation more elegantly
* CoverBrowser: fix up some plurals
* Statistics: remove random use of template function
* Use ngettext for minute/minutes and second/seconds
* Change KB/MB/GB to kB/MB/GB SI units
- Add IconWidget, use it for icons instead of ImageWidget.
Specify icons by name only, look for them (with either
.svg or .png suffixes) in multiple directories (including
koreader/settings/icons/ to allow customizing them).
Don't crash when icon name not found, shown a black
background warning icon instead.
- Don't trust the icons' native sizes: replace
scale_for_dpi=true with width/height=DGENERIC_ICON_SIZE,
so all icons get the same (tunable) size - except in
a few specific use cases.
- Top and bottom menu bars: normalize, and have icons
properly centered in them, extend vertical line
separators up to the edges.
- TOC: adjust expand/collapse icons size to items size
This way we still get the passed/expected actual values in Busted's
output
Which helps not make this any more maddening than it already is to
update...
* Top menu & bottom menu tapzones are now full-width, thanks to the now sane override system.
* A separate, extra tapzone was also created for each of them to provide a taller, but narrower tapzone extension (one that doesn't clash with the prev/next tapzones).
* The footer tapzone was made much taller.
On devices where the event time is the uptime (time since
boot), we don't need to trash it and use TimeVal:now()
and lose precision. We can still use these times for
relative delays and durations computations, which is
mostly all we use them for.
We just need a real clock time in GestureDetector for
two timers (long-press and double tap), where we do
the event time type detection on the first event, on
all devices.
* Update the data collection format & handler to make it much less tortuous
* Update the pagecount & resync the stats on document layout changes
* Update the database schema to allow doing most queries against a SQL view that rescales the collected data to be accurate regardless of document layout (thanks to @marek-g for the SQL magic ;)).
* Add a "reset stats for current book" entry in the list of reset options, one that won't horribly break stats in said book ;).
* Fixed a couple of resource (SQL connection) leaks (in ReaderStatistics:getCurrentBookStats & ReaderStatistics:getCurrentBookStats).
* Flush stats to the DB on periodical metadata saves.
* Minor cosmetic tweaks to the code
* Uses bunch of new plumbing in base to configure screen rotations in hardware (koreader normally does this via blit buffer rotations, except for android).
* Some PB specific kludges that used to pollute core/framebuffer_linux are brought into PBs frontend driver.
A BackgroundRunner plugin instance will stop running
(rescheduling a check every 2 seconds) when there is no
(or no more) job to run.
Clients of this service now have to emit an event after
adding a job into PluginShare.backgroundJobs, so an
already loaded but stopped BackgroundRunner can notice
it and start running again.
* Revamped most actions that require an internet connection to a new/fixed backend that allows forwarding the initial action and running it automatically once connected. (i.e., it'll allow you to set "Action when Wi-Fi is off" to "turn_on", and whatch stuff connect and do what you wanted automatically without having to re-click anywhere instead of showing you a Wi-Fi prompt and then not doing anything without any other feedback).
* Speaking of, fixed the "turn_on" beforeWifi action to, well, actually work. It's no longer marked as experimental.
* Consistently use "Wi-Fi" everywhere.
* On Kobo/Cervantes/Sony, implemented a "Kill Wi-Fi connection when inactive" system that will automatically disconnect from Wi-Fi after sustained *network* inactivity (i.e., you can keep reading, it'll eventually turn off on its own). This should be smart and flexible enough not to murder Wi-Fi while you need it, while still not keeping it uselessly on and murdering your battery.
(i.e., enable that + turn Wi-Fi on when off and enjoy never having to bother about Wi-Fi ever again).
* Made sending `NetworkConnected` / `NetworkDisconnected` events consistent (they were only being sent... sometimes, which made relying on 'em somewhat problematic).
* restoreWifiAsync is now only run when really needed (i.e., we no longer stomp on an existing working connection just for the hell of it).
* We no longer attempt to kill a bogus non-existent Wi-Fi connection when going to suspend, we only do it when it's actually needed.
* Every method of enabling Wi-Fi will now properly tear down Wi-Fi on failure, instead of leaving it in an undefined state.
* Fixed an issue in the fancy crash screen on Kobo/reMarkable that could sometime lead to the log excerpt being missing.
* Worked-around a number of sneaky issues related to low-level Wi-Fi/DHCP/DNS handling on Kobo (see the lengthy comments [below](https://github.com/koreader/koreader/pull/6424#issuecomment-663881059) for details). Fix#6421
Incidentally, this should also fix the inconsistencies experienced re: Wi-Fi behavior in Nickel when toggling between KOReader and Nickel (use NM/KFMon, and run a current FW for best results).
* For developers, this involves various cleanups around NetworkMgr and NetworkListener. Documentation is in-line, above the concerned functions.
* Mangle stupid defaults test so that it compares tables, and not a non-deterministic string representation of one.
It's still extremely dumb and annoying to update. (i.e., feel free to kill it with fire in a subsequent PR, I think everybody would cheer).
* Rewrite DepGraph to be deterministic
i.e., fully array based, no more hashes, which means no more pairs randomly re-ordering stuff.
Insertion order is now preserved.
Pretty sure a couple of bugs have been fixed and/or added along the way
;p.
* Resync frontend/apps/filemanager/lib/md.lua w/ upstream
And use orderedPairs in the attribute parsing code, just to make that stupid test happy.
* Allow locking the gyro to the current screen mode (i.e., orientation).
* Tweak the "sticky rota" option to work both ways
* More rotation constant usage instead of magic numbers
* landscape FM / Refactor rotation
refactor and simplify the orientation handling code. the user generally cares about the rotation (what direction the device is facing) and not about if koreader is displaying in portrait or landscape mode
* bump base
update luasocket, libjpeg-turbo, curl
add logging to evernote-sdk-lua
update framebuffer for proper rotation
joins calibre metadata search and calibre wireless connections into a single plugin
search metadata changes:
- search directly into calibre metadata files.
- search can be performed on more than one library (configurable from a menu)
- device scans now find all calibre libraries under a given root
- search options can be configured from a menu. (case sensitive, find by title, author and path)
- removed legacy global variables.
- *option* to search from the reader
- *option* to generate a cache of books for faster searches.
calibre wireless connection changes:
- keep track of books in a library (includes prunning books from calibre metadata if the file was deleted locally)
- remove files on device from calibre
- support password protected connections
- FM integration: if we're in the inbox dir it will be updated each time a book is added or deleted.
- disconnect when requested by calibre, available on newer calibre versions (+4.17)
- remove unused opcodes.
- better report of client name, version and device id
- free disk space checks for all calibre versions
- bump supported extensions to match what KOReader can handle. Users can override this with their own list of extensions (or from calibre, by configuring the wireless device).
* Update bundled fonts
* Minor FreeFont update
* Noto Sans CJK downgrade to fix a number of issues w/ the git version
* Initial import of Noto Naskh Arabic
* Replace Noto Sans Arabic UI by Noto Naskh Arabic in the CRe fallback
list
Bump crengine:
- Text: fix occasional BiDi bad word splitting
- Font: fix HB fallback measurement/drawing mismatches
- Font: do not add letter spacing on diacritics
- Text: tunable use of letter spacing for justification
- Text: dont adjust space after initial quotation mark/dash (rework)
- Text: fix possible bad widths after collapsed spaces
xtext.cpp: pick crengine fix with HB fallback
measurement/drawing mismatches
Adds "Word Expansion" to bottom Font size menu, and moved
"Word Spacing" there too, to balance the panels' heights
to 4 items.
Includes:
- Fix a few clang-tidy warnings
- Add support for <img src="data:image/png;base64,...>
- XML parsing: add more HTML5 named entities, optimize search
- Text: fix standalone BR not making an empty line
- Fix BR with "display: block" not making an empty line
- Fix hyphens from soft-hyphens not part of highlighted segments
- Use libunibreak for line breaking
- Adds TextLangMan for text typography by language
Tweak ReaderHyphenation to work with the new TextLangMan
(even if it will be replaced soon by ReaderTypography).
* Kindle FL shenanigans
Either fix#5986, or break a whole crapload of weird corner-cases.
Possibly the insane AutoFrontLight checks.
* Make fl step 0 usable on devices where 0 doesn't turn the light off.
By fudging an extra step on our own side.
* Also, add some debug logging around wmctrl to try to figure out what's
happening there...
* Allow running Shell/Python scripts from the FM
* Show an InfoMessage before/after running the script
Since we're blocking the UI ;).
* Allow running scripts from the favorites menu, too.
Strings are prefilled by msgmerge with closely matching ones to reduce the amount of labor required by the translator. Often the string requires only minor adjustment, making the process faster, and when the string isn't a good match it's no big deal. However, these so-called fuzzy strings shouldn't be treated as if they were actually translated.
Fixes the effect seen in <https://github.com/koreader/koreader/issues/5806>.
Bump base for util.fsyncOpenedFile() and util.fsyncDirectory().
Use these to force flush to storage device when
saving luasetting, docsetting, and history.lua.
Also dont rotate them to .old until they are at least
60 seconds old.
Also make auto_save_paging_count count right.
Also bump crengine: open (as text) small or empty files
Happens with "Inverse reading order", which re-registers
the Reader tap forward/backward zones and would prevent
tap menu and bookmark from working.
Before, when removing them, we would lose all the override=
set on them by other touch zones (tap menu, bookmarks...),
and so they were no more ensured after re-adding the zone.
So, make sure we don't lose that info.
Includes:
- New option to tune word spacing: space width scale percent
- Text: look for hyphenation in more words if needed
- CSS: fix "hyphens:none" should override inherited "hyphens:auto"
- getHtml(): grab dir= and lang= attributes from upper nodes
Replace our Word Gap/Space condensing toggle/setting with
a new Word Spacing toggle/setting, made of 2 values:
- 1st number scales the normal width of spaces in all font
(100% uses the font space width untouched)
- 2nd number applies after the 1st has been applied, and
tells how much these spaces can additionally be condensed
to make more text fit on a line.
- fix incorrectly shown awake, sleeping, charging and discharging,
- remove unneeded debug mode and logging to external file,
- prevent showing values like inf, -inf, nan in estimated times
(we now show "n/a"), and values below zero,
- show extra confirm box when we want to reset data,
- show time in format xxhxxm instead of only pure minutes,
- check at initialization that the device was charging when it was
turned off (battery level larger than at the time of exit).
Includes:
- Fonts: round FT metrics instead of floor'ing them
- Fonts: switch to no hinting when native hinting fails
- Fonts: fix issue with Harfbuzz fallback font drawing
- Text: split measuring and word drawing by unicode script
- Page splitting: fix small memory leak
- Fix "background-color: black" ignored on inline elements
- Fix decoding of recent MOBI files
- Hardcoded elements list: add <font>
base/xtext.cpp: small cleanup, no logic change
closes#5612
the change in `ReaderView:recalculate()` causes the viewable page size to be calculated by not including the footer, causing the text not to get cut off.
since the page area was not drawing under the footer `ReaderView:drawPageSurround()` had to be fixed to draw the margin under the footer so when tapping the footer off the area should be dran the background color.
Alternative code to size, split lines and draw text with
the help of the xtext Lua C module.
Enabled by default (can be disabled via an added menu
item in "Developer options").
New properties can be specified by calling widgets, only
used when xtext is used:
- lang: use specified language instead of the UI language
- para_direction_rtl: true/false to override the default
direction for the UI language
- auto_para_direction: detect direction of each paragraph
in text
- alignment_strict: prevent the inversion of the specified
alignment= that is done when a paragraph direction is
set or detected as RTL.
Also: fix possible memory leak (present even when not using xtext)
by calling :free() in onCloseWidget() like it's done in ImageWidget.
Alternative code to size and draw text with the help
of the xtext Lua C module.
Enabled by default (can be disabled via an added menu
item in "Developer options").
New properties can be specified by calling widgets, only
used when xtext is used:
- lang: use specified language instead of the UI language
- para_direction_rtl: true/false to override the default
direction for the UI language
- auto_para_direction: detect direction of each paragraph
in text
It might be slower, but is needed to properly display
books in arabic, indic...
Also bump crengine: use FreeType "light" hinting algorithm,
which hints on the y-axis only (and so avoid messing with
advances and kerning on the x-axis).
* Restore '+' prefix when charging in the footer w/ letters prefixes
Regression since #5537
* Don't use a full battery icon in the menu entry
Make it clear that one isn't in-sync with the current gauge
* Unify battery meters look'n feel
Switch to proper (vertical) battery icons (one per 10% steps).
Fix#5535
* Bump fonts (https://github.com/koreader/koreader-fonts/pull/7)
Because we need the new symbols from nerdfonts ;).
* Don't break `doc_pages` updates when it's completely disabled. (c.f., the commit's comment, doing it on every page turn seems a bit overkill, but I get that it's probably done that way because it was easier ;)).
* Detect the empty footerTextGenerator properly in `_updateFooterText` (it's `""`, not `nil`).
* Abort `_updateFooterText` early when the footer is invisible (and has been for a while/ever, i.e., when not requesting a repaint, like a mode switch would).
* Never send `SetBottomPageMargin` events twice.
* Minor tweaks to touchmenu & configdialog to avoid useless repaints (don't redraw the reader/fm unless we absolutely need to).
* Make sure we show the progress bar alone instead of nothing when disabling the last enabled mode in the menu.
When reasonably possible, the program should only crash in debug mode.
Adds a couple of extra unit tests to prevent regressions and adds docs.
Fixes <https://github.com/koreader/koreader/issues/5356>.
This commit standardizes the various todos around the code a bit in a manner recognized by LDoc.
Besides drawing more attention by being displayed in the developer docs, they're also extractable with LDoc on the command line:
```sh
ldoc --tags todo,fixme *.lua
```
However, whether that particular usage offers any advantage over other search tools is questionable at best.
* and some random beautification
- show icons or letters as prefix of items
- various footer separators
- progress percentage format with decimal digits
- time in 12/24 format
- two duration formats (1:30, 1h30')
- move some options into Settings submenu
Aleksa Sarai
yparitcher
NiLuJe
Martín Fernández
Brian Hughes
Frans de Jonge
Steady Quad
Alex Vanderpot
poire-z
Martín Fdez
ezdiy
Mustafa Ali Mutlu
Robert