This is a maintenance PR.
With scheduling the next autodim_check on every tap, we can keep the device longer in standby mode. So we have more power savings.
The long story: The current version works like that:
Let's say I set the idle-time to 3 min and I need 2.5 min to read a page.
Start at 0 min (next scheduled check is at/in 3 min)
Read and tap to turn page after 2.5 min (next scheduled check is at 3 min - in 0.5 min)
At 3 min there is a scheduled_check (wakeup) -> reschedule check in 3 min.
Read and tap to turn page after 2.5 min (no wakeup)
Then after 1 min there is a wakeup -> reschedule.
---> So I get an additional wakeup on every second page.
This PR solves the problem: During normal reading I don't get any wakeup during reading. (The only drawback is, that one function is unscheduled/scheduled on every tap, but that is no big deal).
* Get title, author and number of pages from document's metadata
* Add exportable_title parsed from document's title, used in export file generation.
* JSON: export number of pages
Add a copy button and save word context (off by default), shown
via the three-dot menu of word entries.
Also some db refactoring and minor UI improvements:
- a dedicated book title table in order to shrink db size by storing
references to title names instead of repeated actual strings,
- alignment of different forms of the "more" button and possible
clipped words in translations.
- fix plugin name so it can be disabled
This was originally designed for finer-grainbed control (specifically,
to correct by the amount of time spent in standby), but the final fix
only ever sets this to 0.
Simplify and rename (consumeInputEarlyAfterPM) to avoid useless computations.
Long story short: the LeaveStandby event is sent via `tickAfterNext`, so if we tear down the plugin right after calling it (in this case, that means that the very input event that wakes the device up from suspend is one that kills ReaderUI or FileManager), what's in UIManager's task queue isn't the actual function, but the anonymous nextTick wrapper constructed by `tickAfterNext` (c.f.,
https://github.com/koreader/koreader/issues/9112#issuecomment-1133999385).
Tweak `UIManager:tickAfterNext` to return a reference to said wrapper, so that we can store it and unschedule that one, too, in `AutoSuspend:onCloseWidget`.
Fix#9112 (many thanks to [@boredhominid](https://github.com/boredhominid) for his help in finding a repro for this ;)).
Re: #8638, as the extra debugging facilities (i.e., ebb81b9845) added during testing might help pinpoint the root issue for that one, too.
Also includes a minor simplification to `UIManager:_checkTasks`, and various other task queue related codepaths (e.g., `WakeupMgr`) ;).
Changed:
- select multiple targets and export to them in a single click.
- local targets (html, json and text) now are timestamped. Exporting booknotes on already exported documents will generate a new file with all the highlights present at export time. Previous files won't be deleted.
Fixed:
- chapters are now correctly represented in html output.
- json issues when exporting the whole history.
- joplin and readwise crashes when they're unable to reach the server
- joplin update notes mechanism.
- joplin is able to recreate the notebook if the user deletes or renames its current one.
- highlights of read-only documents are also added when exporting the whole history (affects mostly android, might affect desktop targets)
Co-authored-by: Utsob Roy <roy@utsob.me>
Titlebar button navigation in menu, including ListMenu and MosaicMenu in FileManager, History, Favourites and Shortcuts.
Hide show password checkbox in non-touch devices
* Disable all non power management related input during suspend. (This prevents wonky touch events from being tripped when closing a sleep cover on an already-in-suspend device, among other things).
* Kobo: Use our WakeupMgr instance, not the class.
* WakupMgr: split `removeTask` in two:
* `removeTask`, which *only* takes a queue index as input, and only removes a single task. Greatly simplifies the function (i.e., it's just a `table.remove`).
* `removeTasks`, which takes an epoch or a cb ref, and removes *every* task that matches.
* Both of these will also *always* re-schedule the next task (if any) on exit, since we can have multiple WakeupMgr tasks queued, but we can only have a single RTC wake alarm set ;).
* `wakeupAction` now takes a `proximity` argument, which it passes on to its `validateWakeupAlarmByProximity` call, allowing call sites to avoir having to duplicate that call themselves when they want to use a custom proximity window.
* `wakeupAction` now re-schedules the next task (if any) on exit.
* Simplify `Kobo:checkUnexpectedWakeup`, by removing the duplicate `WakerupMgr:validateWakeupAlarmByProximity` call, now that we can pass a proximity window to `WakeuoMgr:wakeupAction`.
* The various network activity timeouts are now halved when autostandby is enabled.
* Autostandby: get rid of the dummy deadline_guard task, as it's no longer necessary since #9009.
* UIManager: The previous change allows us to simplify `getNextTaskTimes` into a simpler `getNextTaskTime` variant, getting rid of a table & a loop.
* ReaderFooter & ReaderHeader: Make sure we only perform a single refresh when exiting standby.
* Kobo: Rewrite sysfs writes to use ANSI C via FFI instead of stdio via Lua, as it obscured some common error cases (e.g., EBUSY on /sys/power/state).
* Kobo: Simplify `suspend`, now that we have sane error handling in sysfs writes.
* Kobo.powerd: Change `isCharging` & `isAuxCharging` behavior to match the behavior of the NTX ioctl (i.e., Charging == Plugged-in). This has the added benefit of making the AutoSuspend checks behave sensibly in the "fully-charged but still plugged in" scenario (because being plugged in is enough to break PM on `!canPowerSaveWhileCharging` devices).
* AutoSuspend: Disable our `AllowStandby` handler when auto standby is disabled, so as to not interfere with other modules using `UIManager:allowStandby` (fix#9038).
* PowerD: Allow platforms to implement `isCharged`, indicating that the battery is full while still plugged in to a power source (battery icon becomes a power plug icon).
* Kobo.powerd: Implement `isCharged`, and kill charging LEDs once battery is full.
* Kindle.powerd: Implement `isCharged` on post-Wario devices. (`isCharging` is still true in that state, as it ought to).
Since the bookmarks table is sorted based on the bookmarks'
position in the book, it is possible to sort clippings in
the exporter plugin based on that.
* Make sure AutoStandby cohabitates nicely with AutoSuspend (regression since #8985), specifically:
* Disable standby during suspend.
* Ensure that leaving standby restores the scheduled suspend properly, with the appropriate remaining amount of time based on the last user input.
* Handle devices with an auxiliary battery better when scheduling suspend (assume it's only charging when the aux battery is charging, not the ereader's).
* Tweak debug logging to be able to remote debug corner-cases more easily without requiring code changes.
* Fix erroneous behavior when awoken from standby by something that doesn't trigger an InputEvent Event (e.g., rtc alarm, gyro, or random bullshit).
Specifically, don't forcibly unschedule/schedule on every input event, instead, let the scheduled task figure out if the deadline came to pass or not ;).
c.f., https://github.com/koreader/koreader/pull/8970#issuecomment-1092775830
Besides getting rid of some overhead, this allows proper scheduling after a task that would have blocked for longer than the standby timeout.
So we actually have a chance of understanding why it fails to load ;).
(Also, properly require posix FFI stuff, in the unlikely event they aren't already in the main namespace).
Allows the device to go into standby (if available in `/sys/power/state`) to save power.
Adds an entry in the device menu to tune the timeout for standby.
(Shows total standby- and suspend-time in system statistics.)
My volumize PR introduced too many issues. Revert to previous version of NewsDownloader until they can be resolved.
Fixes#8867
Reintroduces #8799, #6234
I've implemented a better JSON export format that removes redundant lists and objects and introduces the `entries` key containing all the entries.
It also add `drawer` values from highlight so that user can use this piece of metadata to generate desirable output.
* Calibre plugin - Series/Tag browser updated to show back button.
* Handling page number
* Few updates to search logic:
- Moved "find books" option from CalibreSearch:find(option)
into CalibreSearch:browse(option) function. This way all search options
are handled in one place only.
- Menu is created only once and it is in CalibreSearch:browse(option)
function. This is where it is also invoked with UIManager:show(w).
Switching between different menu content (tags/series/books) is
done using CalibreSearch:switchResults function which will internally
call Menu:switchItemTable. Previously menu was being instantiated
in 2 places depending on are we searching books or series/tags
- Return arrow navigation: Border around Menu is gone to give space
for back arrow. Navigation seems to be working fine so far but will
give it some time to test in case I find any other bugs
`Manual` could mean several things, most notably some kind of documentation and something you do by hand.
The added context is partially to aid translators and partially to preclude potential collisions in the future.
Cf. <https://github.com/koreader/koreader-translations/pull/138>.
* Rejig frontlight warmth API to more closely match the existing API, and, hopefully, clarify some of its quirks, and reduce boilerplate and duplicate code in platform implementations.
* Tweak Kindle:setDateTime to prefer using the platform's custom script, as in interacts better with the stock UI. And make the fallbacks handle old busybox versions better.
* Add Kindle PW5 support ;).
* Add warmth support to the Kindle platform.
* Random TextBoxWidget cleanups: make sure we immediately free destroyed instances.
* FrontLightWidget: Refactor to make it slightly less obnoxious to grok and update; i.e., separate layout from update, and properly separate brightness from warmth handling. Move to simpler widgets instead of reinventing the wheel.
* TextBoxWidgets: Implement `setText` to match TextWidget's API, as some callers may be using the two interchangeably (i.e., Button).
* NaturalLightWidget: Make sure we pass a string to InputText
* InputText: Add debug guards to catch bad callers not passing strings ;).
FocusManager: fix round x use y layout
FocusManager: add tab and shift tab focus navigation support
FocusManager: handle Press key by default
FocusManager: make sure selected in instance level
FocusManager: add hold event support
FocusManager: Half move instead of edge move
FocusManager: add keymap override support
FocusManager: refocusWidget will delegate to parent FocusManager
Focusmanager: refocusWidget can execute on next tick
inputtext: can move out of focus on back
inputtext: fix cannot exit for non-touch device
inputtext: fix cannot input text with kindle dx physical keyboard
fontlightwidget: add non-touch support
datetimewidget: add non-touch support
datetimewidget: fix set date failed in kindle DX, fix datetimewidget month range to 1~23 by default
datetimewidget: make hour max value to 23
multiinputdialog: add non-touch support
checkbox: focusable and focus style
virtualkeyboard: no need to press two back to unfocus inputtext
virtualkeyboard: collect FocusManager event key names to let VirtualKeyboard disable them
openwithdialog: add non-touch support
inputdialog: can close via back button
enable all InputDialog and MultiInputDialog can be close by back
keyboardlayoutdialog: non-touch support
readertoc: non touch device can expand/collapse in toc
bookstatuswidget: non touch support
keyvaluepage: non-touch support
calendarview: non-touch support
Currently the progress sent to the server can be either a string or an int (depending on whetther the document has pages).
The following are both payload sent from koreader to the server.
```
{"percentage":0.005,"device":"device_name","device_id":"B78EA04ACC3A453DBA220D720C0BE102","document":"348e34463a44ba68659fc6fe814a6778","progress":3}
```
where document `348e34463a44ba68659fc6fe814a6778` is a pdf file.
```
{"percentage":1,"device":"device_name","device_id":"B78EA04ACC3A453DBA220D720C0BE102","document":"4eb484b229696cb39cd8fe5495aa1bbe","progress":"\/body\/DocFragment[30]\/body\/p\/img.0"}
```
where document `4eb484b229696cb39cd8fe5495aa1bbe` is an epub file.
This may add extra work to the backend server. A few commits were added to my personal fork of [kosyncsrv](https://github.com/yeeac/kosyncsrv) (a kosync backend server). kosyncsrv initially tries to decode progress as a string. It then failed on document with pages (in which, progress is just integer page number). I then change the field's type, only to [revert it later](8a642e31a0).
I believe it is more appropriate for us to fix the progress type to string.
(1) ButtonDialogTitle: new method setTitle()
(2) OPDSbrowser
-use setTitle() instead of open/close ButtonDialogTitle (no visual changes to the download dialog)
-reduce logger.info output to avoid flooding crash.log
(3) CloudStorage
-enhance download dialog (similar to OPDS), much optimized code of downloadFile
-fix a bug: cannot create new folder or upload files to the root of the Dropbox storage
Reset self.kv stack/link on showing first widget.
This avoids return arrow to possibly show previously
displayed old and unrelated KeyValuePage.
Note: when closing a leaf KeyValuePage, the previous
one will leak as self.kv.
Fixes current page statistics (duration) ignored when
document closed, or when suspending and resuming.
On suspend/resume, update the page start ts by the
time spent sleeping, so its duration will be the time
spent on it not sleeping.
New real terminal emulator, replacing the old plugin.
The emulator is basically a vt52 terminal (enriched with
some ANSI-sequences, as ash, vi and mksh don't behave well
on a vt52 term).
So far working: ash, mksh, bash, nano, vi, busybox, watch...
The input supports: tab-completion; cursor movement;
backspace; start of line, end of line (long press);
page up, page down (long press).
User scripts may be placed in the koterm.koplugin/scripts/
folder, aliases can be put in the file aliases and startup
command in the file profile.user in that folder.
In this PR https://github.com/koreader/koreader/pull/8637
It worked on my sev environnement, but when updating my device, there
were synchronization errors.
Apparently sometimes documents do not have a summary.
To avoid an error during the synchronisation this change adds checks
when getting tags for the document.
For consistency with BookMap and PageBrowser widgets
where swipe south (the usual gesture to quick close)
can't be used for closing and we had to use any
multiswipe instead, allow any multiswipe to close
these other fullscreen widgets too:
Menu (ToC, Bookmarks), KeyValuePage, ImageViewer,
BookStatusWidget, ReaderProgress, CalendarView.
When this feature is enabled on a feed and that feed is
synced, all new feed entries will be collected into a new
single EPUB file. This is achieved by implementing a feed
history feature (downloaded feeds are added as M5D hashes
to a LuaSettings file), and by introducing additional
methods into epubdownloader.lua that allow for multiple
HTML documents to be added into single EPUB file.
regression from #8491
papered over in #8582Fixes#8644#8687
Root cause was util.secondsToClockDuration (called by duration) takes seconds and we were giving it hours as `Change per hour` needs hours
- Book map: shows a map of content, including TOC,
boomarks, read pages, non-linear flows...
- Page browser: shows thumbnails of pages.
- ReaderThumbnail: new Reader module that provides
a service for generating thumbnails of book pages.
It makes available these 2 new fullscreen widgets.
- ReaderBookmark, ReaderLink, Statistics: add methods
to return new views of bookmarks, previous locations
and read pages, that are needed by BookMapWidget.
- ReaderToc: compute TOC max_depth.
- ReaderBookmark, ReaderHighlight: send events on
bookmark add/update/remove so thumbnails of the
pages impacted can be trashed.
Changes in accordance with #8535:
-add html and body tags to support older embedded browsers
-bookmark note formatting: grey letters, indented, without title
(Massive changes are caused with the indentation along the tags).
This extends exporter.koplugin with support for [Readwise.io](https://readwise.io), a highlight/notes aggregation service.
[Readwise API documentation](https://readwise.io/api_deets)
This additionally improves the highlight exporter's ability to get the correct title and author of a document, by checking actual metadata instead of inferring from filename. It also includes a modification to the plugin's highlight parsing logic to separate the highlight contents in `.text` from the notes in `.note`. This change actually fixes an existing bug in the HTML export template note.tpl, which has been missing notes because of the lack of the `.note` field.
Move Default and extra buttons above Cancel/OK.
Default values shown in the default button.
Precisions can be set for both values separately.
Minor geometry fix for consistence with SpinWidget.
Allow setting a folder as a book shortcut, with 2 options:
- open file browser: opens the FM in that folder
- last book: opens the most recently read book (via
ReadHistory) in that folder
I've found that some OPDS catalogs have multiple downloads of the same filetype, but optimized or formatted in different ways. The Title of the download is much more descriptive in this case, so I thought it would be better to display the title if available.
The OPDS catalog at https://standardebooks.org/opds is a good example. Note how entries in https://standardebooks.org/opds/new-releases have three different epub downloads, titled "Recommended compatible epub", "Advanced epub", and "Kobo Kepub epub".
This plugin provides support for Japanese deinflection during lookup as
well as making long-hold word selection actually select whole words
properly. With this plugin, word lookups in Japanese text in KOReader
become much easier, and no longer requires users to use special
dictionaries that have synonym-based deinflection rules defined (which
were always fairly annoying to use).
The basic idea and deinflection data for this plugin come from
Yomichan (which is also a GPL-3.0+ project), but everything was
implemented specifically for KOReader.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Reword "Purge .sdr" to "Reset settings".
When purging, remove only the known document metadata
files, and not those for a document with the same name but
a different suffix.
* Don't run multiple collectSubprocesses tasks in parallel.
UIManager:scheduleIn doesn't return anything, using that as a boolean trap made no sense.
* Make the in-place removal of collected pids use a slightly more common idiom
(table.remove in a reverse iteration of said array).
cores
* Only keep a single core online most of the time.
* Device: Add an enableCPUCores method to allow controlling the amount of
online CPU cores.
* Move the initial core onlining setup to Kobo:init, instead of the startup script.
* Enable two CPU cores while hinting new (e.g., cache miss) pages in PDF land.
* Enable two CPU cores while processing book metadata.
* Drive-by fix to isolate the DocCache pressure check to KoptInterface
and actually apply it when it matters most (e.g., k2pdfopt stuff).
* Fix XML, introduce some better error messages, etc.
See roygbyte/newsdownloader.koplugin for more info
* Fix feed attribute saving snafu; Change menu labels; etc.
Also:
- Change default feed
- Change "remove" label to "delete"
- Hide menu after feed sync
* Add comments to a few functions
* Fix bug associated with arxiv catalog.
See comments in genItemTableFromCatalog. Basically, though, the bug
was related to the checking of the acquisition urls. Arxiv only has
PDFs available to download, and the block wasn't catching these with
its existing logic. By adding another clause to look for PDF link
types, and fixing href values that were missing the PDF suffix, we can
successfully download PDFs from Arxiv.
* Gestures: standardize hold to long-press
* Common settings menu: standardize hold to long-press
* Readerstyletweak: standardize hold to long-press
* Readersearch: standardize hold to long-press
* Cleanup util.secondsFrom*Clock stuff (simpler maths, tail calls, meaningful printf tokens).
* Use util.secondsToClockDuration in ReadTimer instead of reinventing the wheel three different ways.
* Reschedule unexpired timers properly on resume (as best as we can, given the unreliable nature of REALTIME).
* Make clock timers tick on the dot, instead of at the same second as when being set.
* Speaking of clock timers, leave the math to os.date & os.time, don't reinvent the wheel yet again.
Add the possibility to autostart profiles when KOReader is started with filemanager or lastfile.
Discussed in #8020
If more profiles are selected, they are started in alphabetically order.
OPDS catalog can be invoked from reader with gesture.
Then on closing OPDS catalog, KOReader crashes.
Let's jump to the download folder instead.
Closes#7784.
The hyphenation of a word can be changed from its default
by long pressing for 3 seconds and selecting 'Hyphenate'.
These overrides are stored in a per-language file, i.e:
koreader/settings/user-German.hyph.
It can happen in perfectly sane contexts.
CReDocument: Don't destroy internal engine data when Document just
decreased the refcount (as opposed to actually tore down the document
userdata if it were the last ref).
PdfDocument: Only write edited documents if the Doc instance was torn
down.
PicDocument: Silence some DocumentRegistry related warnings
Make 'em match backward & forward.
Now that we have working overrides and the gesture manager, trying to fit them in a weird superset of the top corner tapzones in a vain attempt to avoid bad interactions doesn't make much sense anymore, and just makes the Gesture Manager UI confusing.
Also make sure the corner zones override the L/R ones for double taps, like it's the case with other gestures.
Fix#7710
Add a new reader module: ReaderScrolling, that exposes
some Scrolling options to the menu (which are to be used
by and implemented in ReaderPaging and ReaderRolling
themselves) and implement some inertial scrolling logic
used by both of them.
Default to "Classic scrolling" which is the expected
behaviour on phones and tablets.
The old CreDocument buggy behaviour is available as
"Turbo scrolling" for both Paging and Rolling documents.
Added a "On release scrolling" option that might be
useful on eInk to avoid dynamic pan/scrolling.
Try to avoid bad interactions between pan and swipe,
cancelling unwanted panning if we ended up doing a
swipe or multiswipe.
* FileManager/ReaderUI: Clarify the current instance accessor
Make it clearer that we actually store it in a *module/class* member, not an *instance* member.
Also, warn if there's a close/open mismatch.
* Allow doing away with CacheItem
Now that we have working FFI finalizers on BBs, it's mostly useless overhead.
We only keep it for DocCache, because it's slightly larger, and memory pressure might put us in a do or die situation where waiting for the GC might mean an OOM kill.
* Expose's LRU slot-only mode
And use it for CatalogCache, which doesn't care about storage space
* Make GlyphCache slots only (storage space is insignificant here, it was
always going to be evicted by running out of slots).
* More informative warning when we chop the cache in half
Ought to be faster than our naive array-based approach.
Especially for the glyph cache, which has a solid amount of elements,
and is mostly cache hits.
(There are few things worse for performance in Lua than
table.remove @ !tail and table.insert @ !tail, which this was full of :/).
DocCache: New module that's now an actual Cache instance instead of a
weird hack. Replaces "Cache" (the instance) as used across Document &
co.
Only Cache instance with on-disk persistence.
ImageCache: Update to new Cache.
GlyphCache: Update to new Cache.
Also, actually free glyph bbs on eviction.
* Minor updates to the min & max cache sizes (16 & 64MB). Mostly to satisfy my power-of-two OCD.
* Purge broken on-disk cache files
* Optimize free RAM computations
* Start dropping LRU items when running low on memory before pre-rendring (hinting) pages in non-reflowable documents.
* Make serialize dump the most recently *displayed* page, as the actual MRU item is the most recently *hinted* page, not the current one.
* Use more accurate item size estimations across the whole codebase.
TileCacheItem:
* Drop lua-serialize in favor of Persist.
KoptInterface:
* Drop lua-serialize in favor of Persist.
* Make KOPTContext caching actually work by ensuring its hash is stable.
* Ensure that going from one to the other tears down the former and
its plugins before instantiating the latter and its plugins.
UIManager: Unify Event sending & broadcasting
* Make the two behave the same way (walk the widget stack from top to
bottom), and properly handle the window stack shrinking shrinking
*and* growing.
Previously, broadcasting happened bottom-to-top and didn't really
handle the list shrinking/growing, while sending only handled the list
shrinking by a single element, and hopefully that element being the one
the event was just sent to.
These two items combined allowed us to optimize suboptimal
refresh behavior with Menu and other Menu classes when
opening/closing a document.
e.g., the "opening document" Notification is now properly regional,
and the "open last doc" option no longer flashes like a crazy person
anymore.
Plugins: Allow optimizing Menu refresh with custom menus, too.
Requires moving Menu's close_callback *after* onMenuSelect, which, eh,
probably makes sense, and is probably harmless in the grand scheme of
things.
Plugins are loaded *once*, but torn-down/instantiated multiple times,
and sometimes in the reverse order.
As such, if we use a public function member as the scheduled task, we're
always pointing to the same function, and going from FM to RD
effectively *un*schedules it.
Instead, use an instance-specific closure, so that each instance
schedules & unschedules don't affect each other.
In the same vein, settings ought to be read at instantiation, not at
loading, otherwise, changing a setting in the FM, then switching to the
reader will not pick up the changes.
Kindles are not flagged canPowerOff, although that's technically not
entirely warranted, but sorta makes sense.
The Plugin already handles that sanely, and will only expose/honor the
suspend timer.
The object was never re-assigned, so closing a smaller menu (e.g.,
Calibre metadata search) made the underlying one (e.g., CoverBrowser's
ListMenu) inherit the smaller dimensions...
Instead of creating the object in the Class constructor, create it in the
instance constructor (i.e., :init).
Similar cleanups in other Menu* related classes.
* Input: Don't create a new TimeVal object for input frame timestamps, just promote our existing table by assigning it the `TimeVal` metatable.
* TimeVal: Export (const) `zero` & `huge` TimeVal objects, because they're common enough in our codebase. (NOTE: not actually const, that's a Lua 5.4 feature ;p).
* GestureDetector: Explain the behavior of the `last_tevs` & `first_tevs` tables, and why one needs a new object and not the other.
* Speaking of, simplify the copy method for `first_tevs`, because it doesn't need to create a new TimeVal object, we can just reference the original, it's unique and re-assigned for each frame.
* Wireless: Optimize memory usage in StreamMessageQueue (use an array of string ropes, that we only concatenate once). Allowed to relax the throttling, making transfers that much faster.
* Persist: Add a "zstd" codec, that uses the "luajit" codec, but compressed via zstd. Since both of those are very fast, it pretty much trounces everything in terms of speed and size ;).
* Persist: Implemented a "writes_to_file" framework, much like the existing "reads_from_file" one. And use it in the zstd codec to avoid useless temporary string interning.
* Metadata: Switch to the zstd codec.
There have been a couple of these this month, and keeping stuff that should only ever run once piling up in their respective module was getting ugly, especially when it's usually simple stuff (settings, files).
So, move everything to a dedicated module, run by reader.lua on startup, and that will actually only do things once, when necessary.
* CalibreMetadata: Get rid of the now useless NULL-hunt: here, this was basically looking for `rapidjson.null` to replace them with... `rapidjson.null` :?. IIRC, that's a remnant of a quirk of the previous JSON parser (possibly even the previous, *previous* JSON parser ^^).
* CalibreSearch: Update the actually relevant NULL-hunt to make it explicit: replace JSON NULLs with Lua nils, instead of relying on an implementation detail of Lua-RapidJSON, because that detail just changed data type ;).
* UIManager: Make sure tasks scheduled during the final ZMQ callback are honored. e.g., the Calibre "Disconnect" handler. This happened to mostly work purely by chance before the event loop rework.
* Calibre: Restore a proper receiveCallback handler after receiving a book, in order not to break the "Disconnect" handler's state (and, well, get a working Disconnect handler, period ^^).
* Calibre: Unbreak metadata cache when it's initialized by a search (regression since #7159).
* Calibre: Handle UTC <-> local time conversions when checking the cache's timestamp against the Calibre metadata timestamp.
* Bump base (Unbreak CRe on Android, update RapidJSON)
Requires https://github.com/koreader/koreader-base/pull/1344 & https://github.com/koreader/koreader-base/pull/1346 (fix#7485)
Assorted input fixes:
* Actually handle errors in the "there's a callback timer" input polling loop.
* Don't break timerfd when the clock probe was inconclusive.
Not directly related, but noticed because of duplicate onInputEvent handlers:
* HookContainer: Fix deregistration to actually deregister properly. "Regression" extant since its inception in #2933 (!).
* Made sure the three plugins (basically the trio of AutoThingies ;p) that were using HookContainer actually unschedule their task on teardown.
* Change 'Find a file' to 'File search' for consistency
There is 'File search' in the Gesture manager already.
There is 'Fulltext search' in the readermenu.
Some help text added.
Allows to hide keyboard on start to see more text.
Default is to show keyboard.
Some wordings unification.
`Open file` goes down, closer to the history of opened files.
* 'Lua check' button
* ReaderDictionary: Port delay computations to TimeVal
* ReaderHighlight: Port delay computations to TimeVal
* ReaderView: Port delay computations to TimeVal
* Android: Reset gesture detection state on APP_CMD_TERM_WINDOW.
This prevents potentially being stuck in bogus gesture states when switching apps.
* GestureDetector:
* Port delay computations to TimeVal
* Fixed delay computations to handle time warps (large and negative deltas).
* Simplified timed callback handling to invalidate timers much earlier, preventing accumulating useless timers that no longer have any chance of ever detecting a gesture.
* Fixed state clearing to handle the actual effective slots, instead of hard-coding slot 0 & slot 1.
* Simplified timed callback handling in general, and added support for a timerfd backend for better performance and accuracy.
* The improved timed callback handling allows us to detect and honor (as much as possible) the three possible clock sources usable by Linux evdev events.
The only case where synthetic timestamps are used (and that only to handle timed callbacks) is limited to non-timerfd platforms where input events use
a clock source that is *NOT* MONOTONIC.
AFAICT, that's pretty much... PocketBook, and that's it?
* Input:
* Use the <linux/input.h> FFI module instead of re-declaring every constant
* Fixed (verbose) debug logging of input events to actually translate said constants properly.
* Completely reset gesture detection state on suspend. This should prevent bogus gesture detection on resume.
* Refactored the waitEvent loop to make it easier to comprehend (hopefully) and much more efficient.
Of specific note, it no longer does a crazy select spam every 100µs, instead computing and relying on sane timeouts,
as afforded by switching the UI event/input loop to the MONOTONIC time base, and the refactored timed callbacks in GestureDetector.
* reMarkable: Stopped enforcing synthetic timestamps on input events, as it should no longer be necessary.
* TimeVal:
* Refactored and simplified, especially as far as metamethods are concerned (based on <bsd/sys/time.h>).
* Added a host of new methods to query the various POSIX clock sources, and made :now default to MONOTONIC.
* Removed the debug guard in __sub, as time going backwards can be a perfectly normal occurrence.
* New methods:
* Clock sources: :realtime, :monotonic, :monotonic_coarse, :realtime_coarse, :boottime
* Utility: :tonumber, :tousecs, :tomsecs, :fromnumber, :isPositive, :isZero
* UIManager:
* Ported event loop & scheduling to TimeVal, and switched to the MONOTONIC time base.
This ensures reliable and consistent scheduling, as time is ensured never to go backwards.
* Added a :getTime() method, that returns a cached TimeVal:now(), updated at the top of every UI frame.
It's used throughout the codebase to cadge a syscall in circumstances where we are guaranteed that a syscall would return a mostly identical value,
because very few time has passed.
The only code left that does live syscalls does it because it's actually necessary for accuracy,
and the only code left that does that in a REALTIME time base is code that *actually* deals with calendar time (e.g., Statistics).
* DictQuickLookup: Port delay computations to TimeVal
* FootNoteWidget: Port delay computations to TimeVal
* HTMLBoxWidget: Port delay computations to TimeVal
* Notification: Port delay computations to TimeVal
* TextBoxWidget: Port delay computations to TimeVal
* AutoSuspend: Port to TimeVal
* AutoTurn:
* Fix it so that settings are actually honored.
* Port to TimeVal
* BackgroundRunner: Port to TimeVal
* Calibre: Port benchmarking code to TimeVal
* BookInfoManager: Removed unnecessary yield in the metadata extraction subprocess now that subprocesses get scheduled properly.
* All in all, these changes reduced the CPU cost of a single tap by a factor of ten (!), and got rid of an insane amount of weird poll/wakeup cycles that must have been hell on CPU schedulers and batteries..
* Add a new socketutil module with a few helper functions that allow us to:
* Always use a sane User-Agent (previously, only Wikipedia did so)
* Set timeouts in an almost sane manner. Doing it explicitly prevents an interaction with KOSync that does crazy stuff I don't even want to try to understand.
* Unified said timeouts based on the request's intended usage (except for Wikipedia, which already had meaningful timeout values).
* Stopped using LuaSec directly, LuaSocket defers to LuaSec sanely on its own. Everything now transparently supports HTTPS without code duplication.
* Unify logging with AutoSuspend (e.g., keep ourselves to showing the delay in seconds, not the raw timestamp, as that's way harder to interpret, and the RTC module and/or logger will do that for us when the time comes).
* Speaking of, minor revamp of RTC related logging to make it more human-readable.
* On Kobo, if we hit the unexpected wakeup limit, re-engage AutoSuspend's *suspend* check, so that the device has a chance to poweroff instead of being kept awake.
* Screensaver: Unbreak screensaver_stretch_images
We don't have real ternary operators in Lua, if the second argument evaluates to false, it doesn't work.
Invert the test to avoid this pitfall.
(c.f., http://lua-users.org/wiki/TernaryOperator).
Fix#7402, regression since #7371
* Free a few similar constructs (incidentally, some of 'em also tweaked in #7371 ^^).
* LuaSettings/DocSettings: Updated readSetting API to allow proper initialization to default.
Use it to initialize tables, e.g., fixing corner-cases in readerFooter that could prevent settings from being saved.
(Fixes an issue reported on Gitter).
* LuaSettings/DocSettings: Add simpler API than the the flip* ones to toggle boolean settings.
* Update LuaSettings/DocSettigns usage throughout the codebase to use the dedicated boolean methods wher appropriate, and clean up some of the more mind-bending uses.
* FileChooser: Implement an extended default exclusion list (fix#2360)
* ScreenSaver: Refactor to avoid the pile of kludges this was threatening to become. Code should be easier to follow and use, and fallbacks now behave as expected (fix#4418).
