Archives
/
lnav
mirror of https://github.com/tstack/lnav
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[docs] some more doc text

Browse Source
pull/857/head
Timothy Stack 3 years ago
parent 0f238f7972
commit f5c72736cb
12 changed files with 191 additions and 83 deletions
Show Stats Download Patch File Download Diff File

41
ARCHITECTURE.md
Unescape Escape View File

@ -71,6 +71,14 @@ As files are being indexed, if a matching format is found, the file is
it is added to the [logfile_sub_source](src/logfile_sub_source.hh), which
collates all log messages together into a single index.
### Timestamp Parsing
Since all log messages need to have a timestamp, timestamp parsing needs to be
very efficient. The standard `strptime()` function is quite expensive, so lnav
includes an optimized custom parser and code-generator in the
[ptimec](src/ptimec.hh) component. The code-generator is used at compile-time
to generate parsers for several [common formats](src/time_formats.am).
## Log Formats
[log_format](src/log_format.hh) instances are used to parse lines from files
@ -83,9 +91,32 @@ implemented in the [log_format_impls.cc](src/log_format_impls.cc) file.
## User Interface
[![lnav TUI](docs/lnav-tui.png)](https://whimsical.com/lnav-tui-MQjXc7Vx23BxQTHrnuNp5F)
The lnav text-user-interface is built on top of the basic drawing functionality
provided by [ncurses](https://invisible-island.net/ncurses/announce.html).
The lnav text-user-interface is built on top of
[ncurses](https://invisible-island.net/ncurses/announce.html).
However, the higher-level functionality of panels, widgets, and such is not
used.
used. Instead, the following custom components are built on top of the ncurses
primitives:
* [view_curses](src/view_curses.hh) - Provides the basics for text roles, which
allows for themes to color and style text. The `mvwattrline()` function does
all the heavy lifting of drawing ["attributed" lines](src/attr_line.hh),
which are strings that have attributes associated with a given range of
characters.
* [listview_curses](src/listview_curses.hh) - Displays a list of items that are
provided by a source.
* [textview_curses](src/textview_curses.hh) - Builds on the list view by adding
support for searching, filtering, bookmarks, etc... The main panel that
displays the logs/plaintext/help is a textview.
* [statusview_curses](src/state-extension-functions.cc) - Draws the status bars
at the top and bottom of the TUI.
* [vt52_curses](src/vt52_curses.hh) - Adapts vt52 escape codes to the ncurses
API.
* [readline_curses](src/readline_curses.hh) - Provides access to the readline
library. The readline code is executed in a child process since readline
does not get along with ncurses. The child process and readline is set to
use a vt52 terminal and the vt52_curses view is uses to translate those
escape codes to ncurses.
The following diagram shows the underlying components that make up the TUI:
[![lnav TUI](docs/lnav-tui.png)](https://whimsical.com/lnav-tui-MQjXc7Vx23BxQTHrnuNp5F)

22
docs/source/formats.rst
Unescape Escape View File

@ -47,11 +47,21 @@ own formats or if you need to modify existing ones. Format directories can
also contain '.sql' and '.lnav' script files that can be used automate log file
analysis.
The contents of the format configuration should be a JSON object with a field
for each format defined by the file. Each field name should be the symbolic
name of the format. This value will also be used as the SQL table name for
the log. The value for each field should be another object with the following
fields:
An **lnav** format file must contain a single JSON object, preferably with a
:code:`$schema` property that refers to the
`config-v1.schema <https://lnav.org/schemas/config-v1.schema.json>`_,
like so:
.. code-block:: json
{
"$schema": "https://lnav.org/schemas/config-v1.schema.json"
}
Each format to be defined in the file should a separate field in the top-level
object. The field name should be the symbolic name of the format. This value
will also be used as the SQL table name for the log. The value for each field
should be another object with the following fields:
:title: The short and human-readable name for the format.
:description: A longer description of the format.
@ -253,6 +263,7 @@ Example format:
.. code-block:: json
{
"$schema": "https://lnav.org/schemas/format-v1.schema.json",
"example_log" : {
"title" : "Example Log Format",
"description" : "Log format used in the documentation example.",
@ -293,6 +304,7 @@ with the following contents:
.. code-block:: json
{
"$schema": "https://lnav.org/schemas/format-v1.schema.json",
"example_log" : {
"regex" : {
"custom1" : {

53
src/Makefile.am
Unescape Escape View File

@ -61,58 +61,7 @@ builtin-sh-scripts.h builtin-sh-scripts.cc: bin2c$(BUILD_EXEEXT) $(BUILTIN_SHSCR
%-json.cc %-json.h: $(srcdir)/%.json bin2c$(BUILD_EXEEXT)
$(BIN2C_V)./bin2c$(BUILD_EXEEXT) $(*)-json $<
TIME_FORMATS = \
"@%@" \
"%Y-%m-%d %H:%M:%S" \
"%Y-%m-%d %H:%M:%S%z" \
"%Y-%m-%d %H:%M:%S %z" \
"%Y-%m-%d %H:%M" \
"%Y-%m-%dT%H:%M:%S.%f%z" \
"%y-%m-%dT%H:%M:%S.%f%z" \
"%Y-%m-%dT%H:%M:%SZ" \
"%Y-%m-%dT%H:%M:%S%z" \
"%Y-%m-%dT%H:%M:%S" \
"%Y-%m-%dT%H:%M:%S%z" \
"%Y/%m/%d %H:%M:%S" \
"%Y/%m/%d %H:%M:%S %z" \
"%Y/%m/%d %H:%M:%S%z" \
"%Y/%m/%d %H:%M" \
"%Y %b %d %a %H:%M:%S.%L" \
"%Y %b %d %H:%M:%S.%L" \
"%Y %b %d %H:%M:%S" \
"%a %b %d %H:%M:%S %Y" \
"%a %b %d %H:%M:%S.%f %Y" \
"%a %b %d %H:%M:%S %Z %Y" \
"%a %b %d %H:%M:%S " \
"%a %b %d %H:%M:%S.%L " \
"%d/%b/%Y:%H:%M:%S +0000" \
"%d/%b/%Y:%H:%M:%S %z" \
"%d-%b-%Y %H:%M:%S %z" \
"%d-%b-%Y %H:%M:%S %Z" \
"%d %b %Y %H:%M:%S" \
"%d %b %Y %H:%M:%S.%L" \
"%d %b %Y %H:%M:%S,%L" \
"%b %d %H:%M:%S" \
"%b %d %k:%M:%S" \
"%b %d %l:%M:%S" \
"%b %e, %Y %l:%M:%S %p" \
"%m/%d/%y %H:%M:%S" \
"%m/%d/%Y %I:%M:%S:%L %p %Z" \
"%m/%d/%Y %I:%M:%S %p %Z" \
"%m/%d/%Y %l:%M:%S %p %Z" \
"%m/%e/%Y %I:%M:%S %p" \
"%m/%e/%Y %l:%M:%S %p" \
"%d/%b/%y %H:%M:%S" \
"%m%d %H:%M:%S" \
"%H:%M:%S" \
"%M:%S" \
"%m/%d %H:%M:%S" \
"%Y-%m-%d" \
"%Y-%m" \
"%Y/%m/%d" \
"%Y/%m" \
"%s.%f" \
$()
include time_formats.am
time_fmts.cc: ptimec$(BUILD_EXEEXT)
$(PTIME_V)./ptimec$(BUILD_EXEEXT) $(TIME_FORMATS) > $@

39
src/base/future_util.hh
Unescape Escape View File

@ -33,6 +33,16 @@
#include <deque>
#include <future>
namespace lnav {
namespace futures {
/**
* Create a future that is ready to immediately return a result.
*
* @tparam T The result type of the future.
* @param t The value the future should return.
* @return The new future.
*/
template<class T>
std::future<std::decay_t<T>> make_ready_future( T&& t ) {
std::promise<std::decay_t<T>> pr;
@ -41,9 +51,18 @@ std::future<std::decay_t<T>> make_ready_future( T&& t ) {
return r;
}
template<typename T>
/**
* A queue used to limit the number of futures that are running concurrently.
*
* @tparam T The result of the futures.
* @tparam MAX_QUEUE_SIZE The maximum number of futures that can be in flight.
*/
template<typename T, int MAX_QUEUE_SIZE = 8>
class future_queue {
public:
/**
* @param processor The function to execute with the result of a future.
*/
explicit future_queue(std::function<void(const T&)> processor)
: fq_processor(processor) {};
@ -51,11 +70,24 @@ public:
this->pop_to();
}
/**
* Add a future to the queue. If the size of the queue is greater than the
* MAX_QUEUE_SIZE, this call will block waiting for the first queued
* future to return a result.
*
* @param f The future to add to the queue.
*/
void push_back(std::future<T>&& f) {
this->fq_deque.emplace_back(std::move(f));
this->pop_to(8);
this->pop_to(MAX_QUEUE_SIZE);
}
/**
* Removes the next future from the queue, waits for the result, and then
* repeats until the queue reaches the given size.
*
* @param size The new desired size of the queue.
*/
void pop_to(size_t size = 0) {
while (this->fq_deque.size() > size) {
this->fq_processor(this->fq_deque.front().get());
@ -67,4 +99,7 @@ public:
std::deque<std::future<T>> fq_deque;
};
}
}
#endif

2
src/command_executor.cc
Unescape Escape View File

@ -816,7 +816,7 @@ future<string> pipe_callback(exec_context &ec, const string &cmdline, auto_fd &f
HELP_MSG_1(X, "to close the file"));
}
return make_ready_future(std::string());
return lnav::futures::make_ready_future(std::string());
}
}

18
src/file_collection.cc
Unescape Escape View File

@ -160,7 +160,7 @@ file_collection::watch_logfile(const std::string &filename,
int rc;
if (this->fc_closed_files.count(filename)) {
return make_ready_future(retval);
return lnav::futures::make_ready_future(retval);
}
if (loo.loo_fd != -1) {
@ -177,14 +177,14 @@ file_collection::watch_logfile(const std::string &filename,
this->fc_file_names.end()) {
retval.fc_file_names.emplace(wilddir, logfile_open_options());
}
return make_ready_future(retval);
return lnav::futures::make_ready_future(retval);
}
if (!S_ISREG(st.st_mode)) {
if (required) {
rc = -1;
errno = EINVAL;
} else {
return make_ready_future(retval);
return lnav::futures::make_ready_future(retval);
}
}
}
@ -192,7 +192,7 @@ file_collection::watch_logfile(const std::string &filename,
if (required) {
retval.fc_name_to_errors[filename] = strerror(errno);
}
return make_ready_future(retval);
return lnav::futures::make_ready_future(retval);
}
auto stat_iter = find_if(this->fc_new_stats.begin(),
@ -204,7 +204,7 @@ file_collection::watch_logfile(const std::string &filename,
if (stat_iter != this->fc_new_stats.end()) {
// this file is probably a link that we have already scanned in this
// pass.
return make_ready_future(retval);
return lnav::futures::make_ready_future(retval);
}
this->fc_new_stats.emplace_back(st);
@ -214,7 +214,7 @@ file_collection::watch_logfile(const std::string &filename,
if (file_iter == this->fc_files.end()) {
if (this->fc_other_files.find(filename) != this->fc_other_files.end()) {
return make_ready_future(retval);
return lnav::futures::make_ready_future(retval);
}
auto func = [filename, loo, prog = this->fc_progress, errs = this->fc_name_to_errors]() mutable {
@ -329,7 +329,7 @@ file_collection::watch_logfile(const std::string &filename,
}
}
return make_ready_future(retval);
return lnav::futures::make_ready_future(retval);
}
/**
@ -338,7 +338,7 @@ file_collection::watch_logfile(const std::string &filename,
* @param path The glob pattern to expand.
* @param required Passed to watch_logfile.
*/
void file_collection::expand_filename(future_queue<file_collection> &fq,
void file_collection::expand_filename(lnav::futures::future_queue<file_collection> &fq,
const std::string &path,
logfile_open_options &loo,
bool required)
@ -404,7 +404,7 @@ void file_collection::expand_filename(future_queue<file_collection> &fq,
file_collection file_collection::rescan_files(bool required)
{
file_collection retval;
future_queue<file_collection> fq([&retval](auto &fc) {
lnav::futures::future_queue<file_collection> fq([&retval](auto &fc) {
retval.merge(fc);
});

6
src/file_collection.hh
Unescape Escape View File

@ -83,8 +83,10 @@ struct file_collection {
file_collection rescan_files(bool required = false);
void
expand_filename(future_queue<file_collection> &fq, const std::string &path,
logfile_open_options &loo, bool required);
expand_filename(lnav::futures::future_queue<file_collection> &fq,
const std::string &path,
logfile_open_options &loo,
bool required);
std::future<file_collection>
watch_logfile(const std::string &filename, logfile_open_options &loo,

15
src/internals/config-v1.schema.json
Unescape Escape View File

@ -26,7 +26,11 @@
"cache-ttl": {
"title": "/tuning/archive-manager/cache-ttl",
"description": "The time-to-live for unpacked archives, expressed as a duration (e.g. '3d' for three days)",
"type": "string"
"type": "string",
"examples": [
"3d",
"12h"
]
}
},
"additionalProperties": false
@ -357,15 +361,18 @@
"type": "object",
"patternProperties": {
"((?:x[0-9a-f]{2})+)": {
"description": "The hexadecimal encoding of key codes to map to a command.",
"description": "Map of key codes to commands to execute. The field names are the keys to be mapped using as a hexadecimal representation of the UTF-8 encoding. Each byte of the UTF-8 should start with an 'x' followed by the hexadecimal representation of the byte.",
"title": "/ui/keymap-defs/<keymap_name>/<key_seq>",
"type": "object",
"properties": {
"command": {
"title": "/ui/keymap-defs/<keymap_name>/<key_seq>/command",
"description": "The command to execute for the given key sequence.",
"description": "The command to execute for the given key sequence. Use a script to execute more complicated operations.",
"type": "string",
"pattern": "[:|;].*"
"pattern": "[:|;].*",
"examples": [
":goto next hour"
]
},
"alt-msg": {
"title": "/ui/keymap-defs/<keymap_name>/<key_seq>/alt-msg",

16
src/lnav_config.cc
Unescape Escape View File

@ -403,8 +403,11 @@ static void config_error_reporter(const yajlpp_parse_context &ypc,
static struct json_path_container key_command_handlers = {
yajlpp::property_handler("command")
.with_synopsis("<command>")
.with_description("The command to execute for the given key sequence.")
.with_description(
"The command to execute for the given key sequence. Use a script "
"to execute more complicated operations.")
.with_pattern("[:|;].*")
.with_example(":goto next hour")
.FOR_FIELD(key_command, kc_cmd),
yajlpp::property_handler("alt-msg")
.with_synopsis("<msg>")
@ -414,7 +417,12 @@ static struct json_path_container key_command_handlers = {
static struct json_path_container keymap_def_handlers = {
yajlpp::pattern_property_handler("(?<key_seq>(?:x[0-9a-f]{2})+)")
.with_description("The hexadecimal encoding of key codes to map to a command.")
.with_synopsis("<utf8-key-code-in-hex>")
.with_description(
"Map of key codes to commands to execute. The field names are "
"the keys to be mapped using as a hexadecimal representation of "
"the UTF-8 encoding. Each byte of the UTF-8 should start with "
"an 'x' followed by the hexadecimal representation of the byte.")
.with_obj_provider<key_command, key_map>([](const yajlpp_provider_context &ypc, key_map *km) {
key_command &retval = km->km_seq_to_cmd[ypc.ypc_extractor.get_substr("key_seq")];
@ -863,6 +871,7 @@ static struct json_path_container ui_handlers = {
static struct json_path_container archive_handlers = {
yajlpp::property_handler("min-free-space")
.with_synopsis("<bytes>")
.with_description(
"The minimum free space, in bytes, to maintain when unpacking "
"archives")
@ -870,9 +879,12 @@ static struct json_path_container archive_handlers = {
.for_field(&_lnav_config::lc_archive_manager,
&archive_manager::config::amc_min_free_space),
yajlpp::property_handler("cache-ttl")
.with_synopsis("<duration>")
.with_description(
"The time-to-live for unpacked archives, expressed as a duration "
"(e.g. '3d' for three days)")
.with_example("3d")
.with_example("12h")
.for_field(&_lnav_config::lc_archive_manager,
&archive_manager::config::amc_cache_ttl),
};

2
src/textview_curses.hh
Unescape Escape View File

@ -770,7 +770,7 @@ public:
int start,
int end);
bool is_searching() { return this->tc_searching > 0; };
bool is_searching() const { return this->tc_searching > 0; };
void set_follow_search_for(int64_t ms_to_deadline,
std::function<bool()> func) {

53
src/time_formats.am
Unescape Escape View File

@ -0,0 +1,53 @@
TIME_FORMATS = \
"@%@" \
"%Y-%m-%d %H:%M:%S" \
"%Y-%m-%d %H:%M:%S%z" \
"%Y-%m-%d %H:%M:%S %z" \
"%Y-%m-%d %H:%M" \
"%Y-%m-%dT%H:%M:%S.%f%z" \
"%y-%m-%dT%H:%M:%S.%f%z" \
"%Y-%m-%dT%H:%M:%SZ" \
"%Y-%m-%dT%H:%M:%S%z" \
"%Y-%m-%dT%H:%M:%S" \
"%Y-%m-%dT%H:%M:%S%z" \
"%Y/%m/%d %H:%M:%S" \
"%Y/%m/%d %H:%M:%S %z" \
"%Y/%m/%d %H:%M:%S%z" \
"%Y/%m/%d %H:%M" \
"%Y %b %d %a %H:%M:%S.%L" \
"%Y %b %d %H:%M:%S.%L" \
"%Y %b %d %H:%M:%S" \
"%a %b %d %H:%M:%S %Y" \
"%a %b %d %H:%M:%S.%f %Y" \
"%a %b %d %H:%M:%S %Z %Y" \
"%a %b %d %H:%M:%S " \
"%a %b %d %H:%M:%S.%L " \
"%d/%b/%Y:%H:%M:%S +0000" \
"%d/%b/%Y:%H:%M:%S %z" \
"%d-%b-%Y %H:%M:%S %z" \
"%d-%b-%Y %H:%M:%S %Z" \
"%d %b %Y %H:%M:%S" \
"%d %b %Y %H:%M:%S.%L" \
"%d %b %Y %H:%M:%S,%L" \
"%b %d %H:%M:%S" \
"%b %d %k:%M:%S" \
"%b %d %l:%M:%S" \
"%b %e, %Y %l:%M:%S %p" \
"%m/%d/%y %H:%M:%S" \
"%m/%d/%Y %I:%M:%S:%L %p %Z" \
"%m/%d/%Y %I:%M:%S %p %Z" \
"%m/%d/%Y %l:%M:%S %p %Z" \
"%m/%e/%Y %I:%M:%S %p" \
"%m/%e/%Y %l:%M:%S %p" \
"%d/%b/%y %H:%M:%S" \
"%m%d %H:%M:%S" \
"%H:%M:%S" \
"%M:%S" \
"%m/%d %H:%M:%S" \
"%Y-%m-%d" \
"%Y-%m" \
"%Y/%m/%d" \
"%Y/%m" \
"%s.%f" \
$()

7
src/unique_path.hh
Unescape Escape View File

@ -39,6 +39,9 @@
#include "ghc/filesystem.hpp"
/**
* A source of a path for the unique_path_generator.
*/
class unique_path_source {
public:
virtual ~unique_path_source() = default;
@ -65,6 +68,10 @@ private:
std::string ups_unique_path;
};
/**
* Given a collection of filesystem paths, this class will generate a shortened
* and unique path for each of the given paths.
*/
class unique_path_generator {
public:
void add_source(const std::shared_ptr<unique_path_source>& path_source);

Write Preview
Loading…
Cancel
Save