diff --git a/docs/source/data.rst b/docs/source/data.rst index 0365505c..20cb8a83 100644 --- a/docs/source/data.rst +++ b/docs/source/data.rst @@ -4,3 +4,26 @@ Extracting Data **Note**: This feature is still in **BETA**, you should expect bugs and incompatible changes in the future. + +Log messages contain a good deal of useful data, but it's not always easy to +get at. The log parser built into **lnav** is able to extract data as +described by log formats as well as discovering data in plain text messages. +This data can then be queried and processed using the SQLite front-end that is +also incorporated into **lnav**. As an example, the following Syslog message +from :cmd:`sudo` will be parsed and several + + Jul 31 11:42:26 Example-MacBook-Pro.local sudo[87024]: testuser : TTY=ttys004 ; PWD=/Users/testuser/github/lbuild ; USER=root ; COMMAND=/usr/bin/make install + + + + Current Time: 2013-07-31T11:42:26.000 Original Time: 2013-07-31T11:42:26.000 Offset: +0.000 + Known message fields: + ├ log_hostname = Example-MacBook-Pro.local + ├ log_procname = sudo + ├ log_pid = 87024 + Discovered message fields: + ├ col_0 = testuser + ├ TTY = ttys004 + ├ PWD = /Users/testuser/github/lbuild + ├ USER = root + └ COMMAND = /usr/bin/make install diff --git a/docs/source/formats.rst b/docs/source/formats.rst index 5108837c..c898764a 100644 --- a/docs/source/formats.rst +++ b/docs/source/formats.rst @@ -30,10 +30,10 @@ can consult that file when writing your own formats or if you need to modify existing ones. The contents of the format configuration should be a JSON object with a field -for each format defined by the file. - -The symbolic name of the format. This value will also be - used as the SQL table name for the log. +for each format defined by the file. Each field name should be the symbolic +name of the format. This value will also be used as the SQL table name for +the log. The value for each field should be another object with the following +fields: :title: The short and human-readable name for the format. :description: A longer description of the format. @@ -51,10 +51,22 @@ The symbolic name of the format. This value will also be in order of severity, are: **fatal**, **critical**, **error**, **warning**, **info**, **debug**, **trace**. - :value: + :value: This object contains the definitions for the values captured by the + regexes. - :kind: **string**, **integer**, **float**. - :collate: + :kind: The type of data that was captured **string**, **integer**, + **float**. + :collate: The collation function for this value. + :identifier: A boolean that indicates whether or not this field represents + an identifier and should be syntax colored. + :foreign-key: A boolean that indicates that this field is a key and should + not be graphed. This should only need to be set for integer fields. + + :sample: A list of objects that contain sample log messages. All formats + must include at least one sample and it must be matched by one of the + included regexes. Each object must contain the following field: + + :line: The sample message. Example format:: @@ -89,3 +101,5 @@ Example format:: Modifying an Existing Format ---------------------------- + + diff --git a/docs/source/index.rst b/docs/source/index.rst index f76832c4..01618075 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -22,6 +22,7 @@ Contents: config hotkeys commands + sqlext Indices and tables ================== diff --git a/docs/source/sqlext.rst b/docs/source/sqlext.rst new file mode 100644 index 00000000..cec00fc4 --- /dev/null +++ b/docs/source/sqlext.rst @@ -0,0 +1,138 @@ + +SQLite Extensions Reference +=========================== + +To make it easier to analyze log data from within **lnav**, there are several +built-in extensions that provide extra functions and collators beyond those +`provided by sqlite `_. The majority +of the functions are from the +`extensions-functions.c `_ file available from +the `sqlite.org `_ web site. + +*Tip*: You can include a SQLite database file on the command-line and use +**lnav**'s interface to perform queries. The database will be attached with +a name based on the database file name. + +Math +---- + +Basic mathematical functions: + +* cos(n) +* sin(n) +* tan(n) +* cot(n) +* cosh(n) +* sinh(n) +* coth(n) +* acos(n) +* asin(n) +* atan(r1,r2) +* atan2(r1,r2) +* exp(n) +* log(n) +* log10(n) +* power(x,y) +* sign(n) - Return one of 3 possibilities +1,0 or -1 when the argument is + respectively positive, 0 or negative. +* sqrt(n) +* square(n) +* ceil(n) +* floor(n) +* pi() + +* degrees - Convert radians to degrees +* radians - Convert degrees to radians + +Aggregate functions: + +* stddev +* variance +* mode +* median +* lower_quartile +* upper_quartile + +String +------ + +Additional string comparison and manipulation functions: + +* difference(s1,s2) - Computes the number of different characters between the + soundex value fo 2 strings. +* replicate(s,n) - Given a string (s) in the first argument and an integer (n) + in the second returns the string that constains s contatenated n times. +* proper(s) - Ensures that the words in the given string have their first + letter capitalized and the following letters are lower case. +* charindex(s1,s2), charindex(s1,s2,n) - Given 2 input strings (s1,s2) and an + integer (n) searches from the nth character for the string s1. Returns the + position where the match occured. Characters are counted from 1. 0 is + returned when no match occurs. +* leftstr(s,n) - Given a string (s) and an integer (n) returns the n leftmost + (UTF-8) characters if the string has a length<=n or is NULL this function is + NOP. +* rightstr(s,n) - Given a string (s) and an integer (n) returns the n rightmost + (UTF-8) characters if the string has a length<=n or is NULL this function is + NOP +* reverse(s) - Given a string returns the same string but with the characters + in reverse order. +* padl(s,n) - Given an input string (s) and an integer (n) adds spaces at the + beginning of (s) until it has a length of n characters. When s has a length + >=n it's a NOP. padl(NULL) = NULL +* padr(s,n) - Given an input string (s) and an integer (n) appends spaces at + the end of s until it has a length of n characters. When s has a length >=n + it's a NOP. padr(NULL) = NULL +* padc(s,n) - Given an input string (s) and an integer (n) appends spaces at + the end of s and adds spaces at the begining of s until it has a length of n + characters. Tries to add has many characters at the left as at the right. + When s has a length >=n it's a NOP. padc(NULL) = NULL +* strfilter(s1,s2) - Given 2 string (s1,s2) returns the string s1 with the + characters NOT in s2 removed assumes strings are UTF-8 encoded. +* regexp(re,s) - Return 1 if the regular expression 're' matches the given + string. +* regexp_replace(str, re, repl) - Replace the portions of the given string + that match the regular expression with the replacement string. **NOTE**: + The arguments for the string and the regular expression in this function are + reversed from the plain regexp() function. This is to be somewhat compatible + with functions in other database implementations. +* startswith(s1,prefix) - Given a string and prefix, return 1 if the string + starts with the given prefix. +* endswith(s1,suffix) - Given a string and suffix, return 1 if the string ends + with the given suffix. + +File Paths +---------- + +File path manipulation functions: + +* basename(s) - Return the file name part of a path. +* dirname(s) - Return the directory part of a path. +* joinpath(s1,s2,...) - Return the arguments joined together into a path. + +Networking +---------- + +Network information functions: + +* gethostbyname - Convert a host name into an IP address. The host name could + not be resolved, the input value will be returned. +* gethostbyaddr - Convert an IPv4/IPv6 address into a host name. If the + reverse lookup fails, the input value will be returned. + +Internal State +-------------- + +The following functions can be used to access **lnav**'s internal state: + +* log_top_line() - Return the line number at the top of the log view. +* log_top_datetime() - Return the timestamp of the line at the top of the log + view. + +Collators +--------- + +* naturalcase - Compare strings "naturally" so that number values in the string + are compared based on their numeric value and not their character values. + For example, "foo10" would be considered greater than "foo2". +* naturalnocase - The same as naturalcase, but case-insensitive. +* ipaddress - Compare IPv4/IPv6 addresses. diff --git a/src/fs-extension-functions.cc b/src/fs-extension-functions.cc index 1ba7ad5e..62f2b969 100644 --- a/src/fs-extension-functions.cc +++ b/src/fs-extension-functions.cc @@ -165,6 +165,10 @@ int fs_extension_functions(const struct FuncDef **basic_funcs, { "dirname", 1, 0, SQLITE_UTF8, 0, sql_dirname }, { "joinpath", -1, 0, SQLITE_UTF8, 0, sql_joinpath }, + /* + * TODO: add other functions like readlink, normpath, ... + */ + { NULL } };