[docs] more sections

docs/source/data.rst

@@ -4,3 +4,26 @@ Extracting Data
 
 **Note**: This feature is still in **BETA**, you should expect bugs and
 incompatible changes in the future.
+
+Log messages contain a good deal of useful data, but it's not always easy to
+get at.  The log parser built into **lnav** is able to extract data as
+described by log formats as well as discovering data in plain text messages.
+This data can then be queried and processed using the SQLite front-end that is
+also incorporated into **lnav**.  As an example, the following Syslog message
+from :cmd:`sudo` will be parsed and several 
+
+    Jul 31 11:42:26 Example-MacBook-Pro.local sudo[87024]:  testuser : TTY=ttys004 ; PWD=/Users/testuser/github/lbuild ; USER=root ; COMMAND=/usr/bin/make install
+
+
+
+    Current Time: 2013-07-31T11:42:26.000  Original Time: 2013-07-31T11:42:26.000  Offset: +0.000
+    Known message fields:
+    ├ log_hostname = Example-MacBook-Pro.local
+    ├ log_procname = sudo
+    ├ log_pid      = 87024
+    Discovered message fields:
+    ├ col_0        = testuser
+    ├ TTY          = ttys004
+    ├ PWD          = /Users/testuser/github/lbuild
+    ├ USER         = root
+    └ COMMAND      = /usr/bin/make install

docs/source/formats.rst

@@ -30,10 +30,10 @@ can consult that file when writing your own formats or if you need to modify
 existing ones.
 
 The contents of the format configuration should be a JSON object with a field
-for each format defined by the file.
-
-The symbolic name of the format.  This value will also be
-  used as the SQL table name for the log.
+for each format defined by the file.  Each field name should be the symbolic
+name of the format.  This value will also be used as the SQL table name for
+the log.  The value for each field should be another object with the following
+fields:
 
   :title: The short and human-readable name for the format.
   :description: A longer description of the format.
@@ -51,10 +51,22 @@ The symbolic name of the format.  This value will also be
     in order of severity, are: **fatal**, **critical**, **error**,
     **warning**, **info**, **debug**, **trace**.
 
-  :value:
+  :value: This object contains the definitions for the values captured by the
+    regexes.
 
-    :kind:  **string**, **integer**, **float**.
-    :collate:
+    :kind: The type of data that was captured **string**, **integer**,
+      **float**.
+    :collate: The collation function for this value.
+    :identifier: A boolean that indicates whether or not this field represents
+      an identifier and should be syntax colored.
+    :foreign-key: A boolean that indicates that this field is a key and should
+      not be graphed.  This should only need to be set for integer fields.
+
+  :sample: A list of objects that contain sample log messages.  All formats
+    must include at least one sample and it must be matched by one of the
+    included regexes.  Each object must contain the following field:
+
+    :line: The sample message.
 
 Example format::
 
@@ -89,3 +101,5 @@ Example format::
 
 Modifying an Existing Format
 ----------------------------
+
+

docs/source/index.rst

@@ -22,6 +22,7 @@ Contents:
    config
    hotkeys
    commands
+   sqlext
 
 Indices and tables
 ==================

docs/source/sqlext.rst

@@ -0,0 +1,138 @@
+
+SQLite Extensions Reference
+===========================
+
+To make it easier to analyze log data from within **lnav**, there are several
+built-in extensions that provide extra functions and collators beyond those
+`provided by sqlite <http://www.sqlite.org/lang_corefunc.html>`_.  The majority
+of the functions are from the
+`extensions-functions.c <http://www.sqlite.org/contrib>`_ file available from
+the `sqlite.org <http://sqlite.org>`_ web site.
+
+*Tip*: You can include a SQLite database file on the command-line and use
+**lnav**'s interface to perform queries.  The database will be attached with
+a name based on the database file name.
+
+Math
+----
+
+Basic mathematical functions:
+
+* cos(n)
+* sin(n)
+* tan(n)
+* cot(n)
+* cosh(n)
+* sinh(n)
+* coth(n)
+* acos(n)
+* asin(n)
+* atan(r1,r2)
+* atan2(r1,r2)
+* exp(n)
+* log(n)
+* log10(n)
+* power(x,y)
+* sign(n) - Return one of 3 possibilities +1,0 or -1 when the argument is
+  respectively positive, 0 or negative.
+* sqrt(n)
+* square(n)
+* ceil(n)
+* floor(n)
+* pi()
+
+* degrees - Convert radians to degrees
+* radians - Convert degrees to radians
+
+Aggregate functions:
+
+* stddev
+* variance
+* mode
+* median
+* lower_quartile
+* upper_quartile
+
+String
+------
+
+Additional string comparison and manipulation functions:
+
+* difference(s1,s2) - Computes the number of different characters between the
+  soundex value fo 2 strings.
+* replicate(s,n) - Given a string (s) in the first argument and an integer (n)
+  in the second returns the string that constains s contatenated n times.
+* proper(s) - Ensures that the words in the given string have their first
+  letter capitalized and the following letters are lower case.
+* charindex(s1,s2), charindex(s1,s2,n) - Given 2 input strings (s1,s2) and an
+  integer (n) searches from the nth character for the string s1. Returns the
+  position where the match occured. Characters are counted from 1. 0 is
+  returned when no match occurs.
+* leftstr(s,n) - Given a string (s) and an integer (n) returns the n leftmost
+  (UTF-8) characters if the string has a length<=n or is NULL this function is
+  NOP.
+* rightstr(s,n) - Given a string (s) and an integer (n) returns the n rightmost
+  (UTF-8) characters if the string has a length<=n or is NULL this function is
+  NOP
+* reverse(s) - Given a string returns the same string but with the characters
+  in reverse order.
+* padl(s,n) - Given an input string (s) and an integer (n) adds spaces at the
+  beginning of (s) until it has a length of n characters.  When s has a length
+  >=n it's a NOP. padl(NULL) = NULL
+* padr(s,n) - Given an input string (s) and an integer (n) appends spaces at
+  the end of s until it has a length of n characters. When s has a length >=n
+  it's a NOP. padr(NULL) = NULL
+* padc(s,n) - Given an input string (s) and an integer (n) appends spaces at
+  the end of s and adds spaces at the begining of s until it has a length of n
+  characters.  Tries to add has many characters at the left as at the right.
+  When s has a length >=n it's a NOP. padc(NULL) = NULL
+* strfilter(s1,s2) - Given 2 string (s1,s2) returns the string s1 with the
+  characters NOT in s2 removed assumes strings are UTF-8 encoded.
+* regexp(re,s) - Return 1 if the regular expression 're' matches the given
+  string.
+* regexp_replace(str, re, repl) - Replace the portions of the given string
+  that match the regular expression with the replacement string.  **NOTE**:
+  The arguments for the string and the regular expression in this function are
+  reversed from the plain regexp() function.  This is to be somewhat compatible
+  with functions in other database implementations.
+* startswith(s1,prefix) - Given a string and prefix, return 1 if the string
+  starts with the given prefix.
+* endswith(s1,suffix) - Given a string and suffix, return 1 if the string ends
+  with the given suffix.
+
+File Paths
+----------
+
+File path manipulation functions:
+
+* basename(s) - Return the file name part of a path.
+* dirname(s) - Return the directory part of a path.
+* joinpath(s1,s2,...) - Return the arguments joined together into a path.
+
+Networking
+----------
+
+Network information functions:
+
+* gethostbyname - Convert a host name into an IP address.  The host name could
+  not be resolved, the input value will be returned.
+* gethostbyaddr - Convert an IPv4/IPv6 address into a host name.  If the
+  reverse lookup fails, the input value will be returned.
+
+Internal State
+--------------
+
+The following functions can be used to access **lnav**'s internal state:
+
+* log_top_line() - Return the line number at the top of the log view.
+* log_top_datetime() - Return the timestamp of the line at the top of the log
+  view.
+
+Collators
+---------
+
+* naturalcase - Compare strings "naturally" so that number values in the string
+  are compared based on their numeric value and not their character values.
+  For example, "foo10" would be considered greater than "foo2".
+* naturalnocase - The same as naturalcase, but case-insensitive.
+* ipaddress - Compare IPv4/IPv6 addresses.

src/fs-extension-functions.cc

@@ -165,6 +165,10 @@ int fs_extension_functions(const struct FuncDef **basic_funcs,
         { "dirname",   1, 0, SQLITE_UTF8, 0, sql_dirname  },
         { "joinpath", -1, 0, SQLITE_UTF8, 0, sql_joinpath },
 
+        /*
+         * TODO: add other functions like readlink, normpath, ... 
+         */
+
         { NULL }
     };