lnav/src/help.txt

  lnav - A fancy log file viewer

DESCRIPTION
-----------

The log file navigator, lnav, is an enhanced log file viewer that
takes advantage of any semantic information that can be gleaned from
the files being viewed, such as timestamps and log levels.  Using this
extra semantic information, lnav can do things like interleaving
messages from different files, generate histograms of messages over
time, and providing hotkeys for navigating through the file.  It is
hoped that these features will allow the user to quickly and
efficiently zero in on problems.


OPTIONS
-------

Lnav takes a list of files to view and/or you can use the flag
arguments to load well-known log files, such as the syslog or apache
log files.  The flag arguments are:

  -s         Load the most recent syslog messages file. (Default)
  -a         Load all of the most recent log file types.
  -r         Load older rotated log files as well.

When using the flag arguments, lnav will look for the files relative
to the current directory and its parent directories.  In other words,
if you are working within a directory that has the well-known log
files, those will be preferred over any others.

Any files given on the command-line are scanned to determine their log
file format and to create an index for each line in the file.  You do
not have to manually specify the log file format.  The currently
supported formats are: syslog, apache, strace, tcsh history, and
generic log files with timestamps.

Lnav will also display data piped in on the standard input.  The
following options are available when doing so:

  -t         Prepend timestamps to the lines of data being read in
             on the standard input.
  -w file    Write the contents of the standard input to this file.


DISPLAY
-------

The main part of the display shows the log lines from the files interleaved
based on time-of-day.  New lines are automatically loaded as they are appended
to the files and, if you are viewing the bottom of the files, lnav will scroll
down to display the new lines, much like 'tail -f'.

On color displays, the lines will be highlighted as follows:

  * Errors will be colored in red;
  * warnings will be yellow;
  * boundaries between days will be underlined; and
  * various color highlights will be applied to: IP addresses, SQL keywords,
    XML tags, file and line numbers in Java backtraces, and quoted strings.

To give you an idea of where you are in the file spatially, the right
side of the display has a proportionally sized 'scrollbar' that
indicates your current position in the file.

Above and below the main body are status lines that display:

  * the current time;
  * the number of errors/warnings above and below your current
    position;
  * the number of search hits, which updates as more are found;
  * the line number for the top line in the display; and
  * the name of the file the top line was pulled from.

Finally, the last line on the display is where you can enter search
patterns and execute internal commands, such as converting a
unix-timestamp into a human-readable date.  The command-line is by
the readline library, so the usual set of keyboard shortcuts can
be used.


The body of the display is also used to display other content, such
as: the help file, histograms of the log messages over time, and
SQL results.  The views are organized into a stack so that any time
you activate a new view with a key press or command, the new view
is pushed onto the stack.  Pressing the same key again will pop the
view off of the stack and return you to the previous view.  Note
that you can always use 'q' to pop the top view off of the stack.


KEY BINDINGS
------------

To help navigate through the file there are many hotkeys that should
make it easy to zero-in on a specific section of the file or scan
through the file.

  ?                 View/leave this help message.
  q                 Leave the current view or quit the program when in
                    the log file view.

  home              Move to the top of the file.
  end               Move to the end of the file.
  space/pgdn        Move down a page.
  b/bs/pgup         Move up a page.
  j/cr/down-arrow   Move down a line.
  k/up-arrow        Move up a line.
  h/left-arrow      Move to the left.
  l/right-arrow     Move to the right.

  e/E               Move to the next/previous error.
  w/W               Move to the next/previous warning.
  n/N               Move to the next/previous search hit.
  f/F               Move to the next/previous file.  In the log view, this
                    moves to the next line from a different file.  In the
		    text view, this rotates the view to the next file.

  >/<               Move horizontally to the next/previous search hit.

  s                 Toggle "scrubbing" of log lines in the log view.  Log
                    formats that support scrubbing will try to eliminate
                    redundant information on a single line.  For example, the
                    year part of a timestamp or the log level text.

  t                 Switch to/from the text file view.  The text file view is
                    for any files that are not recognized as log files.

  o/O               Move forward/backward 60 minutes from the current
                    position in the log file.

  d/D               Move forward/backward 24 hours from the current
                    position in the log file.

  1-6/Shift 1-6     Move to the next/previous n'th ten minute of the
                    hour.  For example, '4' would move to the first
                    log line in the fortieth minute of the current
                    hour in the log.  And, '6' would move to the next
                    hour boundary.

  0/Shift 0         Move to the next/previous day boundary.

  m                 Mark/unmark the line at the top of the display.
		    The line will be highlighted with reverse video to
		    indicate that it is a user bookmark.  You can use
		    the 'u' hotkey to iterate through marks you have
		    added.

  M                 Mark/unmark all the lines between the top of the
		    display and the last line marked/unmarked.

  J                 Mark/unmark the next line after the previously
                    marked line.

  K                 Like 'J' except it toggles the mark on the
                    previous line.

  c                 Copy the marked text to the X11 selection buffer or OS X
                    clipboard.

  C                 Clear all marked lines.

  u/U               Move forward/backward through any user bookmarks
		    you have added using the 'm' key.

  T                 Toggle the display of the "time offset" column that shows
                    the time elapsed since the beginning of the logs or the
                    offset from the previous bookmark.

  i                 View/leave a histogram of the log messages over
                    time.  The histogram counts the number of
                    displayed log lines for each bucket of time.  The
                    bars are layed out horizontally with colored
                    segments representing the different log levels.
                    You can use the 'z' hotkey to change the size of
                    the time buckets (e.g. ten minutes, one hour, one
                    day).

  I                 Switch between the log and histogram views while
		    keeping the time displayed at the top of each view
		    in sync.  For example, if the top line in the log
		    view is "11:40", hitting 'I' will switch to the
		    histogram view and scrolled to display "11:00" at
		    the top (if the zoom level is hours).

  z/Shift Z         Zoom in or out one step in the histogram view.

  /<regexp>         Start a search for the given regular expression.
                    The search is live, so when there is a pause in
                    typing, the currently running search will be
                    canceled and a new one started.  History is
                    maintained for your searches so you can rerun them
                    easily.  If there is an error encountered while
                    trying to interpret the expression, the error will
                    be displayed in red on the status line.  While the
                    search is active, the 'hits' field in the status
                    line will be green, when finished it will turn
                    back to black.

		    Note: The regular expression format used by is PCRE
		    (Perl-Compatible Regular Expressions).  For example,
		    if you wanted to search for ethernet device names,
		    regardless of their ID number, you can type:

		      eth\d+

		    You can find more information about Perl regular
		    expressions at:

		      http://perldoc.perl.org/perlre.html

  :<command>        Execute an internal command.  The commands are
                    listed below.  History is also supported in this
                    context as well as tab-completion for commands and
                    some arguments.  The result of the command
                    replaces the command you typed.

  ;<sql>            Execute an SQL query.  Most supported log file
                    formats provide a sqlite virtual table backend
                    that can be used in queries.  See the SQL section
                    below for more information.

  y/Y               Move forward/backward through the log view based on the
                    "line_number" column in the SQL result view.

  v                 Switch to/from the SQL result view.

  V                 Switch between the log and SQL result views while
                    keeping the top line number in the log view in
		    sync with the line_number column in the SQL view.
		    For example, doing a query that selects for
		    "idle_msecs" and "line_number", you can move the
		    top of the SQL view to a line and hit 'V' to switch
		    to the log view and move to the line number that was
		    selected in the "line_number" column.

  p                 Enable or disable the display of the fields that the
                    log message parser knows about or has discovered.
                    This overlay is temporarily enabled when the semicolon
                    key (;) is pressed so that it is easier to write queries.

MOUSE SUPPORT (experimental)
----------------------------

If you are using Xterm, or a compatible terminal, you can use the mouse to
mark lines of text and move the view by grabbing the scrollbar.

NOTE: You need to manually enable this feature by setting the LNAV_EXP
environment variable to "mouse".

COMMANDS
--------

  help              Switch to this help text view.

  unix-time <secs-or-date>
                    Convert a unix-timestamp in seconds to a
                    human-readable form or vice-versa.
		    BEWARE OF TIMEZONE DIFFERENCES.

  current-time      Print the current time in human-readable form and
                    as a unix-timestamp.

  goto <line#|N%>   Go to the given line number or N percent into the
                    file.

  highlight <regex> Highlight strings that match the given regular
                    expression.

  filter-in <regex> Only display lines that match the given regular
                    expression.  This command can be used multiple
                    times to add more lines to the display.

  filter-out <regex>
                    Do not display lines that match the given regular
                    expression.  This command can be used multiple
                    times to remove more lines from the display.  If a
                    'filter-in' expression is also active, it takes
                    priority and the filter-out will remove lines that
                    were matched by the 'filter-in'.

  disable-filter <regex>
                    Disable an active 'filter-in' or 'filter-out'
                    expression.

  enable-filter <regex>
                    Enable a inactive 'filter-in' or 'filter-out'
                    expression.

  graph <regex>     Graph the value of numbers in the file(s) over
                    time.  The given regular expression should capture
                    the number to be displayed.  For example:

                      my stats: (\d+\.\d+)

                    Will graph all the "stats" values found in the
                    file.  XXX This is still mostly a toy...

  append-to <file>  Append any marked lines to the given file.

  write-to <file>   Write any marked lines to the given file.

  session <cmd>     Add the given command to the session file
                    (~/.lnav/session).  Any commands listed in the session file
                    are executed on startup.  Only the highlight and
                    filter-related commands can be added to the session file.


SQL QUERIES (experimental)
-----------

Lnav has support for performing SQL queries on log files using the
Sqlite3 "virtual" table feature.  For all supported log file types,
lnav will create tables that can be queried using the subset of SQL
that is supported by Sqlite3.  For example, to get the top ten URLs
being accessed in any loaded Apache log files, you can execute:

  ;select cs_uri_stem, count(*) as total from access_log 
     group by cs_uri_stem order by total desc limit 10;

The query result view shows the results as text and graphs any number
values found in the result, much like the histogram view.

The builtin set of log tables are listed below.  Note that only the
log messages that match a particular format can be queried by a
particular table.  You can find the file format and table name for
the top log message by looking in the upper right hand corner of the
log file view.

The log table names are as follows:

  access_log       Apache common access log format
  syslog_log       Syslog format
  glog_log         Google glog format
  strace_log       Strace log format
  generic_log      'Generic' log format.  This table contains messages
                   from files that have a very simple format with a
                   leading timestamp followed by the message.

The columns available for the top log line in the view will
automatically be displayed after pressing the semicolon (;) key.
All log tables contain at least the following columns:

    line_number    The line number in the file, starting at zero.
    log_time       The time of the log entry.
    idle_msecs     The amount of time, in milliseconds, between the
                   current log message and the previous one.
    level          The log level (e.g. info, error, etc...).
    path           The full path to the file.
    raw_line       The raw line of text.

The following tables include the basic columns as listed above and
include a few more columns since the log file format is more
structured.

  syslog_log

    log_hostname   The hostname the message was received from.
    log_procname   The name of the process that sent the message.
    log_pid        The process ID of the process that sent the message.

  access_log       (The column names are the same as those in the
                    Microsoft LogParser tool.)

    c_ip           The client IP address.
    cs_username    The client user name.
    cs_method      The HTTP method.
    cs_uri_stem    The stem portion of the URI.
    cs_uri_query   The query portion of the URI.
    cs_version     The HTTP version string.
    sc_status      The status number returned to the client.
    sc_bytes       The number of bytes sent to the client.
    cs_referrer    The URL of the referring page.
    cs_user_agent  The user agent string.

  strace_log       (Currently, you need to run strace with the
                    "-tt -T" options so there are timestamps for
                    each function call.)

    funcname       The name of the syscall.
    result         The result code.
    duration       The amount of time spent in the syscall.
    arg0 - arg9    The arguments passed to the syscall.

These tables are created dynamically and not stored in memory or on
disk.  If you would like to persist some information from the tables,
you can attach another database and create tables in that database.
For example, if you wanted to save the results from the earlier
example of a top ten query into the "/tmp/topten.db" file, you can do:

  ;attach database "/tmp/topten.db" as topten;
  ;create table topten as select cs_uri_stem, count(*) as total
     from access_log group by cs_uri_stem order by total desc
     limit 10;


DYNAMIC LOG LINE TABLE (experimental)
----------------------

(NOTE: This feature is still very new and not completely reliable yet,
 use with care.)

For log formats that lack message structure, lnav can parse the log
message and attempt to extract any data fields that it finds.  This
feature is available through the "logline" log table.  This table is
dynamically created and defined based on the message at the top of
the log view.  For example, given the following log message from "sudo",
lnav will create the "logline" table with columns for "TTY", "PWD",
"USER", and "COMMAND":

  May 24 06:48:38 Tim-Stacks-iMac.local sudo[76387]: stack : TTY=ttys003 ; 
    PWD=/Users/stack/github/lbuild ; USER=root ;
    COMMAND=/bin/echo Hello, World!

Queries executed against this table will then only return results for
other log messages that have the same format.  So, if you were to
execute the following query while viewing the above line, you might
get the following results:

  ;select USER,COMMAND from logline;

  USER | COMMAND
  ---- | -------------------------
  root | /bin/echo Hello, World!
  mal  | /bin/echo Goodbye, World!


The log parser works by examining each message for key/value pairs
separated by an equal sign (=) or a colon (:).  For example, in the
previous example of a "sudo" message, the parser sees the "USER=root"
string as a pair where the key is "USER" and the value is "root".
If no pairs can be found, then anything that looks like a value is
extracted and assigned a numbered column.  For example, the following
line is from "dhcpd":

  Sep 16 22:35:57 drill dhcpd: DHCPDISCOVER from 00:16:ce:54:4e:f3 via hme3

In this case, the lnav parser recognizes that "DHCPDISCOVER", the MAC
address and the "hme3" device name are values and not normal words.  So,
it builds a table with three columns for each of these values.  The
regular words in the message, like "from" and "via", are then used to
find other messages with a similar format.