lnav - A fancy log file viewer DESCRIPTION =========== The log file navigator, lnav, is an enhanced log file viewer that takes advantage of any semantic information that can be gleaned from the files being viewed, such as timestamps and log levels. Using this extra semantic information, lnav can do things like interleaving messages from different files, generate histograms of messages over time, and providing hotkeys for navigating through the file. It is hoped that these features will allow the user to quickly and efficiently zero in on problems. OPENING PATHS/URLs ================== The main arguments to lnav are the files, directories, glob patterns, or URLs to be viewed. If no arguments are given, the default syslog file for your system will be opened. These arguments will be polled periodically so that any new data or files will be automatically loaded. If a previously loaded file is removed or replaced, it will be closed and the replacement opened. Note: When opening SFTP URLs, if the password is not provided for the host, the SSH agent can be used to do authentication. OPTIONS ======= Lnav takes a list of files to view and/or you can use the flag arguments to load well-known log files, such as the syslog or apache log files. The flag arguments are: -a Load all of the most recent log file types. -r Load older rotated log files as well. When using the flag arguments, lnav will look for the files relative to the current directory and its parent directories. In other words, if you are working within a directory that has the well-known log files, those will be preferred over any others. Any files given on the command-line are scanned to determine their log file format and to create an index for each line in the file. You do not have to manually specify the log file format. The currently supported formats are: syslog, apache, strace, tcsh history, and generic log files with timestamps. Lnav will also display data piped in on the standard input. The following options are available when doing so: -t Prepend timestamps to the lines of data being read in on the standard input. -w file Write the contents of the standard input to this file. To automatically execute queries or lnav commands after the files have been loaded, you can use the following options: -c cmd A command, query, or file to execute. The first character determines the type of operation: a colon is used for the built-in commands; a semi-colon for SQL queries; and a pipe symbol (|) for executing a file containing other commands. For example, to open the file "foo.log" and go to the tenth line in the file, you can do: lnav -c ':goto 10' foo.log This option can be given multiple times to execute multiple operations in sequence. -f file A file that contains commands, queries, or files to execute. This option is a shortcut for "-c '|file'". You can use a dash (-) to execute commands from the standard input. To execute commands/queries without the opening the interactive text UI, you can pass the '-n' option. This combination of options allows you to write scripts for processing logs with lnav. For example, to get a list of IP addresses that dhclient has bound to in CSV format: #! /usr/bin/lnav -nf # Usage: dhcp_ip.lnav /var/log/messages # Only include lines that look like: # Apr 29 00:31:56 example-centos5 dhclient: bound to 10.1.10.103 -- renewal in 9938 seconds. :filter-in dhclient: bound to # The log message parser will extract the IP address as col_0, so we # select that and alias it to "dhcp_ip". ;select distinct col_0 as dhcp_ip from logline; # Finally, write the results of the query to stdout. :write-csv-to - DISPLAY ======= The main part of the display shows the log lines from the files interleaved based on time-of-day. New lines are automatically loaded as they are appended to the files and, if you are viewing the bottom of the files, lnav will scroll down to display the new lines, much like 'tail -f'. On color displays, the lines will be highlighted as follows: * Errors will be colored in red; * warnings will be yellow; * boundaries between days will be underlined; and * various color highlights will be applied to: IP addresses, SQL keywords, XML tags, file and line numbers in Java backtraces, and quoted strings. To give you an idea of where you are spatially, the right side of the display has a proportionally sized 'scroll bar' that indicates your current position in the files. The scroll bar will also show areas of the file where warnings or errors are detected by coloring the bar yellow or red, respectively. Tick marks will also be added to the left and right hand side of the bar, for search hits and bookmarks. A bar on the left side is color coded and broken up to indicate which messages are from the same file. Pressing the left-arrow or 'h' will reveal the source file names for each message and pressing again will show the full paths. When at the bottom of the log view, a summary line will be displayed on the right-hand-side to give you some more information about your logs, including: how long ago the last message was generated, the number of log files, the error rate, and how much time the logs cover. The error rate display shows the errors-per-minute over the last five minutes. A bar chart is also overlaid on the "Error rate" label to show the error rate over the past ten seconds. For example, if there have not been many errors in the past five minutes and there is a sudden spike, the bar chart will fill up completely. But, if there has been a steady stream of errors, then the chart will only partially fill based on the recent error frequency. Above and below the main body are status lines that display: * the current time; * the name of the file the top line was pulled from; * the log format for the top line; * the current view; * the line number for the top line in the display; * the current search hit and the total number of hits; * the number of lines not displayed because of filtering. Finally, the last line on the display is where you can enter search patterns and execute internal commands, such as converting a unix-timestamp into a human-readable date. The command-line is implemented using the readline library, so the usual set of keyboard shortcuts are available. Most commands and searches also support tab-completion. The body of the display is also used to display other content, such as: the help file, histograms of the log messages over time, and SQL results. The views are organized into a stack so that any time you activate a new view with a key press or command, the new view is pushed onto the stack. Pressing the same key again will pop the view off of the stack and return you to the previous view. Note that you can always use 'q' to pop the top view off of the stack. KEY BINDINGS ============ Views ----- ? View/leave this help message. q Leave the current view or quit the program when in the log file view. Q Similar to 'q', except it will try to sync the top time between the current and former views. For example, when leaving the spectrogram view with 'Q', the top time in that view will be matched to the top time in the log view. a/A Restore the view that was previously popped with 'q/Q'. The 'A' hotkey will try to match the top times between the two views. X Close the current text file or log file. Spatial Navigation ------------------ g/home Move to the top of the file. G/end Move to the end of the file. If the view is already at the end, it will move to the last line. space/pgdn Move down a page. b/bs/pgup Move up a page. j/cr/down-arrow Move down a line. k/up-arrow Move up a line. h/left-arrow Move to the left. In the log view, moving left will reveal the source log file names for each line. Pressing again will reveal the full path. l/right-arrow Move to the right. H/Shift+left Move to the left by a smaller increment. L/Shift+right Move to the right by a smaller increment. e/E Move to the next/previous error. w/W Move to the next/previous warning. n/N Move to the next/previous search hit. When pressed repeatedly within a short time, the view will move at least a full page at a time instead of moving to the next hit. f/F Move to the next/previous file. In the log view, this moves to the next line from a different file. In the text view, this rotates the view to the next file. >/< Move horizontally to the next/previous search hit. o/O Move forward/backward to the log message with a matching 'operation ID' (opid) field. u/U Move forward/backward through any user bookmarks you have added using the 'm' key. This hotkey will also jump to the start of any log partitions that have been created with the 'partition-name' command. y/Y Move forward/backward through the log view based on the "log_line" column in the SQL result view. s/S Move to the next/previous "slow down" in the log message rate. A slow down is detected by measuring how quickly the message rate has changed over the previous several messages. For example, if one message is logged every second for five seconds and then the last message arrives five seconds later, the last message will be highlighted as a slow down. Chronological Navigation ------------------------ d/D Move forward/backward 24 hours from the current position in the log file. 1-6/Shift 1-6 Move to the next/previous n'th ten minute of the hour. For example, '4' would move to the first log line in the fortieth minute of the current hour in the log. And, '6' would move to the next hour boundary. 0/Shift 0 Move to the next/previous day boundary. r/R Move forward/backward based on the relative time that was last used with the 'goto' command. For example, executing ':goto a minute later' will move the log view forward a minute and then pressing 'r' will move it forward a minute again. Pressing 'R' will then move the view in the opposite direction, so backwards a minute. Bookmarks --------- m Mark/unmark the line at the top of the display. The line will be highlighted with reverse video to indicate that it is a user bookmark. You can use the 'u' hotkey to iterate through marks you have added. M Mark/unmark all the lines between the top of the display and the last line marked/unmarked. J Mark/unmark the next line after the previously marked line. K Like 'J' except it toggles the mark on the previous line. c Copy the marked text to the X11 selection buffer or OS X clipboard. C Clear all marked lines. Display options --------------- P Switch to/from the pretty-printed view of the log or text files currently displayed. In this view, structured data, such as XML, will be reformatted to make it easier to read. t Switch to/from the text file view. The text file view is for any files that are not recognized as log files. Ctrl-L (Lo-fi mode) Exit screen-mode and write the displayed log lines in plain text to the terminal until a key is pressed. Useful for copying long lines from the terminal without picking up any of the extra decorations. T Toggle the display of the "elapsed time" column that shows the time elapsed since the beginning of the logs or the offset from the previous bookmark. Sharp changes in the message rate are highlighted by coloring the separator between the time column and the log message. A red highlight means the message rate has slowed down and green means it has sped up. You can use the "s/S" hotkeys to scan through the slow downs. i View/leave a histogram of the log messages over time. The histogram counts the number of displayed log lines for each bucket of time. The bars are layed out horizontally with colored segments representing the different log levels. You can use the 'z' hotkey to change the size of the time buckets (e.g. ten minutes, one hour, one day). I Switch between the log and histogram views while keeping the time displayed at the top of each view in sync. For example, if the top line in the log view is "11:40", hitting 'I' will switch to the histogram view and scrolled to display "11:00" at the top (if the zoom level is hours). z/Shift Z Zoom in or out one step in the histogram view. v Switch to/from the SQL result view. V Switch between the log and SQL result views while keeping the top line number in the log view in sync with the log_line column in the SQL view. For example, doing a query that selects for "log_idle_msecs" and "log_line", you can move the top of the SQL view to a line and hit 'V' to switch to the log view and move to the line number that was selected in the "log_line" column. If there is no "log_line" column, lnav will find the first column with a timestamp and move to corresponding time in the log view. TAB/Shift+TAB In the SQL result view, cycle through the columns that are graphed. Initially, all number values are displayed in a stacked graph. Pressing TAB will change the display to only graph the first column. Repeatedly pressing TAB will cycle through the columns until they are all graphed again. p In the log view: enable or disable the display of the fields that the log message parser knows about or has discovered. This overlay is temporarily enabled when the semicolon key (;) is pressed so that it is easier to write queries. In the DB view: enable or disable the display of values in columns containing JSON-encoded values in the top row. The overlay will display the JSON-Pointer reference and value for all fields in the JSON data. CTRL-W Toggle word-wrapping. CTRL-P Show/hide the data preview panel that may be opened when entering commands or SQL queries. x Toggle the hiding of log message fields. The hidden fields will be replaced with three bullets and highlighted in yellow. F2 Toggle mouse support. Query ----- / Start a search for the given regular expression. The search is live, so when there is a pause in typing, the currently running search will be canceled and a new one started. The first ten lines that match the search will be displayed in the preview window at the bottom of the view. History is maintained for your searches so you can rerun them easily. Words that are currently displayed are also available for tab-completion, so you can easily search for values without needing to copy-and-paste the string. If there is an error encountered while trying to interpret the expression, the error will be displayed in red on the status line. While the search is active, the 'hits' field in the status line will be green, when finished it will turn back to black. Note: The regular expression format used by is PCRE (Perl-Compatible Regular Expressions). For example, if you wanted to search for ethernet device names, regardless of their ID number, you can type: eth\d+ You can find more information about Perl regular expressions at: http://perldoc.perl.org/perlre.html If the search string is not valid PCRE, a search is done for the exact string instead of doing a regex search. : Execute an internal command. The commands are listed below. History is also supported in this context as well as tab-completion for commands and some arguments. The result of the command replaces the command you typed. ; Execute an SQL query. Most supported log file formats provide a sqlite virtual table backend that can be used in queries. See the SQL section below for more information. |