E-mail log reports

May 25, 2010 at 07:57:29
Specs: Benelix Unix Linux
I have mentioned a detailed requirement to be very clear. The points with '*' are the tasks that the script is
expected to do. All other information provided is for clear understanding of the script functionality.

The basic form of the script will simply send a copy of all lines that contain either "error" or "warning"
to the user specified in the script as RECIPIENT. Note that RECIPIENT is a shell variable that will contain
an e-mail address.
The basic script will be able to look at a log file periodically and report on any new lines that contain
either "error" or "warning". The search for this text is to be "case insensitive". This means that the script
needs to accept a line containing either "error", "Error", "ERROR" or the word error spelt in any other
combination of upper and lower case letters as being functionally equivalent.

The basic script, named "logmon" (for log monitor) will need to do the following:

* be able to run in an environment that does not include a path. (i.e. the environment provided by
cron)

* accept the name of a log file as the sole command line argument
* search the specified log file for lines that have been added since the previous search.
* construct an e-mail reporting any lines that contain either "error" or "warning" in any combination
of upper and lower case letters
* send the e-mail to the address specified in the RECIPIENT shell variable which is defined within the
script

Optional requirements for now
Include only putting a single copy of a message in an e-mail, but reporting
how many times the message is repeated. Also modifying the script so that the same script will either
report via e-mail, or directly to a terminal, depending on how it is called. The final option will
be a system that reports errors directly to the root console.
(Note that this will need to be implemented in a way that sends the message to your account and does not actually send a message to root for testing purpose.)

Please add a comment or comments in your script that state which, if any, of the optional requirements have been implemented.

* modify the script so that, instead of listing multiple lines with the same message, it lists one instance
of each duplicated line (Note that for the purposes of this requirement, two lines that share everything
except the timestamp are considered to be identical lines.)

* follows the single instance of a duplicated line with a line that reports how many times the previous
line repeats

* add code that will report the above to the command line when it is called with the name "logcheck"
(How could you have one script that has two different names?)


Note that any data that needs to be passed from one time when the script runs to the next time cannot be passed in a temporary file. It will
need to be stored in a secure location.

Certain commands that I thought might be helpful are diff, wc, tail, grep, awk, and sort. Can anyone think of a way to do the above. Any suggestions
are most welcome.

Thanks in Advance for all the help.


See More: E-mail log reports

Report •


#1
May 31, 2010 at 23:23:05
Came up with some parts of the script that i have done are below:

#!/bin/sh -x
#
#
#
PRINTF=/usr/bin/printf
GREP=/usr/bin/grep
TAIL=/usr/bin/tail
LOG_FILE="$1"
#To check if log file to check has been given as argument when running the script
if [ $# -ne 0 ] ; then
$PRINTF "%s: Is the name of the log file you will be scanning. \n" $1
$PRINTF $LOG_FILE"\n"

if [ -f $LOG_FILE ]; #if logfile exists
then
echo "File $LOG_FILE exists \n"
else
echo "File $LOG_FILE does not exists \n"
fi

else
$PRINTF "%s: Need the name of the log that needs to be checked\n" $0
# EXIT (1)
fi

In the above code I want to check if the logfile exists anywhere in the system & not just the current directory and then use tail -f on it. But i am unable to figure out how to do it.

Some parts of the code that i tried coming up with but not included in the script are below:

# script to send simple email
SUBJECT="Enter Subject Here" # email subject
RECIPIENT="yourmailid@domain.com" # Email To ?
EMAILMESSAGE="My test script" # Email text/message

/bin/mailx -s "$SUBJECT" "$RECIPIENT" < $EMAILMESSAGE # send an email using /bin /mail

# script to check for string to search
SEARCH='grep -i -E -q 'error|warning'' #search for strings error or warning in a ny form
SCAN_LOG='tail $1|[${SEACH}]' #scanning logfile for string pattern

#Keep track of similar entries
MESSAGE_COUNTER=0
MESSAGE="$LOG_ENTRY"


Report •

#2
June 2, 2010 at 12:23:06
You can use the Unix find command to look for a file any where on the file system:

# UNTESTED
find /tmp -type f -name "$LOG_FILE" -print|while read filename
do
   echo "$filename"
#   break
done

My example searches /tmp, but you can use any directory that exists.

You need to make a decision what you are going to do in case you find more than one log file. One method would be to break out of the while loop after the first file.


Report •

#3
June 2, 2010 at 17:11:06
The script now basically works fine. Certain advanced requirements of the script task that i have not implemented are:

1. If same script is renamed to logcheck then the reports the script generates should be displayed in the command line.
2. Error messages that are repeated are to be counted and only one instance of error message is to be reported followed by the comment "The above message appeared n times"

Note: Time stamp is being ignored when making the above improvements. i.e a log entry having identical time stamps will not be considered when grouping messages.

i came up with a solution for the script. here is my solution. if ne1 cud improvise it it would b gr8:

#!/bin/sh
#set -x
#
#
#############
#
#THE PURPOSE OF THIS SCRIPT 'logmon'?
#THE SCRIPT IS INTENDEED TO MONITOR LOG FILES AND REPORT ANY ENTRIES THAT CONTAIN THE PATTERN "ERROR" OR "WARNING" IN ANY FORMAT
#THE SCRIPT SENDS AN EMAIL TO THE ID MENTIONED IN THE RECIPIENT VARIABLE IF THE PATTERN IS FOUND.
# NOTE:- THE SCRIPT IS INTENDED TO CHECK A LOG FILE COMPLETELY WHEN IT IS SCANNED FOR THE FIRST TIME AND THEN CHECKS ONLY THE NEWLY ADDED LOG ENTRIES
#HENCEFORTH.
#
#############
#
#WHAT DOES THE SCRIPT REQUIRE TO EXECUTE?
#THIS SCRIPT TAKES AN ARGUMENT WHICH IS THE ABSOLUTE PATH WITH THE NAME OF THE FILE THAT YOU WISH TO SCAN.
#
############
#
#WHAT DOES MY SCRIPT DO SO FAR?
#IT CHECKS IF AN ARGUMENT IS PASSED WHEN THE SCRIPT IS BEING EXECUTED.
#IT EXITS OUT WITH AN ERROR MESSAGE IF THE ARGUMENT IS NOT GIVEN.
#ON PROPER EXECUTION, IT FIRST GIVES INFORMATION OF WHICH FILE IS GOING TO BE SCANNED.
#FINALLY IT DISPLAYS A MESSAGE OF WHETHER AN E-MAIL WAS SENT OR NOT
#IF THE LOG FILE BEING SEARCHED IS NOT LOCATED IN THE PATH DISPLAYED EARLIER,IT DISPLAYS A FILE NOT FOUND MESSAGE AND EXITS THE SCRIPT
#NOTE: THE SCRIPT CHECKS WHETHER THE CURRENTLY SCANNED LOG FILE HAS BEEN PREVIOUSLY SCANNED OR NOT. IF NOT SCANNED A TMP_LOG FILE IS CREATED
#AND CONTENTS OF THE LOG FILE IS DUMPED INTO TMP_LOG. IF SCANNED PREVIOUSLY, THEN ALL UPDATED ENTRIES ARE PLACED IN A TEMPORARY FILE AND ONLY
#THOSE ENTRIES ARE CHECKED & REPORTED FOR THE SEARCH PATTERN
#
############
#
PRINTF=/usr/bin/printf
EGREP=/usr/bin/egrep
TAIL=/usr/bin/tail
COPY=/usr/bin/cp
CAT=/usr/bin/cat
DIFF=/usr/bin/diff
REMOVE=/usr/bin/rm
#LOG_FILE_PATH="$1" # should contain the absolute path of the file to be scanned
SEARCH=`$EGREP -i 'error|warning' $1` #search for strings error or warning in any form
RECIPIENT="validmailid@domain.extension" # Email To ?
SUBJECT="Error/Warning messages in logfile" # email subject


if [ $# -ne 0 ] ; #To check if log file to be scanned has been given as argument when running the script
then
$PRINTF "%s is the log file that will be read \n" $1

if [ -f $1 ]; #check if log file to be scanned exists
then
$PRINTF "File exists \n"

if [ -f ./tmp_log ]; #To check if current log file was scanned for the first time
then

$DIFF -b $1 ./tmp_log > ./diff_result #Resource that contains only updated entries of the current log file
SEARCH_UPDATES_ONLY=`$EGREP -i 'error|warning' ./diff_result`
$REMOVE ./diff_result
EMAILMESSAGE="$SEARCH_UPDATES_ONLY" # Email text/message Contains log entries from diff_result
if [ -z "$SEARCH_UPDATES_ONLY" ];
then
$PRINTF "There is no message to send \n"

else
$PRINTF "%s\n" "$EMAILMESSAGE" | /bin/mailx -s "$SUBJECT" "$RECIPIENT" #sends email of updated log entries that match the pattern
$PRINTF "Mail sent to %s" $RECIPIENT
fi
else
EMAILMESSAGE="$SEARCH" # Email text/message #Contains log entries from actual log file
if [ -z "$SEARCH" ];
then
$PRINTF "There is no message to send \n"
else
$PRINTF "%s\n" "$EMAILMESSAGE" | /bin/mailx -s "$SUBJECT" "$RECIPIENT" #send email with all log entries that matche the search pattern
$PRINTF "Mail sent to %s" $RECIPIENT
fi
fi
$COPY $1 ./tmp_log # To keep track of what is updated when this log file is scanned again
else
$PRINTF "File %s does not exist \n" $1
$PRINTF "Please place log file to be scanned in the above mentioned path \n"
fi

else
$PRINTF "Script %s: needs the absolute path of the log that needs to be checked\n" $0
fi

The output of a successful run will be as below depending on the situation:

sh logmon /<dir>/<dir>/<dir>/log.01
/<dir>/<dir>/<dir>/log.01 is the log file that will be read
File exists
There is no message to send

or

sh logmon /<dir>/<dir>/<dir>/log.01
/<dir>/<dir>/<dir>/log.01 is the log file that will be read
File exists
Mail sent to validmailid@domain.extension>


Report •

Related Solutions


Ask Question