It looks like your log file has only HH:MM:SS. If the log file spans multiple days, hard to tell if an entry is 30 minutes old or n days and 30 minutes old. In attempt to solve that problem, I process only the last 200 lines of the log file. This needs to be a figure that will get at least the last 30 minutes but not more than 24 hours. Processing more than the last 24 hours could produce false positives, and processing too few lines would result in failing to indicate a true positive. If 1000 lines were a daily average, I would suggest processing the last 500 lines.
I don't think we can check for consecutive entries because I see other non-related entries interspersed. I believe you are wanting the script to indicate when it sees a current (recent) "waiting" message, and also sees that these messages have been showing up for at least the last 20 minutes. My script defines windows of time. To be a "waiting" situation, it must see a log entry no older than 180 seconds AND must see at least one entry that is 20-25 minutes old. You need to put a top limit on it because you do not want it to indicate a waiting situation if it sees a recent entry and an entry 3 hours old, because that old entry would be for a prior event.
These windows of time are defined in the awk code as:
agedbeg=1500
agedend=1200
recent=180
which says that "recent" entries are those in the last 180 seconds, and "aged" entries are those that are 1200-1500 seconds old.
As currently coded, it does not care if there have been any entries between the "recent" entries and the "aged" entries. But if you wanted the script to do that, you could define a few more windows, such as 3-8 minutes ago, 8-13 minutes ago, etc, and the script could ensure that there has been at least one entry in each time window. Simple to do - just a few more lines of code.
The logic detects and handles midnight wrap around.
awk could print a message, but I figure you want to do something besides print a message (send and email?), so I exit awk with either a 0 or 1.
If you want to use other than ksh, you will probably need to recode the computation of csec which is the current hours, minutes and seconds converted into seconds.
# !/bin/ksh
date "+%H %M %S" |
read hr min sec
((csec=hr*3600+min*60+sec))
#echo "TEST: hr=$hr min=$min sec=$sec csec=$csec"
tail -200 cara.log |
awk -v csec=$csec 'BEGIN {\
#print "TEST: csec=" csec
agedbeg=1500
agedend=1200
recent=180 }
/Waiting in scheduler queue/ {\
gsub(":"," ")
tsec=$1*3600+$2*60+$3
if (tsec>csec)
cseca=csec+86400
else
cseca=csec
if (tsec+recent>=cseca)
{rcnt++;next}
if (tsec+agedbeg>=cseca && cseca>=tsec+agedend)
{acnt++;next}
} END {\
if (rcnt>0 && acnt>0)
exit 1
else
exit 0
}'
if [ $? -eq 1 ] ; then
echo "Waiting in scheduler queue detected"
fi
exit 0
