[olug] Splunk and log scraping

[Ed Pluta]

Irish wrote:
> On Fri, Dec 18, 2009 at 11:58 AM, T. J. Brumfield <enderandrew at gmail.com>wrote:
>
>   
>> The problem I'm seeing with most solutions is they are geared
>> specifically at syslog, Windows event logs, etc.
>>
>> Specifically at the moment we're looking for a solution for custom
>> application logs. We're re-writting the app in question right now, so
>> altering the log format itself is fairly easy. Currently when a person
>> has a problem, we have them upload their logs from a client to a
>> central place where one person looks at them manually. We want to
>> instead automatically push all logs to a central repository where we
>> can monitor for problems in real-time.
>>
>> * I need something that I can point at a directory, and it will
>> monitor all logs in that directory. If I have to specify specific file
>> names ahead of time, it won't work. The logs wil have different names
>> for different users, and that will change over time. I don't want
>> something with massive maintenace overhead.
>> * I need something where I can create custom searches on the fly when I
>> need to.
>> * I need something where I can also specify specific searches ahead of
>> time to monitor for automatically, and then trigger an event. Exactly
>> how that integrates with external tools (ticketing systems, SCOM,
>> SiteScope) I can work around later.
>>
>>     
>
> Custom logs? Get ready to pay up to a vendor for the parcing ability, or do
> it your self in house (also expensive). Sounds like you may need to
> standardize your application log names & formats before tackling this
> elephant.
>
> Log scraping? Sounds like marketing buzz word bingo. I never listen to them
> - give me the real technical guy. ;)
>
> AFAIK, there is no products out there you can just point at a directory &
> let it go to town, magically figuring out what files to monitor & archive
> and what to do with the contents. If I only had a patent...
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> https://lists.olug.org/mailman/listinfo/olug
>
>   
I'm not really sure what splunk does, but simply finding keywords in 
files in a directory/list of directories is really not that hard. What 
happens with that info and how it is managed can be very simply or very 
complex. First step really should be standardizing file names and 
formats, that would greatly ease the process. Here's my take on simply 
finding keywords/ phrases.

If you are re-writing the app already you should be able to create a 
custom, standardized log format. When the app takes an error, or other 
event of "interest" have the application create the log and send it to a 
central server. I know .net can do this and would assume Java should 
have something similar. I hate to say this but XML would be a good 
format, since you could mine for data later using XQUERY or something 
similar, most languages have XML parsing modules or functions built in 
any more. You could also just use pipe or comma delimited files, with a 
fixed format (field1=userid, field2=hostname, field3=program, etc). Or 
log all info in a DB. But the user side application needs to standardize 
its error message format.

On the server side set up a daemon to look for files in the incoming 
directory. When it finds a file spawn a parser program/script to check 
for tags of interest. If none are found the script/program can move it 
to an "archive" directory. If it finds something that needs to be 
addressed it can moved to a "flagged" directory, and an email can be 
sent or some other action can be taken. All of that is pretty simple IF, 
big if, the format of the logs is known and standardized. It really 
depends on the format of the logs and what you are looking to get out of 
them. If the logs are just free form text you would need to identify key 
phrases, but I don't see why it would be overly complex to scan for 
them. It could be a simple as a shell script or a complex as a whole new 
application.

A quick script to give you an idea (may be some slight errors, you get 
the idea):

#!/bin/ksh
while [ 1 ];
do
for fle in $(ls $1)
do
    grep "ERROR|Critical" $fle    # search for error
    rc=$?  
    if [ $rc = 0 ];then       # if no error
       mv $fle /archive/dir  
    else
       sendmail some.one at who.cares
       $(do something else
       mve $fle /flagged/dir
    fi
done
sleep 30    # start all over again in 30 seconds
done
exit 0