[olug] Splunk and log scraping

[Irish]

On Fri, Dec 18, 2009 at 11:58 AM, T. J. Brumfield <enderandrew at gmail.com>wrote:

> The problem I'm seeing with most solutions is they are geared
> specifically at syslog, Windows event logs, etc.
>
> Specifically at the moment we're looking for a solution for custom
> application logs. We're re-writting the app in question right now, so
> altering the log format itself is fairly easy. Currently when a person
> has a problem, we have them upload their logs from a client to a
> central place where one person looks at them manually. We want to
> instead automatically push all logs to a central repository where we
> can monitor for problems in real-time.
>
> * I need something that I can point at a directory, and it will
> monitor all logs in that directory. If I have to specify specific file
> names ahead of time, it won't work. The logs wil have different names
> for different users, and that will change over time. I don't want
> something with massive maintenace overhead.
> * I need something where I can create custom searches on the fly when I
> need to.
> * I need something where I can also specify specific searches ahead of
> time to monitor for automatically, and then trigger an event. Exactly
> how that integrates with external tools (ticketing systems, SCOM,
> SiteScope) I can work around later.
>

Custom logs? Get ready to pay up to a vendor for the parcing ability, or do
it your self in house (also expensive). Sounds like you may need to
standardize your application log names & formats before tackling this
elephant.

Log scraping? Sounds like marketing buzz word bingo. I never listen to them
- give me the real technical guy. ;)

AFAIK, there is no products out there you can just point at a directory &
let it go to town, magically figuring out what files to monitor & archive
and what to do with the contents. If I only had a patent...