[olug] Splunk and log scraping

[Irish]

TJ,

I would suggest reading a few of the articles & white papers published over
the last few years regarding log management & SIM/SEM solutions (or rather,
products). Your use of 'log scraping' is not a term we are using in this
realm. Second, there are a few white papers that list some products that may
meet your needs & desires. Getting a better understanding of what products
are out there, with the background and terms used will assist in your
determination of your organizations needs & desires (wants & must haves).
Determine what capabilities you need, then find the product - not the other
way around.

To consider: What size environment is the target? What log sources? Any
custom logs?

Once you determine what your organizations needs & desires (wants & must
haves) are, and have a handful of possible products we can give you the
corporate knowledge (the good, bad, & the ugly).

You might want to prioritize a list like this before trying to figure out
what product is "best". You may find that different products and tools are
better at some areas and weak in others. No one product will fulfill all of
these goals to your complete satisfaction. It is very expensive to do all of
these things.

to aggregate, archive and index event logs to support identification of
incidents from endpoint technologies where centralization does not exist
to sort and prioritize events a human cannot process with more screens
to provide a sequentially interactive hypothesis/proof view into event logs
to speed up event triage and investigation activities
to be the knowledge base that holds learned, highly repetitive analysis
algorithms (shudder to use the term expert system)
to process high frequency events into lower frequency incidents by
automating analysis algorithms
to reduce cost of creating an incident report
to manage workflow from interesting event to a closed incident
to centralize reporting on security metrics that evaluate the effectiveness
of existing security processes
to provide compliance/certification evidence reports to third party
assessors/auditors/accreditors

Are you limiting yourself to commerial products, or also Open Source
projects? Also, after my initial research for a log management solution, we
realized that the majority of log management solutions that were available
just a few years ago are now rolled into vendor's SIM or SEM products.
Depending on management views, a full blown SIM/SEM might be overkill for
your log management requirement, but something to consider.

If you want a log repository (log management solution), these are great for
aggregating all of your logs (and automating this process); and depending on
the product good for you to data mine the collection. Some log management
solutions are starting to add the alerting functionality - blurring the
lines between these tools and the SIM/SEM products, though not having quite
the same capabilities.