[olug] Splunk and log scraping

Fri Dec 18 18:58:45 UTC 2009

The problem I'm seeing with most solutions is they are geared
specifically at syslog, Windows event logs, etc.

Specifically at the moment we're looking for a solution for custom
application logs. We're re-writting the app in question right now, so
altering the log format itself is fairly easy. Currently when a person
has a problem, we have them upload their logs from a client to a
central place where one person looks at them manually. We want to
instead automatically push all logs to a central repository where we
can monitor for problems in real-time.

* I need something that I can point at a directory, and it will
monitor all logs in that directory. If I have to specify specific file
names ahead of time, it won't work. The logs wil have different names
for different users, and that will change over time. I don't want
something with massive maintenace overhead.
* I need something where I can create custom searches on the fly when I need to.
* I need something where I can also specify specific searches ahead of
time to monitor for automatically, and then trigger an event. Exactly
how that integrates with external tools (ticketing systems, SCOM,
SiteScope) I can work around later.

Ultimately I need to identify problems sooner.

Some of the Splunk guys I've talked to use the term log scraping. It
is the term I've always heard. A Google search for it shows Splunk and
other major products. I'm not sure why you feel the term isn't
appropriate. Any time you pull targetted information out of a large
group of data, it can be referred to as scraping.

On another day I might focus on a great overall tool for the company,
but as a corporation we don't exactly always standardize on tools, and
such decisions are above my pay grade. I need a solution for a
specific use case, and $300,000 for Splunk is pretty crazy for the one
app we'd use it for. I imagine I might even get the functionality I
need with a basic custom-wrtitten web app. We have some Savvion
developers asking if there are processes we'd like to see handled with
custom Savvion apps in the future, and I may just go that route.

I'll look at all the products you listed and see if any of them would
work for us.

-- T. J.

On Fri, Dec 18, 2009 at 12:17 PM, Irish <irish.masms at gmail.com> wrote:
> TJ,
>
> I would suggest reading a few of the articles & white papers published over
> the last few years regarding log management & SIM/SEM solutions (or rather,
> products). Your use of 'log scraping' is not a term we are using in this
> realm. Second, there are a few white papers that list some products that may
> meet your needs & desires. Getting a better understanding of what products
> are out there, with the background and terms used will assist in your
> determination of your organizations needs & desires (wants & must haves).
> Determine what capabilities you need, then find the product - not the other
> way around.
>
> To consider: What size environment is the target? What log sources? Any
> custom logs?
>
> Once you determine what your organizations needs & desires (wants & must
> haves) are, and have a handful of possible products we can give you the
> corporate knowledge (the good, bad, & the ugly).
>
> You might want to prioritize a list like this before trying to figure out
> what product is "best". You may find that different products and tools are
> better at some areas and weak in others. No one product will fulfill all of
> these goals to your complete satisfaction. It is very expensive to do all of
> these things.
>
> to aggregate, archive and index event logs to support identification of
> incidents from endpoint technologies where centralization does not exist
> to sort and prioritize events a human cannot process with more screens
> to provide a sequentially interactive hypothesis/proof view into event logs
> to speed up event triage and investigation activities
> to be the knowledge base that holds learned, highly repetitive analysis
> algorithms (shudder to use the term expert system)
> to process high frequency events into lower frequency incidents by
> automating analysis algorithms
> to reduce cost of creating an incident report
> to manage workflow from interesting event to a closed incident
> to centralize reporting on security metrics that evaluate the effectiveness
> of existing security processes
> to provide compliance/certification evidence reports to third party
> assessors/auditors/accreditors
>
> Are you limiting yourself to commerial products, or also Open Source
> projects? Also, after my initial research for a log management solution, we
> realized that the majority of log management solutions that were available
> just a few years ago are now rolled into vendor's SIM or SEM products.
> Depending on management views, a full blown SIM/SEM might be overkill for
> your log management requirement, but something to consider.
>
> If you want a log repository (log management solution), these are great for
> aggregating all of your logs (and automating this process); and depending on
> the product good for you to data mine the collection. Some log management
> solutions are starting to add the alerting functionality - blurring the
> lines between these tools and the SIM/SEM products, though not having quite
> the same capabilities.
>
> From what I have seen, there is no log management or SIM/SEM solution that
> meets my requirements or desires - but some come close. Different products
> have different focus, and any product that claims to do it all, likely
> doesn't understand the space.
>
> Many folks swear by Splunk, think it is awesome - but I think its more
> geared toward generic log aggragation and searching instead of pure SIM/SEM.
> My notes here on Splunk are from around April 2008, and they have extremely
> improved the product since then.
>
> Pro:
> Platform independent
> Database direct queries
> Fast drill downs
> Traditional search bar functionality (like Google)
> Public log message repository
>
> Con:
> Pricing based on database input speeds
> MySQL back end, unknown if there are other options
> Brand new product, not reviewed by publishers, minimal information on web
> Lacking in ad hock reporting
> Some of the functionality requires off site resources (Public log message
> repository)
> Good open source & information sharing, not good for a classified
> environment
> Not as robust as other solutions in the version tested.
> Early in the product development, suggest reevaluation after a few version
> releases.
>
> Other info:
> Application demo available online
> Configured a text box locally for an evaluation & review. Setup and
> evaluation was great, but we feel it is not as robust or has the essential
> elements for the task in the version tested.
>
>
> Other options (products) I know of:
> - Enterprise Security Analyzer (ESA) from EIQ networks
> - Security Center 3.0, Log Correlation Engine, & passive vulnerability
> scanner from Tenable Network Security
> - ELM Enterprise Manager™4.0 from TNT Software
> - Splunk
> - Novell Sentinel 5
> - Sensage
> - Activeworkx Security Center – CrossTec Corporation
> - eTrust Security commend center
> - Eventia Analyzer 2.0 by Checkpoint
> - Insight Security manager by Consul
> - Intellitactics
> - Security Management Center (SMC) by OpenService
> - ArcSight Enterprise Security Manager (ArcSight ESM)
> - Big Brother Log Analyzer (BBLA)
> - Doriansoft
> - Event Tracker by Prism Microsystems
> - fwlogwatch
> - Kiwi Syslog
> - LogCaster, from Rippletech
> - MARS 200 and SIMS (CiscoWorks) – Cisco
> - Metalog
> - Modular Syslog (Msyslog)
> - MonitorWare Line, from Adiscon
> - NetIQ
> - nsyslog
> - Open Source Host-based Intrusion Detection System (OSSEC HIDS)
> - Open Source Security Information Management (OSSIM)
> - rsyslog
> - San Diego Supercomputer Center (SDSC) Secure Syslog
> - Security Event Log Monitor (S.E.L.M.) by GFI LANguard
> - Syslog New Generation (Syslog-ng)
> - WinSyslog
> - The Simple Event Correlator -
> http://www.estpak.ee/~risto/sec/<http://www.estpak.ee/%7Eristo/sec/>
> - Lasso -  http://blog.loglogic.com/project_lasso/
>
> The appliance types:
> - enVision by Network Intelligence
> - HighTower Security Event Manager
> - LogLogic
> - Q1 Labs QRadar
> - Snare Server from InterSect Alliance
> - Symantec - SIM 9500 Series
> - TriGeo
>
> Does all this help, or did just make your life worse? :) Hope this helps! :)
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> https://lists.olug.org/mailman/listinfo/olug