[Novalug] Looking for sample system and event logs..

[greg pryzby]

(Sales pitch for company I work for coming, so take this with a grain of
salt)

On Apr 2, 2008, at 9:51 PM, Scott Musman wrote:
Hi Greg,

I'm aware of splunk and while it's pretty kewl, I (perhaps incorrectly)
categorize it mainly as a interactive search tool. If you know what
you're looking for, then it's a great way to find it in all of the
mounds of log data you have. The key item however, is that it only finds
thing that match what you've asked it to look for.

I have not played with it lots, yet. However a customer with 10s of GB a day
are using this tool effectively. They tell (and showed me) how they can look
at the data and then join logs for 'root cause analysis' and build new ways
to look at the data and save those views.

Again, it was shown to me and I didn't ask lots of questions. This is a
company that spent money of the product.

It also has a fairly primitive "alerting" feature (where I describe it
as being primitive since last time I checked it was non real-time, and
couldn't recognize compound events that span multiple entries, or
correlate across log sources. So it can do more than just allow you to
search through data, but I don't think it does what I'm trying to do..

I don't think it does alerting. And that is why the company I am talking
about also uses Hyperic because of the very cool alerting. The customer set
up Hyperic to generate alerts based on system and application behavior and
then look at splunk to see what is happening at a log level. The two
together can be powerful.

If these comments are "off the mark", I'd welcome the inputs of others
to educate me properly. I certainly  do not pretend to be even remotely
a splunk expert, and I may be doing them a disservice, so please let me
know if splunk does more than I think it does.

Soo... What I think I'm doing that is different from splunk is that that
my anomaly detection piece will find for you those unusual, unexpected
events that occur. Unlike splunk you won't have to specifically look for
them, it will immediately make you aware that something odd (a rare
unusual event) is happening, and point out to you what it is.

Log files are helpful only if the OS/application/etc are writing logs. And
that isn't always the case. Imagine that you can watch and gather
information about the OS, software from a process and internal level and
alert on information gathered. That is what Hyperic does and does pretty
well.

An example might be something like the sudden appearance of "memory
errors" in your syslog. What makes them unusual is that you've never
seen them before! Often when memory fails, these will start to show up
intermittently, and then become more frequent just before the memory
really fails..

What if I am looking at the OS and watch the trending of the memory use and
alert before there is an issue?

Ordinarily you might not think to search your logs for memory errors,
and you might miss seeing what is only a few of them in the 100's of log
messages that occur during any day. But in this case, because my
(envisaged) little tool has profiled your machine it knows that it's
never seen these these messages before and report them to you.

If you wanted too.. I guess you could then use something like splunk to
research the unusual event more, or look for other possibly related
events around the time where the unusual events occur. So it's not
supposed to replace splunk, it's supposed to do something complimentary,
that splunk doesn't do.

Does this make any sense? There is an example of a similar type of tool
at http://www.estpak.ee/~risto/slct/ , (the same guy who developed sec,
which was the precursor to OSSEC) but it tells you what's normal, rather
than what's "not normal", so I think I can do something more useful.

Unlike Risto, unfortunately, I don't have access to all of the log files
he has, to be able to testing the algorithms out, so I'm asking for your
help.

Does this help explain what I want to do, and why I don't think it's
something splunk already does? If not.. keep at me. I hate re-inventing
wheels, so I'd rather someone where point me to an existing solution
than waste my time just because I didn't know. Thanks,


It is debatable if a solution(s) exist. Personally I think HHQ does what you
are looking for and more.

--
greg pryzby
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://calypso.tux.org/pipermail/novalug/attachments/20080402/0f611809/attachment.htm