Treating a log4j log like a database

I’m finally posting a Python module I wrote months ago that I have found to be very handy.

Our log4j log files get big — really big. It doesn’t help that we log just about everything under the sun. This makes it difficult to parse through and find the messages that we are looking for. I got to thinking, “With these set fields, the log files are really just a database. Why can’t I query like that?” I know that Chainsaw does something like this, but I wanted to put it into my own scripts. Python scripts, that is.

Thus loginfo.py was born. This module contains one class called LogInfo that parses the log and then lets you do simple queries on it. So you can grab all INFO messages, all messages containing a certain word, or all messages in X hours or minutes ago. You can also chain these queries — so you can get all INFO messages that occurred in the past 4 hours.

Need an example? Here is the example in the module:

lfile = file(sys.argv[1])
log = LogInfo()

for x in lfile.readlines():
log.addEntry(x)

## prints out all error messages from the past eight hours
for msg in log.error().hoursAgo(8):
print msg['date'],msg['sev'],msg['class'],msg['message']

Simple, huh? Grab the module here: loginfo.py

Jython 2.2b1

Testers wanted.

The Jython Project
The Jython development team is proud to announce a new release: Jython 2.2 Beta1!

It’s been a long, wild ride from Jython 2.1 to this point. Let’s help them get this out the door quickly.

Congrats to all the developers to get the code to this point!

About Recursion

“Fabulous Adventures In Coding : How Not To Teach Recursion” said:
A Joel On Software reader asked the other day for examples of recursive functions other than old chestnuts like Fibonacci or factorial

I’ve never figured out what is difficult about recursion — I’ve always just “gotten it”. Am I that special? Or did I have a good instructor? I vote the latter — the second semester programming (in Pascal) our class was small enough that he had people go to the white board and demonstrate how recursion, linked lists, etc. worked. So you made dang sure that you knew what you were doing! So that may explain when opportunities arise to use recursion, I jump in whole-heartedly.

When you do an analysis of a recursive function, I think it’s easy to get down on big-O numbers. Well, big-O doesn’t take into account hardware, caching, etc. If you use recursion in a function, then you know that you have the function in cache (because you’re running it!) — you just add another item to the stack. And then there’s the fact that recursive functions are usually short and concise — so they are easier to understand (if you are Recursion Enlightened, anyway). So less code to do the work. I consider that a good thing.

So here is my recursion example that finds the greatest common divisor between two integers. My apologies for the bad format — stupid WordPress plugin doesn’t like Python.

def gcd(a,b):
if b == 0:
return a
else:
return gcd(b,a%b)

Eating Cookies with Python

At work, we can get to the production server logs through the browser. This seems nice, but it’s a lot of clicky-clicky just to get to the log file, then you have to download it. You can look at it in a browser, but to get real examination done, you need to open it in an editor. Did I mention that there was a lot of clicky-clicky with it?

So I started to write a Python script to do this, but I ran into a snag right-away. The site uses a cookie to authorize your access to the log server. The version of Python I have installed (2.3.4 — yep, it’s old) doesn’t have anything in urllib or urllib2. A few googles and a few false starts brought me to ClientCookie. Now we’re talking!

Apparently, this comes with Py 2.4 as cookielib, but as I don’t have 2.4, I downloaded and started working. I wasn’t looking forward to putting my username/password into a script, but it turned out that I didn’t have to — ClientCookie can read your browser’s cookie file and use that! This is quite cool and quite easy to do:

 import ClientCookie
cookies = ClientCookie.MozillaCookieJar()
cookies.load("/path/to/firefox/directory/cookies.txt")
site = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cookies))
fp = site.open("http://cookie.foo.com")

fp is the same object you would get from urllib2 — a file-like object. Just read it like a normal file!

This is so incredibly sweet and opens whole new hacking doors . . .

Why I still like Django over Rails

In an addition to this post I want to say that I still prefer Django over Rails.  I’ve spent a few hours the past couple of days going through the Django tutorial again and then started writing an application for it.  I feel like Django flows better for me.

Why is that?  Not just because it’s writting in Python (which I know a lot better than Ruby) but also it seems from a Python mindset.  You make your site, you make your app, and then all your models in that app are in the same file (called “models.py”.  How nice.)  When you make your views, you put all your views for that app in “views.py”. Things are easy in that world.

Oh, and Django’s built-in admin site rocks.  It just works in ways I didn’t expect it to.

It’s not that I don’t like Ruby.  I’ve tried it and it’s nice, but I’ve never found what it had that Python couldn’t give me.  And I think the libraries are better in Python anyway — both the standard library as well as the third-party libraries (ElementTree, anyone?).  I feel myself more productive in Python, hence Django, over Ruby, with or without Rails.