Finally, a use for reduce

I’ve been using Python for a long time but I’m not sure that I’ve ever used the reduce.  A lot of people use it badly and I’ve never really “gotten” it.  But the other day I was tackling a problem and a light bulb went off in my head — “Isn’t this what reduceis for?”

The problem is that I had a list of regular expressions that I wanted to apply to a string and, if once of those hit, I wanted to return True.  Of course, I could put all those strings in one regular expression but I wanted to be able to add and take out those regular expressions on an ad hoc way.  And I don’t want to make three or four simple regex’s into one complex one.

So here is an example of what I did, in a contrived sort of way:

import re

flintstones=["Fred","Wilma","Pebbles"]

def inFamily(p):    result_list = [re.search(x,p) for x in flintstones]

    return reduce(lambda x,y: x or y, result_list)

print inFamily("Fred")print inFamily("Barney")

So the first called to inFamily returns a MatchObject (or, really, True) and the second returns None (or, really, False).

In the right context, reduce turns a complex problem into a simple one.

Powered by ScribeFire.

More Joys Of Python

I haven’t had a chance to do anything in Python lately.  But yesterday I needed to do some screen scraping and, after looking at the HTML I needed to scrap from, I knew that I needed some good tools.

The HTML is table in a form with several rows in it. In cell in each rows has hidden items representing that data.  And, inexplicably, each row also has a couple JavaScript functions embedded in it.  I did a File-Save on a page with ~150 rows in that table, which is large for what I am doing. You won’t believe the file size if I told you, so I’ll just copy-and-paste it in:

$ ls -lh file.htm-rw-r--r-- 1 me mkgroup-l-d 2.9M Aug 31 10:39 file.htm

Yes, one HTML file with one table is a little shy of 3MB.  There is no images embedded in this file — just pure text.  This is why I needed to get out the Big Guns for this exercise.

The Big Gun for this is Beautiful Soup.  It can parse anything that might resemble HTML and give you what you need.  It is not speedy, but it works well.  And for the hunk of HTML I need, correctness was more important than speed.

One thing I discovered about Beautiful Soup is that you can query with a regex.  Remember what I said that each cell in the table had a hidden form field in it?  The ID for each row was in a hidden input named “id0″ for the first ID, “id1″ for the second, and so on.  Similar for the customer column — “CustName0″,”CustName1″, etc.  So, really, this was easy to find:

soup = BeautifulSoup(file(fname))ids = soup.findAll("input", attrs={'name':re.compile("id\d+$")})custs = soup.findAll("input", attrs={'name':re.compile("CustName\d+$")})

What became a horrid parsing problem quickly became a three-liner.  Wow.

I needed to spit data out from this HTML into different files. Since I know what kind of a pain file management can be, I decided it would be great to put them in a Zip file.  I could have done a system call to zip but instead I used Python’s standard zipfile module.

I was storing the data files temporarily in a data directory, but I didn’t want to put that into the Zip file. Luckily that, too, was easy using the zipfile module and the non-standard yet wonderful path.py module:

zfile = ZipFile(zip_file,”w”)   
for f in fnames:       
    zfile.write(f,f.splitpath()[1])   
zfile.close()

Coding in Python has been a refreshing change of pace, because I can use these wonderful modules to bend the data to my will and all I have to it tell it how.  In Java, I have to worry about making sure the right kind of object is being passed and that things are casting right, etc., etc.  Although this is a highly complex problem it was easy in Python. 

Powered by ScribeFire.

Putting Object Ownership in Django

I’m working on a Top Secret web project that I will talk more about in the near to medium future. Because I’m a Python nut and need to make simple web application that has complex business rules (i.e. the technology isn’t complex, but the business logic can be ) I chose [Django](http://www.djangoproject.com/) as my engine of choice.

One of the issues I ran into is Django’s security model. By default, if a user or a group has permission to that object, the users have permissions on **all** those objects. This isn’t what I wanted — I wanted to give the idea of “ownership” to the object, so only the owner and/or the superusers can change that object.

This isn’t as easy as it appears. Django does have a branch called [per-object-permissions](http://code.djangoproject.com/wiki/RowLevelPermissions) that would do it, but it hasn’t been merged with the trunk for a while and I found it too late — I was already writing stuff for Django 0.96! So I had to come up with my own method. But, when that branch is merged, I would like to use it. I would do the merge myself, but it wouldn’t be an easy process and I have other things to do now.

I took some good, hard looks at the admin views and decided that I needed to either rewrite these views or intercept them before going to those functions. The first one that I wrote, the one that displays all the objects a user has permissions to change. This came from `change_list` view in `admin/views/main`

from django.contrib.admin.views.main import ChangeList,IncorrectLookupParameters
def my_objects(request):

app_label = ‘myapp’
model_name = ‘myobject’
model = models.get_model(app_label, model_name)
if model is None:
raise Http404(“App %r, model %r, not found” % (app_label, model_name))
if not request.user.has_perm(app_label + ‘.’ + model._meta.get_change_permission()):
raise PermissionDenied
try:
cl = ChangeList(request, model)

if not request.user.is_superuser:

owner_objs=list(MyObject.objects.filter(owner=request.user))
these_objs = cl.result_list

new_objs = []
for f in owner_objs:
if f in these_objs:
new_objs.append(f)

cl.result_list=new_objs
cl.result_count = len(cl.result_list)

except IncorrectLookupParameters:
if ERROR_FLAG in request.GET.keys():
return render_to_response(‘admin/invalid_setup.html’, {‘title’: _(‘Database error’)})
return HttpResponseRedirect(request.path + ‘?’ + ERROR_FLAG + ‘=1′)

c = template.RequestContext(request, {
‘title’: cl.title,
‘is_popup’: cl.is_popup,
‘cl’: cl,
})
c.update({‘has_add_permission’: c['perms'][app_label][cl.opts.get_add_permission()]}),

return render_to_response(['admin/%s/%s/change_list.html' % (app_label, cl.opts.object_name.lower()),
'admin/%s/change_list.html' % app_label,
'admin/change_list.html'],
context_instance=c)

I’m sure that this could be better — in fact, I’m not sure I had to rewrite the whole thing. I could have just intercepted the objects before making the list, somehow. And I know that the queries could have been better. But this works well.

The view that displays the individual object for editing is much easier. In this function, I simply intercepted the call, checked the permissions, and if they were okay, the view sends them to the proper view, which is `admin.views.main.change_stage`. I like how this one turned out over the view above:

from django.contrib.admin.views.main import change_stage

def edit_object(request,object_id):

if not request.user.is_superuser:
objects=list(MyObject.objects.filter(owner=request.user))
this_obj = MyObject.objects.get(id=object_id)

if not this_obj in objects:
raise PermissionDenied

return change_stage(request,’myapp’,'myobject’,object_id)

After doing all this, you have to change your `urls.py` to go to your new views instead of the standard admin views. So I added these in my `urlpatterns` object there:

(r’^admin/myapp/myobject/$’, ‘myapp.views.my_objects’),
(r’^admin/myapp/myobject/(?P\d+)/$’, ‘myapp.views.edit_object’),

So that’s it — this is what my app will be using until the Per-Object-Permissions gets merged (or a recent trunk gets merged to that branch). I think that solution is better.

Finally . . . generators in Jython

Okay, so I’m behind the Eight Ball. But I’m using Jython 2.2b1 and finally had a reason to use generators. Of course, we are back in 2.2-land, so our first line is this:

from __future__ import generators

Then here is my function. Generators are very nice for processes the result of a SQL query!

def getResults():
conn = getConnection()
cur = conn.cursor()
cur.execute("select * from table")
ret = cur.fetchone()
while ret!=None:
yield ret
ret = cur.fetchone()
cur.close()

Five Things I Hate About Python

brian d foy started this meme at the beginning of this month and many Pythoneers have joined in.  So it’s time I joined the party.

1. The Whitespace.  I love it and I hate it. I love it because it makes the code look clean and readable.  But if you are editing someone else’s scripts and they have different tab settings than you or they used a space and a tab, etc. it’s painful.  Emacs with python-mode makes this a little better, but it’s still horrible.
2. Lack of a automatic repository.  This has been said many times before but CPAN has it right.  I wish I could type a command and Python would find, download, and install a new module for me.  Eggs almost have it, but I still have to download something.
3. time vs datetime modules.  I wish I could use the datetime module for everything related to timestamps, but I can’t.  And I can’t get a lot done in the time module without a lot of work.  This madness must stop.  This wouldn’t bother me so much if I didn’t have to do so much with timestamps.
4. File name handling.  If you stick with the standard library you need to be familiar with  os.path, glob, os, shutil and sys modules to really work with file names. And don’t get me started on os.path.walk. Jason Orendorff’s glorious path module solves these problems.  So why isn’t it in the standard library yet?
5. if __name__=='__main__'.  Who made that up?  Does anyone still think it’s a good idea?  I have a template setup that automatically puts that in a new Python file (along with a few imports) but it just looks weird.