Planet OCLUG

So, at work I’m using Django quite a bit, and I ran into a problem where I need the database transaction to be committed, and then I need to trigger additional server-side code that will act on that data.

Putting all of this into the view function with a manually managed transaction sucks, far too much code. There’s transaction middleware, but by then your view function has returned. What to do?

Simple. I added my own middleware, and I return a new property that I tag into the HttpResponse object. Python is flexible enough to allow this hack.

MIDDLEWARE_CLASSES = (
    'teleworker.lib.middleware.MslEventMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.transaction.TransactionMiddleware',

So in my MslEventMiddleware, I look for a new property in the response, and if it’s present, I execute the requested command, which will happen after the TransactionMiddleware has called commit.

    def process_response(self, request, response):
        if hasattr(response, 'mslevent'):
           msl.event(response.mslevent)

Simple enough. Although a real post-processing API in Django would be helpful.

In any web application user data must be translated from HTML form data to native types and database types, and back again. Django web applcations are no different.

The "right way" to handle custom types is to extend Django's widgets, form fields and model fields. However, understanding exactly how these types perform each step of the conversion can be confusing. This post will attempt to explain how the data is converted at each stage and offer some advice about creating custom widgets, form fields and model fields.

This article is based on Django 1.3 and assumes the reader has experience creating and using Django forms, models and validation.

The third iteration the Arevco Lighting web site is now up.

The old site was simple HTML and images generated from a script, which I quite liked but wasn't the easiest thing for other people to update. The new site has been professionally skinned and is now running a Django CMS with some custom index code for product pages. I used the following:

• Django
• Gerbi (nee django-page-cms)
• Filebrowser no-grappelli fork
• TinyMCE

I've used Gerbi now for a few public web sites. It's a well designed and usable CMS that is quite easy to extend. It also has good multilingual support that will make translating content as easy as editing the pages.

I found this very helpful. It worked great with Django 1.3, PostgreSQL 8.3 in 2011/06:

And the penultimate step (filling in the primary key in existing rows) took around half a minute for almost 200,000 records on a not-particularly well-endowed laptop.

Summary:

CREATE SEQUENCE rcvh7_id_seq;
ALTER TABLE rcvh7 ADD id INT UNIQUE;
ALTER TABLE rcvh7 ALTER COLUMN id SET DEFAULT NEXTVAL('rcvh7_id_seq');
UPDATE rcvh7 SET id = NEXTVAL('rcvh7_id_seq');
ALTER TABLE rcvh7 ALTER COLUMN id SET NOT NULL;

Ottawa Python Authors Group meeting tomorrow Thursday May 26 at 8p.m. Best of all it's not me talking this time!

Hope you can make it out.

I was looking at django sessions and was a bit confused until I read the source code.

django/contrib/sessions/models.py
django/contrib/sessions/backends/base.py
django/contrib/sessions/backends/db.py
django/contrib/sessions/backends/cache.py
django/contrib/sessions/backends/cached_db.py
django/contrib/sessions/backends/file.py

You might think the interesting file to look at is django/contrib/sessions/models.py, but really the “toplevel” session object is defined in base.py. The object in base.py SessionBase. It is a base class for the various session implementations.

If you’re using a database-backed SessionStore, then you’ll be using base.py, db.py and session.py. db.py uses session.py for the database model and database interaction. In a view, the session object in the request object that is supplied by the session middleware is actually a SessionStore object. That is the object that has the methods get_expiry_age, etc.

I wanted to get some info out of every session in a batch mode including expiry age, so I needed to traverse SessionStore.objects.all(), not Session.objects.all().

I was looking at django sessions and was a bit confused until I read the source code.

django/contrib/sessions/models.py
django/contrib/sessions/backends/base.py
django/contrib/sessions/backends/db.py
django/contrib/sessions/backends/cache.py
django/contrib/sessions/backends/cached_db.py
django/contrib/sessions/backends/file.py

You might think the interesting file to look at is django/contrib/sessions/models.py, but really the “toplevel” session object is defined in base.py. The object in base.py SessionBase. It is a base class for the various session implementations.

If you’re using a database-backed SessionStore, then you’ll be using base.py, db.py and session.py. db.py uses session.py for the database model and database interaction. In a view, the session object in the request object that is supplied by the session middleware is actually a SessionStore object. That is the object that has the methods get_expiry_age, etc.

I wanted to get some info out of every session in a batch mode including expiry age, so I needed to traverse SessionStore.objects.all(), not Session.objects.all().

If you write any custom Django widgets or admin list_display callable functions you have probably run into this: Everything looks ok, except the place where your widget should be is just blank. Nothing. No traceback or any clue as to what went wrong.

It seems that Django suppresses all the exceptions sent by widgets rendering except for AssertionError and TypeError. Debugging under those conditions is tricky, so I wrote a function decorator to help. Just import this and put @assert_on_exception before your render method or admin list_display callable function:

def assert_on_exception(fn):
    import sys
    def wrap(*args, **kwargs):
        try:
            return fn(*args, **kwargs)
        except (AssertionError, TypeError):
            raise
        except:
            assert 0, sys.exc_info()[0].__name__ + ": " + str(sys.exc_info()[1])
    wrap.__name__ = fn.__name__
    wrap.__dict__.update(fn.__dict__)
    wrap.__doc__ = fn.__doc__
    wrap.__module__ = fn.__module__
    return wrap

The Ottawa Canada Linux Users Group board of directors has decided to retire their Django meeting announcement web site and replace it with a plain wiki.

But, it's been a great run for close to 4 years:

• 69 meetings posted, always up-to-date
• 65 speakers
• 34 local Linux jobs posted
• hundreds of members' blog posts aggregated
• zero maintenance time (removal of old meetings & jobs is automatic)
• zero reported defects
• zero down time

This is the text from the Nontrivial Django Forms talk I gave last Tuesday at the April meeting of the Ottawa Python Authors Group.

This talk starts where the Django forms documentation leaves off.

So, I’ve been trying to solve an issue in some Django code at work where Django’s insistence on cascading deletions across foreign keys (not necessarily a bad thing) has resulted in unintended deletion.

I have a model, Client. Client has a ForeignKey to Zone. When Zone is deleted I would like to intercept and move any Clients in that Zone to another Zone (the default Zone can’t be deleted). To do this I
registered a pre_delete signal handler to listen to deletions of Zones. Unfortunately it doesn’t work.

It seems that Django cascades the deletion from Zone to Client without yet signaling pre_delete on the Zone, so the signal comes in way too late to move the Clients. I could be wrong, and I am often, but that feels like a design oversight to me. A pre_delete hook should be capable of preventing unintended deletion, no?

So, I’m back to overriding delete() in the Zone model. Too bad, as that has a tendency to introduce circular imports in my models, which I’m trying to avoid.

Django's forms have fields like models, but you access them in a completely different way. I've found it hard to remember the correct way to access form field data properly, so this is a summary of the things I need to do with forms and how to do them. In particular these are useful in the form's __init__ method after calling super() to set up the form the way you want to.

This is a video of the Django 1.1 talk I gave last night for the Ottawa Python Authors Group. We were bumped out of our reserved room so I did the talk in the pub. The timing worked out quite well, I just finished answering questions as game 6 was starting in the background.

Thank you to Andrew Ross and FOSSLC for doing the recording and setting everything up.

You will need Flash 10 to hear the audio for this presentation. If you don't like flash, please send a polite note to the ePresence people asking for them to support other formats.

I will be giving my Django 1.1 talk at The Ottawa Python Authors Group meeting on Tuesday.

UPDATE: A video of my talk is now available

This is the text from the Django 1.1 talk I gave on Friday May 15 at Algonquin College for FOSSLC's Geocamp/Summercamp 2009.

I have tried to format this in a way that is well suited to skimming and easier to access from the web than reading the original slides. If you find this useful, please let me know.

I will be giving an introductory Django talk at The Free and Open Source Software Learning Centre (FOSSLC) Summercamp/Geocamp 2009 Conference. The conference is taking place on May 13, 14, and 15 at Algonquin College, Ottawa, Ontario, Canada, and early bird pricing is available until April 5.

At work I am using Django for a web-based management interface, using PostgreSQL as the database back-end.

Django does not yet come with a migration framework to evolve the database schema, so I wrote a really simple one, based on Rails migrations.

The system is simple. You put fragments of SQL code in separate text files at a predetermined location on disk, and prefix each one of them with a number, representing the version of the database schema. Your migration code also maintains a schema table in the database to keep track of the current versions, and whether the last migration attempt was successful.

So, on upgrade, if the schema version is 30 and the fragments on disk to up to 43, then you run 31-43, wrapping each in a transaction, and rolling-back at the first failure. So, if 41 fails, you’ll get as far as version 40, store an error, and you’re done.

Sounds ok, right? Well, if this is for a single database instance, sure. Unfortunately for this scheme, the product that I work on has roughly 6000 instances in the field in servers all over the world, with less than half running the most recent release. So, stream management becomes an issue. And stream management is something that I find that most of these modern frameworks overlook.

What if a bugfix that I just made on the HEAD has an associated schema change, and I want to backport that fix to the previous maintenance release? If the last schema version of the previous release is, say, 16, and I just added migration fragment 62, then we have a problem. And this is it.

Every migration fragment is dependent on the success of the previous one.

So, I can’t just backport fragment 62, I’d have to backport 17 - 62. Yikes.

The solution is actually simple, and it’s something that the SMEServer’s native databases do already. Each migration fragment is not raw SQL, it’s code, in this case Perl, but it could be anything. So, instead of blinding executing the migration fragments in order, you blinding execute the migration code fragments and let each and every one determine whether they need to do their particular job.

Need to make a varchar(512) a varchar(1024)? No problem, just check it’s size now, and if it’s 1024 then you don’t need to do anything. Now each fragment doesn’t depend on the one that came before it, and you can safely backport only what you wanted to backport.

So how do we know what the database, in this case a relational database like PostgreSQL, looks like now? As it turns out, the standard does have some support for that, and it’s in the information_schema.

This would fetch all of the column names from a table called “clients”:

select column_name from information_schema.columns
     where table_name = 'clients';

And this would determine the current length of the character field in that table called “name”:

select character_maximum_length
    from information_schema.columns
    where table_name = 'clients'
    and column_name = 'name';

So, an ideal migration framework would provide this information to the migration fragments, to keep their job simple. Migration isn’t done constantly so we’re not that concerned with performance. Keep the code simple and bug-free.

Anyone feel like writing it? I suspect I may have to.

The Ottawa Python Authors Group is having a meeting tomorrow (Monday) and Michael Soulier will be presenting talks on concurrency in Django, and an Introduction to Git. It has been a while since the last meeting and this promises to be a good one.