return to OCLUG Web Site
A Django site.
June 23, 2013

Brenda Butler
bjb
linuxbutler
» python's setuptools

One of the nice things about the Ottawa Python Author’s Group irc channel (oftc.net, #opag) is that they occasionally mention a great but under-advertized reference, like this one for setuptools:

http://peak.telecommunity.com/DevCenter/setuptools#basic-use

Thanks Ian!

June 19, 2013

Ian Ward
excess
excess.org - News
» Sean Zicari's Curses and Urwid Talk

Sean Zicari gave a Curses and Urwid presentation at the recent PyCon US in Santa Clara.


» data.gc.ca

A site I've been working on for the past few months has just launched. data.gc.ca "2.0" is a completely rewritten version of the Open Data portal for the Government of Canada.

I worked on the CKAN portion of the site responsible for the data catalogue. CKAN is an open source Data Portal application written in Python. I built a fully bilingual schema with many additional fields for our datasets. I also optimized, fixed and improved many parts of CKAN itself.

Our team built templates based on the Web Experience Toolkit (WET). WET provides building blocks for creating accessible, mobile-friendly web sites. These templates include a map widget for our geospatial datasets and an interactive data table for previews of tabular data.

All of our work is available on the open-data github page and is released under a permissive license.

data.gc.ca might just be the largest, most fully bilingual and most accessible CKAN site anywhere. More to come!

June 5, 2013

Michael P. Soulier
msoulier
But I Digress
» Matplotlib is really cool

Recently at work I needed to produce a pie chart for some data that I had to crawl through using basic scripting. Pulling out the numbers, sorting, filtering, etc., was all tasks that I'm used to doing, but for graphing I normally use Gnuplot. But, Gnuplot can't do pie charts AFAIK, so I needed to look for another way.

I didn't want to use a spreadsheet for the task because I don't find them very scriptable. So, I turned to matplotlib, Python's uber-graphing library.

I found a simple example online, and before I knew it, I had a nice looking pie chart, with very little code.

# make a square figure and axes
figure(1, figsize=(6,6))
ax = axes([0.1, 0.1, 0.8, 0.8])

labels = []
fracs = []

for country in countries:
    count = countries[country]
    labels.append(country)
    fracs.append(count)

pie(fracs, labels=labels, autopct='%1.1f%%', shadow=True)

title('By Country', bbox={'facecolor':'0.8', 'pad':5})

show()
http://www.but-i-digress.ca/static/images/piechart.png

Very impressive. I tried it a few years ago and found it difficult, but it seems to have gotten much simpler. I must delve deeper now.

March 15, 2013

Ian Ward
excess
excess.org - News
» Painting with Braille

This is something I've been wanting to write for a while.

Unicode page U+2800 has all the combinations of a 2x4 grid of Braille dots. Braille dots that line up neatly with the ones on all sides in most fonts. We can paint with this!

February 26, 2013

Ian Ward
excess
excess.org - News
» Iterables, Iterators and Generators: Part 2

This is the second part of the talk I gave January 24, 2013 at the Ottawa Python Authors Group.

Part One introduces Python iterables and iterators and generators. This part covers the advanced use of generators while building an interactive two-player network game.

February 12, 2013

Ian Ward
excess
excess.org - News
» Iterables, Iterators and Generators: Part 1

This is part one of a talk I gave January 24, 2013 at the Ottawa Python Authors Group

Part Two is now also available.

Both parts of this presentation are also available as a single IPython Notebook which you can download and run locally, or view with nbviewer.ipython.org. The complete source is available at https://github.com/wardi/iterables-iterators-generators

December 24, 2012

Michael P. Soulier
msoulier
But I Digress
» Checking the weather

A while back I needed a way to check the weather forecast, simply, from a terminal, as is the preference of most Unix geeks like me. Being a Canadian, I'm not interested in the Weather Channel as much as the Environment Canada data. Thankfully, they do publish an RSS feed, and good for them.

I'm interested in this one: http://www.weatheroffice.gc.ca/rss/city/on-118_e.xml. So, to check the weather, I need to pull the feed and parse it. Pulling a page in Python is as simple as using urllib. From there, I can walk the elements I want like so:

import urllib
from xml.etree.ElementTree import parse

for elem in parse(urllib.urlopen(rssfeed)).findall('channel/item/title'):

Now, I wanted the option of picking a certain number of lines, and wrapping at a certain number of columns. I wanted this for using the script output as input into other apps, like Conky. Skipping lines is easy, intelligently wrapping them is not. Luckily, Python has a textwrap module in the standard library.

You use it like this:

import textwrap

wrapper = textwrap.TextWrapper(width=options.wrap, subsequent_indent="    ")
lines = wrapper.wrap(s)
for line in lines:
    print line

Put together, it's really that simple. I think the majority of my code is option parsing. The core loop is just this:

options = parse_options()
wrapper = textwrap.TextWrapper(width=options.wrap, subsequent_indent="    ")
count = 0
for elem in parse(urllib.urlopen(rssfeed)).findall('channel/item/title'):
    s = elem.text.encode('utf8', 'ignore')
    lines = wrapper.wrap(s)
    for line in lines:
        count += 1
        if options.lines and count > options.lines:
            break
        else:
            print line
    else:
        continue
    break

The whole thing is here. Sample output looks like this:

No watches or warnings in effect, Ottawa (Kanata - Orléans)
Current Conditions: Light Snow, -11.1°C
Sunday night: A few flurries. Low minus 18.
Monday: Sunny. High minus 9.
Monday night: Increasing cloudiness. Low minus 12.
Tuesday: Chance of flurries. High minus 7. POP 60%
Wednesday: A mix of sun and cloud. Low minus 16. High minus 8.
Thursday: Periods of snow. Low minus 8. High minus 2.
Friday: Sunny. Low minus 12. High minus 7.
Saturday: Periods of snow. Low minus 12. High minus 7.

I love building my own tools like this, it's the ultimate in end-user computing. Unix and Python are my playground.

December 21, 2012

Michael P. Soulier
msoulier
But I Digress
» Signatures in OAuth

Hi again. I already went into the basics of OAuth in a previous post, and alluded to the signatures being the hard part of the implementation. I'm using a python-oauth module for the client, but for the server, I decided to implement my own in Perl and Mojolicious, since I had problems with Net::OAuth that I deemed a protocol violation. Besides, I learn more this way.

There are multiple signature methods supported in OAuth 1.0, specifically three.

  1. PLAINTEXT
  2. HMAC-SHA1
  3. RSA-SHA1

Unsurprisingly, PLAINTEXT is the simplest. In this case, the signature is a simple combination of the consumer secret and the token secret, if there is any yet, with a slight twist. To ensure that the characters are treated properly, they must first be "normalized". This means decoding them as you would any percent-encoded string, and then...

  • convert them to UTF-8

  • escape them with percent-encoding from RFC 3986 like so

    • these characters are not encoded: alphanumerics, "-", ".", "_", "~"
    • all other characters must be encoded
    • any hexidecimal used in encoding must be upper-case

I found that hard to read, but I did find this bit of Perl to solve the problem in Net::OAuth, which I converted to a Mojolicious helper.

helper encode => sub {
    my $self = shift;
    my $str = shift;
    $str = "" unless defined $str;
    unless($SKIP_UTF8_DOUBLE_ENCODE_CHECK) {
        if ($str =~ /[\x80-\xFF]/ and !utf8::is_utf8($str)) {
            warn("your OAuth message appears to contain some "
                . "multi-byte characters that need to be decoded "
                . "via Encode.pm or a PerlIO layer first. "
                . "This may result in an incorrect signature.");
        }
    }
    return
    URI::Escape::uri_escape_utf8($str,'^\w.~-');
};

Decoding is even simpler.

helper decode => sub {
    my $self = shift;
    my $str = shift;
    return uri_unescape($str);
};

In Python, the same code looks like so.

import urllib

def escape(s):
    """Escape a URL including any /."""
    return urllib.quote(s, safe='~')

def _utf8_str(s):
    """Convert unicode to utf-8."""
    if isinstance(s, unicode):
        return s.encode("utf-8")
    else:
        return str(s)

escaped = escape(_utf8_str(s))

# Unquoting is just
unquoted = urllib.unquote(s)

So, this needs to be done to each of the parameters, in this case, the consumer key and the token secret, if any, and then they are combined with an ampersand. So, if your consumer key is, say, "myconsumerkey", and you don't have a token yet, then the initial PLAINTEXT signature is myconsumerkey&.

Now, this isn't too bad, but once you get into HMAC-SHA1 signatures, it gets a lot worse. The signature from the PLAINTEXT method becomes the key for the signature, and you'll already have the code for that now, but the raw input to the HMAC-SHA1 algorithm is a base string, that is rather difficult to construct. The input is the HTTP method, the request URI, both normalized like above and contatenated with an ampersand. Then, this will be contatenated with an ampersand to all of the input parameters in the request, constructed in a particular way.

  1. Take all input parameter names and values from all sources, and normalize them like above. (but skip the oauth_signature parameter)
  2. Sort all parameters by the normalized parameter name.
  3. Pair the names and values, contatenated with an "=".
  4. Concatenate all pairs with ampersands in the sorted order.
  5. Escape the entire string using the method above.

This is the base string to the HMAC-SHA1 algorithm, along with the key we mentioned. The final signature should match what the client generated. Oh, and if you're running your service on a nonstandard port (80 or 443), then you must include the port in the URI.

Example:

A call to http://localhost/initiate on port 80 or 443, a GET request, with the following params:

{'oauth_nonce': '21823552', 'oauth_timestamp': 1356129798,
 'oauth_consumer_key': 'Mitel test', 'oauth_signature_method': 'HMAC-SHA1',
 'oauth_version': '1.0', 'oauth_signature': 'pevzNqSnJ8QtqFUDWVlYhVRp8D0=',
 'oauth_callback': 'oob'}

The base string would look like this:

GET&http%3A%2F%2Flocalhost%2Finitiate&oauth_callback%3Doob%26oauth_consumer_key%3DMitel%2520test%26oauth_nonce%3D21823552%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1356129798%26oauth_version%3D1.0

with a key of:

mitelsharedsecret&

and a final signature of:

pevzNqSnJ8QtqFUDWVlYhVRp8D0=

Oddly, if I used the b64encode method in Digest::HMAC_SHA1, a final "=" sign is missing on the final result, so I had to pull in MIME::Base64 and do this instead:

my $sig = encode_base64($hmac->digest);

The equivalent Python in the oauth library does this:

import binascii

# HMAC object.
try:
    import hashlib # 2.5
    hashed = hmac.new(key, raw, hashlib.sha1)
except:
    import sha # Deprecated
    hashed = hmac.new(key, raw, sha)

# Calculate the digest base 64.
return binascii.b2a_base64(hashed.digest())[:-1]

That just leaves RSA-SHA1, but that requires a pre-existing SSL relationship with the server, using SSL certificates. As such, I'm not worrying about it just yet. I don't think it'll be used much.

I'll need to do some interop testing with a few different clients, I'm hoping that they're not all snowflakes. The point of the rigid nature of the base string construction is that the final product is supposed to be reproducable.

The base string construction is definitely the hardest part, and I've read that the signatures were dropped in OAuth 2.0 because they were too hard to do. I'd rather not drop the added security, and while they're a pain, there are sample implementations to follow. I think that OAuth 1.0 is a better choice. And it's, like, finished.

November 25, 2012

Michael P. Soulier
msoulier
But I Digress
» Tweeting on OS X

For some time my desktop has been an older, underpowered Linux box, so using services like Twitter via the website has been painful, as loaded with heavy javascript that would consume my CPU's limited resources.

So, like any Python programmer would do, I wrote my own client. Well, lets be honest, the hard work was done by the tweepy library, I'm just using it. I call it Twit, and I've been using it to post for some time now, lightweight from the command-line. I also keep one running, polling my account for new posts, and notifying me when there are new posts and pulling them down.

My notifications come in many forms, depending on the command-line options. I can just watch them show up in text in the shell, I can use xosd to display notifications on your X11 desktop, It can use libnotify in Gnome to display temporary notifications that drop down in the corner of the screen.

As I'm now playing with an OS X desktop, libnotify isn't supported. I could use xosd along with XQuartz, but I'd prefer more native integration with OS X, as it does have a notification system, via AppKit. Python is preinstalled on OS X, and the AppKit module is included. I found a great example of how to use it online, which made this much, much simpler, so thanks there.

I used it like so:

def notify_appkit(status, options):
    """Thanks to
    https://github.com/albertz/music-player/blob/master/notifications.py
    for how to do this."""
    global notifCenter
    if not notifCenter:
        import AppKit
        notifCenter = \
            AppKit.NSUserNotificationCenter.defaultUserNotificationCenter()
        appDelegate = AppKit.NSApplication.sharedApplication().delegate()
        notifCenter.setDelegate_(appDelegate)

    notif = AppKit.NSUserNotification.alloc().init()
    title = "Tweet by %s" % status.user.name
    notif.setTitle_(title)
    notif.setInformativeText_(status.text)
    notifCenter.deliverNotification_(notif)

Now, this does cause a little icon on the dock to jump up and down, so I'll need to look into that, and how to open the browser to twitter or the embedded link when the notification is clicked on, but it's a good start.

November 21, 2012

Michael P. Soulier
msoulier
But I Digress
» Suspend on lid close in Debian Squeeze

I recently decided that Gnome is not the best desktop for my little EeePC netbook with a little 10.5" screen. So I'm playing around with a window manager that mainly just maximizes everything. I've tried Ratpoison, I've tried wmii, and now I'm trying Awesome.

I have a lot of customizations to do, but one thing that was missing was a way to suspend the netbook when the laptop lid is closed. I could manually run acpitool -s in a shell, or pm-suspend, but it's best handled by DBus, as intended.

Now, a simple way to have DBus do the work is using dbus-send, like so:

dbus-send --print-reply \
          --system \
          --dest=org.freedesktop.UPower \
          /org/freedesktop/UPower \
          org.freedesktop.UPower.Suspend

The hard part is subscribing to the lid close event, so I'm not polling all the time, exactly what DBus was written to prevent. I had a Python script for this, but the API was changed in Squeeze to use the UPower daemon and API.

I had to do some poking around to figure out how to update it, but I just got it working, so I thought I'd share.

#!/usr/bin/python

import dbus, gobject, sys
from dbus.mainloop.glib import DBusGMainLoop

pow_prop_iface = None
pow_iface = None

def handle_lidclose(*args):
    closed = pow_prop_iface.Get('',
                                'LidIsClosed')
    if closed:
        print "lid is closed, suspending"
        pow_iface.Suspend()
    else:
        print "lid is open"

def main():
    global pow_prop_iface, pow_iface

    DBusGMainLoop(set_as_default=True)

    bus = dbus.SystemBus()

    power_proxy = bus.get_object('org.freedesktop.UPower',
                                '/org/freedesktop/UPower')

    pow_prop_iface = dbus.Interface(power_proxy,
                                    'org.freedesktop.DBus.Properties')
    pow_iface = dbus.Interface(power_proxy,
                               'org.freedesktop.UPower')

    print "Registering a signal receiver for upower events..."

    bus.add_signal_receiver(handle_lidclose,
                            dbus_interface="org.freedesktop.UPower",
                            signal_name="Changed")


    loop = gobject.MainLoop()
    loop.run()

if __name__ == '__main__':
    main()

Now I just run it in the background from my .xsession script at X11 login, and it's sitting there waiting for any change in UPower status. Works like a charm.

November 16, 2012

Ian Ward
excess
excess.org - News
» Urwid 1.1.1 and 1.0.3 Released

Urwid maintenance releases 1.1.1 and 1.0.3 are now available. These releases contain only bug-fixes, see the Changelog for details.

November 14, 2012

Ian Ward
excess
excess.org - News
» Urwid Applications at PyCon Canada

I gave a 20-minute talk running through 7 great Urwid Applications at PyCon Canada in Toronto this past weekend.

The "Console Applications with Urwid" video is now available. Huge thanks to the conference organizers for great, first, PyCon Canada. I look forward to the next one.

The programs I covered were:

November 1, 2012

Ian Ward
excess
excess.org - News
» Python Container Literals

This post covers some basic Python syntax that tends to trip up people just starting with the language.

Literal tuple, list, dict and set definitions have some edge cases you need to be aware of when reading and writing Python code. Unfortunately some of these cases aren't consistent or obvious, but once you understand why they exist, they are easy to remember.

October 23, 2012

Ian Ward
excess
excess.org - News
» Urwid 1.1.0 Released - Containers & Documentation

August 6, 2012

Michael P. Soulier
msoulier
But I Digress
» Custom request headers in Python

Just to be fair, here’s the same example in Python, batteries included.

#!/usr/bin/python

import httplib

conn = httplib.HTTPConnection('digitaltorque.ca')
params = {}
headers = { 'X-Bender': 'Bite my shiny metal ass!' }

conn.request("GET", "/", params, headers)

response = conn.getresponse()

print response.read()

Not too painful.


July 13, 2012

Ian Ward
excess
excess.org - News
» Urwid 1.0.2 Released

Urwid maintenance release 1.0.2 is now available.

May 10, 2012

Michael P. Soulier
msoulier
But I Digress
» Post-transaction events in Django

So, at work I’m using Django quite a bit, and I ran into a problem where I need the database transaction to be committed, and then I need to trigger additional server-side code that will act on that data.

Putting all of this into the view function with a manually managed transaction sucks, far too much code. There’s transaction middleware, but by then your view function has returned. What to do?

Simple. I added my own middleware, and I return a new property that I tag into the HttpResponse object. Python is flexible enough to allow this hack.

MIDDLEWARE_CLASSES = (
    'teleworker.lib.middleware.MslEventMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.transaction.TransactionMiddleware',

So in my MslEventMiddleware, I look for a new property in the response, and if it’s present, I execute the requested command, which will happen after the TransactionMiddleware has called commit.

    def process_response(self, request, response):
        if hasattr(response, 'mslevent'):
           msl.event(response.mslevent)

Simple enough. Although a real post-processing API in Django would be helpful.


April 26, 2012

Ian Ward
excess
excess.org - News
» Gerbi CMS

Gerbi CMS (nee django-page-cms) is a multilingual content management system written in Python and based on the Django web framework. It's currently my favourite CMS software and use it for a number of web sites I administer.

I'll be giving a talk about Gerbi CMS at the next OCLUG and OPAG meetings resembling this article.

April 9, 2012

Ian Ward
excess
excess.org - News
» Paranoid Django Templates

If you've ever wanted to know if a Django template is using a variable it shouldn't be, or not using a variable it should, this code will make both cases fail loudly. Django's default template behaviour is to silently replace missing variables with an empty string, and ignore unused variables.

To use this code you can either:

  1. wrap your Context (or RequestContext) object in your view with a ParanoidContextProxy that will fail on any attempt to access a missing variable, or
  2. use the paranoid_render_to_response function (or similar) to also require that every variable you pass be used in the template.