return to OCLUG Web Site
A Django site.
June 5, 2013

Michael P. Soulier
msoulier
But I Digress
» Matplotlib is really cool

Recently at work I needed to produce a pie chart for some data that I had to crawl through using basic scripting. Pulling out the numbers, sorting, filtering, etc., was all tasks that I'm used to doing, but for graphing I normally use Gnuplot. But, Gnuplot can't do pie charts AFAIK, so I needed to look for another way.

I didn't want to use a spreadsheet for the task because I don't find them very scriptable. So, I turned to matplotlib, Python's uber-graphing library.

I found a simple example online, and before I knew it, I had a nice looking pie chart, with very little code.

# make a square figure and axes
figure(1, figsize=(6,6))
ax = axes([0.1, 0.1, 0.8, 0.8])

labels = []
fracs = []

for country in countries:
    count = countries[country]
    labels.append(country)
    fracs.append(count)

pie(fracs, labels=labels, autopct='%1.1f%%', shadow=True)

title('By Country', bbox={'facecolor':'0.8', 'pad':5})

show()
http://www.but-i-digress.ca/static/images/piechart.png

Very impressive. I tried it a few years ago and found it difficult, but it seems to have gotten much simpler. I must delve deeper now.

» Java, usable but pointlessly difficult

I've been working on a REST client in Java at work lately. I know, it scares me too. I've had to revisit much of my old Java knowledge, and pick up a lot of new chops in Swing, Eclipse, and author a decent JSON-parsing HTTPS client.

Our internal servers use self-signed SSL certs at work, so to talk to them I had to disable any host-certificate checking. Should be a simple boolean in the API right? Wrong. I forgot. Java.

First, you need to install your own trust manager.

import javax.net.ssl.X509TrustManager;
import java.security.cert.X509Certificate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.HttpsURLConnection;

 TrustManager[] trustAllCerts = new TrustManager[] {
         new X509TrustManager() {
     @Override
     public java.security.cert.X509Certificate[] getAcceptedIssuers() {
         return null;
     }
     @Override
     public void checkClientTrusted(X509Certificate[] certs, String authType) {
     }
     @Override
     public void checkServerTrusted(X509Certificate[] certs, String authType) {
     }
 }};
 // Install this trust manager.
 try {
     SSLContext sc = SSLContext.getInstance("SSL");
     sc.init(null, trustAllCerts, new java.security.SecureRandom());
     HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
 }

Then, you need to install a hostname verifier that accepts a bad host, as that's likely wrong too, especially if you access the box by IP address when the IP isn't in the cert.

import javax.net.ssl.HostnameVerifier;

 HttpsURLConnection sconnection = (HttpsURLConnection)request.unwrap();
 sconnection.setHostnameVerifier(new HostnameVerifier() {
     @Override
     public boolean verify(String hostname, SSLSession session) {
         return true;
     }
 });

Usable? Yes. But it's no wonder that Java programmers insist on IDEs when it's so damn wordy. I mean, wouldn't a simple boolean have done here?

March 12, 2013

Michael P. Soulier
msoulier
But I Digress
» Joysticks on Mac OS X in Flightgear

I like flight simulators. I've played all of the good ones, Flightgear is an open-source flight simulator that isn't bad, and is steadily getting better with every release. While it is available to run on Mac, there's no way to calibrate the Joystick due to a lack of caring on Apple's part.

But, Flightgear is hackable. The joystick file is just a set of XML bindings, those bindings can run code, so using coefficients to the input and controlling the output, in theory I can calibrate the settings myself. So, I did:

<?xml version="1.0"?>

<PropertyList>
    <axis n="0">
        <desc>Aileron</desc>
        <binding>
            <command>nasal</command>
            <script>
                var value = cmdarg().getNode("setting").getValue();
                if (abs(value) &lt; 0.05) {
                    value = 0;
                }
                else {
                    value -= 0.15;
                    value *= 1.5;
                }
                setprop("/controls/flight/aileron", value);
            </script>
        </binding>
    </axis>
    <axis n="1">
        <desc>Elevator</desc>
        <binding>
            <command>nasal</command>
            <script>
                var value = cmdarg().getNode("setting").getValue();
                if (abs(value) &lt; 0.05) {
                    value = 0;
                }
                else {
                    value -= 0.1;
                    value *= -1.5;
                }
                setprop("/controls/flight/elevator", value);
            </script>
        </binding>
    </axis>
    <axis n="2">
        <desc>Rudder</desc>
        <binding>
            <command>nasal</command>
            <script>
                var value = cmdarg().getNode("setting").getValue();
                if (abs(value) &lt; 0.05) {
                    value = 0;
                }
                else {
                    value *= 1.5;
                }
                setprop("/controls/flight/rudder", value);
            </script>
        </binding>
    </axis>
    <axis n="3">
        <desc>Throttle</desc>
        <binding>
            <command>nasal</command>
            <script>
                var value = cmdarg().getNode("setting").getValue();
                value -= 0.5;
                value *= -1.5;
                setprop("/controls/engines/engine/throttle", value);
            </script>
        </binding>
    </axis>

<axis n="4">
    <desc>View Direction</desc>
    <direction>left</direction>
    <low>
    <repeatable>true</repeatable>
    <binding>
        <command>nasal</command>
        <script>view.panViewDir(1)</script>
    </binding>
    </low>
    <high>
    <repeatable>true</repeatable>
    <binding>
        <command>nasal</command>
        <script>view.panViewDir(-1)</script>
    </binding>
    </high>
    <dead-band type="double">0</dead-band>
    <binding>
    <factor type="double">-1</factor>
    </binding>
</axis>
<axis n="5">
    <desc>View Elevation</desc>
    <direction>upward</direction>
    <low>
    <repeatable>true</repeatable>
    <binding>
        <command>nasal</command>
        <script>view.panViewPitch(1)</script>
    </binding>
    </low>
    <high>
    <repeatable>true</repeatable>
    <binding>
        <command>nasal</command>
        <script>view.panViewPitch(-1)</script>
    </binding>
    </high>
    <dead-band type="double">0</dead-band>
    <binding>
    <factor type="double">-1</factor>
    </binding>
</axis>
<button>
    <desc>Brakes</desc>
    <binding>
    <command>nasal</command>
    <script>controls.applyBrakes(1)</script>
    </binding>
    <mod-up>
    <binding>
        <command>nasal</command>
        <script>controls.applyBrakes(0)</script>
    </binding>
    </mod-up>
</button>
<button n="3">
    <desc>Flaps Up</desc>
    <repeatable>false</repeatable>
    <binding>
    <command>nasal</command>
    <script>controls.flapsDown(-1)</script>
    </binding>
    <mod-up>
    <binding>
        <command>nasal</command>
        <script>controls.flapsDown(0)</script>
    </binding>
    </mod-up>
</button>
<button n="4">
    <desc>Flaps Down</desc>
    <repeatable>false</repeatable>
    <binding>
    <command>nasal</command>
    <script>controls.flapsDown(1)</script>
    </binding>
    <mod-up>
    <binding>
        <command>nasal</command>
        <script>controls.flapsDown(0)</script>
    </binding>
    </mod-up>
</button>
<button n="1">
    <desc>Elevator Trim Forward</desc>
    <repeatable>true</repeatable>
    <binding>
    <command>nasal</command>
    <script>controls.elevatorTrim(0.75)</script>
    </binding>
</button>
<button n="2">
    <desc>Elevator Trim Backward</desc>
    <repeatable>true</repeatable>
    <binding>
    <command>nasal</command>
    <script>controls.elevatorTrim(-0.75)</script>
    </binding>
</button>
<name type="string">WingMan Extreme Digital 3D</name>
</PropertyList>

Right now, the calibration is good, but the controls are really jerky and overly sensitive. I'll have to see if I can smooth them out. But, they work.

» Video on Raspbian

Currently our upstairs TV has a raspberry pi attached to it. It's running the basic Raspbian distribution, based on Debian Linux. I'm using it for development as well as for a media center at the moment, so I'm not using RaspBMC, I'm going to order another one has a dedicated media center, but for now I need to be able to play movies off of a usb stick.

I tried VLC, but the performance was horrible. Apparently it doesn't use the native GPU. I read about it online, and thanks to that finally found omxplayer, which does use the native GPU. Unfortunately, it's not all that polished when playing from the command-line. It leaves behind text in the terminal above and below the video at certain aspect ratios, which is visually distracting.

They haven't fixed it yet so I put a wrapper script around it to work around the issue. I plan to put a whole interface around it at some point just for fun, but for now at least I can watch movies without text in the black bars above and below:

#!/bin/sh

setterm -cursor off
clear
omxplayer -o hdmi $1
clear
setterm -cursor on

Simple, but it works.

January 1, 2013

Michael P. Soulier
msoulier
But I Digress
» Jim Butcher on Men vs. Women

I'm reading Jim Butcher's "Cold Days", his latest installment in the Dresden Files, and I felt the need to share this quote:

So, ladies, if you ever have some conversation with your boyfriend or husband or brother or male friend, and you are telling him something perfectly obvious, and he comes away from it utterly clueless? I know it's tempting to think to yourself, "The man can't possibly be that stupid!"

But yes. Yes, he can.

Our innate strengths just aren't the same. We are the mighty hunters, who are good at focusing on one thing at a time. For crying out loud, we have to turn down the radio in the car if we suspect we're lost and need to figure out how to get where we're going.

Umm, yeah, I've had to turn the radio down...

December 24, 2012

Michael P. Soulier
msoulier
But I Digress
» Checking the weather

A while back I needed a way to check the weather forecast, simply, from a terminal, as is the preference of most Unix geeks like me. Being a Canadian, I'm not interested in the Weather Channel as much as the Environment Canada data. Thankfully, they do publish an RSS feed, and good for them.

I'm interested in this one: http://www.weatheroffice.gc.ca/rss/city/on-118_e.xml. So, to check the weather, I need to pull the feed and parse it. Pulling a page in Python is as simple as using urllib. From there, I can walk the elements I want like so:

import urllib
from xml.etree.ElementTree import parse

for elem in parse(urllib.urlopen(rssfeed)).findall('channel/item/title'):

Now, I wanted the option of picking a certain number of lines, and wrapping at a certain number of columns. I wanted this for using the script output as input into other apps, like Conky. Skipping lines is easy, intelligently wrapping them is not. Luckily, Python has a textwrap module in the standard library.

You use it like this:

import textwrap

wrapper = textwrap.TextWrapper(width=options.wrap, subsequent_indent="    ")
lines = wrapper.wrap(s)
for line in lines:
    print line

Put together, it's really that simple. I think the majority of my code is option parsing. The core loop is just this:

options = parse_options()
wrapper = textwrap.TextWrapper(width=options.wrap, subsequent_indent="    ")
count = 0
for elem in parse(urllib.urlopen(rssfeed)).findall('channel/item/title'):
    s = elem.text.encode('utf8', 'ignore')
    lines = wrapper.wrap(s)
    for line in lines:
        count += 1
        if options.lines and count > options.lines:
            break
        else:
            print line
    else:
        continue
    break

The whole thing is here. Sample output looks like this:

No watches or warnings in effect, Ottawa (Kanata - Orléans)
Current Conditions: Light Snow, -11.1°C
Sunday night: A few flurries. Low minus 18.
Monday: Sunny. High minus 9.
Monday night: Increasing cloudiness. Low minus 12.
Tuesday: Chance of flurries. High minus 7. POP 60%
Wednesday: A mix of sun and cloud. Low minus 16. High minus 8.
Thursday: Periods of snow. Low minus 8. High minus 2.
Friday: Sunny. Low minus 12. High minus 7.
Saturday: Periods of snow. Low minus 12. High minus 7.

I love building my own tools like this, it's the ultimate in end-user computing. Unix and Python are my playground.

December 21, 2012

Michael P. Soulier
msoulier
But I Digress
» Signatures in OAuth

Hi again. I already went into the basics of OAuth in a previous post, and alluded to the signatures being the hard part of the implementation. I'm using a python-oauth module for the client, but for the server, I decided to implement my own in Perl and Mojolicious, since I had problems with Net::OAuth that I deemed a protocol violation. Besides, I learn more this way.

There are multiple signature methods supported in OAuth 1.0, specifically three.

  1. PLAINTEXT
  2. HMAC-SHA1
  3. RSA-SHA1

Unsurprisingly, PLAINTEXT is the simplest. In this case, the signature is a simple combination of the consumer secret and the token secret, if there is any yet, with a slight twist. To ensure that the characters are treated properly, they must first be "normalized". This means decoding them as you would any percent-encoded string, and then...

  • convert them to UTF-8

  • escape them with percent-encoding from RFC 3986 like so

    • these characters are not encoded: alphanumerics, "-", ".", "_", "~"
    • all other characters must be encoded
    • any hexidecimal used in encoding must be upper-case

I found that hard to read, but I did find this bit of Perl to solve the problem in Net::OAuth, which I converted to a Mojolicious helper.

helper encode => sub {
    my $self = shift;
    my $str = shift;
    $str = "" unless defined $str;
    unless($SKIP_UTF8_DOUBLE_ENCODE_CHECK) {
        if ($str =~ /[\x80-\xFF]/ and !utf8::is_utf8($str)) {
            warn("your OAuth message appears to contain some "
                . "multi-byte characters that need to be decoded "
                . "via Encode.pm or a PerlIO layer first. "
                . "This may result in an incorrect signature.");
        }
    }
    return
    URI::Escape::uri_escape_utf8($str,'^\w.~-');
};

Decoding is even simpler.

helper decode => sub {
    my $self = shift;
    my $str = shift;
    return uri_unescape($str);
};

In Python, the same code looks like so.

import urllib

def escape(s):
    """Escape a URL including any /."""
    return urllib.quote(s, safe='~')

def _utf8_str(s):
    """Convert unicode to utf-8."""
    if isinstance(s, unicode):
        return s.encode("utf-8")
    else:
        return str(s)

escaped = escape(_utf8_str(s))

# Unquoting is just
unquoted = urllib.unquote(s)

So, this needs to be done to each of the parameters, in this case, the consumer key and the token secret, if any, and then they are combined with an ampersand. So, if your consumer key is, say, "myconsumerkey", and you don't have a token yet, then the initial PLAINTEXT signature is myconsumerkey&.

Now, this isn't too bad, but once you get into HMAC-SHA1 signatures, it gets a lot worse. The signature from the PLAINTEXT method becomes the key for the signature, and you'll already have the code for that now, but the raw input to the HMAC-SHA1 algorithm is a base string, that is rather difficult to construct. The input is the HTTP method, the request URI, both normalized like above and contatenated with an ampersand. Then, this will be contatenated with an ampersand to all of the input parameters in the request, constructed in a particular way.

  1. Take all input parameter names and values from all sources, and normalize them like above. (but skip the oauth_signature parameter)
  2. Sort all parameters by the normalized parameter name.
  3. Pair the names and values, contatenated with an "=".
  4. Concatenate all pairs with ampersands in the sorted order.
  5. Escape the entire string using the method above.

This is the base string to the HMAC-SHA1 algorithm, along with the key we mentioned. The final signature should match what the client generated. Oh, and if you're running your service on a nonstandard port (80 or 443), then you must include the port in the URI.

Example:

A call to http://localhost/initiate on port 80 or 443, a GET request, with the following params:

{'oauth_nonce': '21823552', 'oauth_timestamp': 1356129798,
 'oauth_consumer_key': 'Mitel test', 'oauth_signature_method': 'HMAC-SHA1',
 'oauth_version': '1.0', 'oauth_signature': 'pevzNqSnJ8QtqFUDWVlYhVRp8D0=',
 'oauth_callback': 'oob'}

The base string would look like this:

GET&http%3A%2F%2Flocalhost%2Finitiate&oauth_callback%3Doob%26oauth_consumer_key%3DMitel%2520test%26oauth_nonce%3D21823552%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1356129798%26oauth_version%3D1.0

with a key of:

mitelsharedsecret&

and a final signature of:

pevzNqSnJ8QtqFUDWVlYhVRp8D0=

Oddly, if I used the b64encode method in Digest::HMAC_SHA1, a final "=" sign is missing on the final result, so I had to pull in MIME::Base64 and do this instead:

my $sig = encode_base64($hmac->digest);

The equivalent Python in the oauth library does this:

import binascii

# HMAC object.
try:
    import hashlib # 2.5
    hashed = hmac.new(key, raw, hashlib.sha1)
except:
    import sha # Deprecated
    hashed = hmac.new(key, raw, sha)

# Calculate the digest base 64.
return binascii.b2a_base64(hashed.digest())[:-1]

That just leaves RSA-SHA1, but that requires a pre-existing SSL relationship with the server, using SSL certificates. As such, I'm not worrying about it just yet. I don't think it'll be used much.

I'll need to do some interop testing with a few different clients, I'm hoping that they're not all snowflakes. The point of the rigid nature of the base string construction is that the final product is supposed to be reproducable.

The base string construction is definitely the hardest part, and I've read that the signatures were dropped in OAuth 2.0 because they were too hard to do. I'd rather not drop the added security, and while they're a pain, there are sample implementations to follow. I think that OAuth 1.0 is a better choice. And it's, like, finished.

December 20, 2012

Michael P. Soulier
msoulier
But I Digress
» Understanding OAuth

We meet again. Thanks for stopping by. This post has a high geek factor, so if you're not one, feel free to move along, nothing to see here.

Still here? K, your funeral. At work, I'm trying to add a ReST Web Service to a product that could desperately use it. I want to do this to permit better remote monitoring of said product and remote control for one of our management services and future UIs that I may implement (think apps on Android and iOS).

This is no small task, and it cannot be taken lightly. It will likely be used by many, so it must be easy to debug and integrate with, hence ReST. I'll rant about the evils of SOAP later. It must be secure, which is easy, Apache is already on the system, with SSL support. It must be authenticated. This is tougher.

Since I expect SSL, I could require client certificates, but they're a pain to provision. You don't authenticate to amazon.com with client certificates for this reason. Setting them up is about as much fun as visiting the passport office, and it hurts the wallet too. Why should anyone pay for the privilege of shopping in a store? Oh...I'll rant about CostCo later.

A popular method of authentication for web services is now OAuth. OAuth was designed to fix a particular problem. Example: I like Twitter. I hate Twitter's web interface. I prefer lightweight and simple. Plus, it's terrible to use from a smartphone. Fortunately, Twitter has a ReST web service that uses OAuth for authentication, so a vendor can release a Twitter app for my phone that I can connect to my account. "But," you ask, "surely you don't give this app your Twitter password!" No. I don't. It's a closed-source app and for all I know, they'd send it to themselves for nefarious purposes. I want the app to access my Twitter account, but I do not want them to have my password. How do we do this? Enter OAuth.

How does it work? The idea is simple. I'm going to go to my Twitter account, and tell Twitter to issue a random "token" in my name. If you think you don't know what a token is, think again. I suspect you carry keys, id badges, smartcards, that kind of thing. There isn't any practical difference, beyond keys being functional mechanically. This token I will then securely provide to the Twitter client on my phone, and when Twitter sees it, it will know that the client is mine, and grant access. The token will be transmitted over SSL to avoid spying and replay attacks, the hard part is securely getting the token into the client, and keeping it convenient. We do not want a repeat of SSL client certificates. Provisioning needs to be simple.

But, automating such things is problematic. Even if you implement a request queue for token requests, you don't want it to fill up with spammed garbage requests, and it will if we don't prevent it. Spammers should be pulled into the street and shot, but I digress. A way to prevent this is to at least have an existing relationship between the app vendor and the service. So, if I want to distribute an app, I need to go into Twitter and tell them that I want to be a "consumer" of their service. They'll ask me all kinds of details about who I am, and then issue me a consumer ID, and a shared secret. The shared secret is a random string that only they and I know. Now, when my app asks them for a token, it identifies itself with the consumer ID, and they validate it, which includes checking a signature built from the shared secret, so they know it's a valid request. BTW, this consumer ID can be used for billing later, as not all web services are free. Here comes SAAS.

Furthermore, it's not Twitter staff that need to approve this request, it's the owner of the resource in question, the Twitter account holder. This is good, as the account holder is the one who really cares here, and the Twitter staff would otherwise be overloaded with handling requests like these. Now, HTTP is a stateless protocol, so OAuth needs to add some state in the approval process. So, initially when we request a token, OAuth responds with a temporary one, so that the resource owner can be redirected to Twitter itself to approve access to their account. Assuming that you approve, a new token called a verifier will be issued. This verifier can be provided in the response, or simply displayed in the web page so that you can copy it into the app in an "out-of-band" manner (ie. cut and paste).

This verifier is then used to request the final set of tokens from the service. It proves to the OAuth subsystem attached to the service that:

  1. The client was written by an approved vendor. consumer id
  2. The owner of the account wishes to grant access to this client. token and verifier
  3. The request was not forged by some random ass-hat on the net. signature including shared secrets

Once done, the app has a set of tokens that it can continually use to make HTTP requests to Twitter's web service. The tokens issued can also be limited in permissions, perhaps providing read access but not write access, and perhaps to only specific areas of the API. The tokens can also be revoked, if you wish to pull said access in the future. This is a "good thing". I use this to fetch tweets, to make them, to manage follower lists, etc. All securely and authenticated over HTTPS. And I do it from the command-line, with my own client written in Python, which I dubbed Twit. I pull my data from Google in the same manner, for calendar events and contacts. Thus far it has worked well.

Now, for what I want to do, I'm implementing OAuth 1.0, even though the 2.0 "standard" is kinda coming. I phrase it this way because it's been coming for a while now, and as communities often do, it's already forked, dropped some security in the form of strong signatures, in the name of developer convenience. So, I'm sticking with 1.0. It's harder to implement, but it's done, and it's more secure. I've only just gotten PLAINTEXT and HMAC-SHA1 signatures working, I'll post on that soon. I'm doing the server in Perl and Mojolicious for legacy reasons, and implementing it myself because I've had issues with the Net::OAuth module for Perl. Maybe I'm using it wrong, but thus far I've gotten no response from the author. Plus, I learn much, much more this way.

This post is running a tad long, so I'll end it here, with more to come on my trip into OAuth. Specifically implementing support for the signatures. Funny thing is that the signing part is simple, but constructing the base string is freakin' hard. You'll get what I mean if you read the RFC, or just stay tuned.

See? Geeky. I warned you.

December 16, 2012

Michael P. Soulier
msoulier
But I Digress
» Getting started with autotools

So, I'm still reading O'Rielly's 21st Century C. I know, too many books on the go and I read slowly, and not often enough. I'm going through the section on GNU autotools, which I've never been a heavy user of, albiet I'm a heavy consumer of. I just don't spend much time distributing C/C++ across platforms.

I have a little C tool that I figured I'd try it on, a small replacement for GNU tree that I wrote a while back, and since tree isn't available on OS X, it seemed a good excuse to port it. Previously I just had a Makefile that I maintained, and it works fine, but it's a good excuse to learn how to use autotools for the future. I do have some libraries, and they're harder to port, which is where libtool comes in.

This build script outlines the process of using autotools for the first time. This script borrows very heavily from the book's author.

#!/bin/sh

echo "Creating Makefile.am"
cat > Makefile.am <<EOF
bin_PROGRAMS=twig
twig_SOURCES=twig.c
EOF

echo "Running autoscan..."
autoscan

echo "Creating configure.ac..."
sed -e 's/FULL-PACKAGE-NAME/twig/' \
    -e 's/VERSION/1.0/'   \
    -e 's|BUG-REPORT-ADDRESS|msoulier@digitaltorque.ca|' \
    -e '10i\
    AM_INIT_AUTOMAKE' \
        < configure.scan > configure.ac

echo "Creating additional files..."
touch NEWS README AUTHORS ChangeLog

echo "Running autoreconf..."
autoreconf -iv

echo "Running configure..."
./configure

echo "Running make distcheck to package sources..."
make distcheck

At this point, it's not ready to ship, as the NEWS, README, AUTHORS and ChangeLog aren't populated yet. But it's close. The configure script works, and I could then build it on OS X using the expected.

./configure --prefix=/usr/local
make
make install

My next project to package is a shared library for work, so that will be more interesting. Still, if you're looking to use autotools for the first time for something simple, this should take the mystery out of kick-starting it. Sure, there's some magic like the AM_INIT_AUTOMAKE macro, and I've a ton to learn yet, but this worked on the first try, and the resulting tarball is good to push to SourceForge or elsewhere if you want to.

As I pick up more, I'll try to share it. I don't find autotools intuitive at all, but with some simple recipes I think I'll survive.

December 9, 2012

Michael P. Soulier
msoulier
But I Digress
» Brewing a better UNIX

Being a Linux user, I'm used to a fairly standard set of tools on my command-line. On OS X, it's essentially BSD UNIX underneath, which I have had some experience with thanks to FreeBSD. The environment is very similar, but not identical, and on FreeBSD, you quickly find yourself using the ports system to install new packages that have been ported to FreeBSD from places like Linux.

It took about five minutes of FreeBSD use for me to go install bash, wget, vim, imagemagick, and a host of other packages. There's a lot already installed with OS X, but a few key things or me were definitely missing. I pulled vim from MacVim, but then I had to look for somewhere to get everything else I needed.

Being UNIX, I had several choices:

  1. Grab the source and build it myself.
  2. Install MacPorts and build it from there.
  3. Install Homebrew and build it from there.

Well, I've done source installs, I've done ports out of FreeBSD, so I figured I'd see how Homebrew works. Basically, it makes /usr/local owned by the user that installs it, which on OS X is me, as an administrative user (like a Windows power user). From there you can double-check that everything is set up properly by running:

brew doctor

It will pick up permission problems, issues with your PATH environment variable, warn you about packages that were built but not symlinked into /usr/local properly, etc. Then it's not much different than using apt-get on Debian, except that the packages are building when they install, they're not pre-built binaries. So wget was just:

brew update
brew install wget

Since then I've installed some essentials, and non-essentials if I include freeciv. Lets see, I have:

msoulier@merlin:~$ brew list
c-ares      git         lame        lua     sdl_mixer
cracklib    glib        libevent    lynx    tmux
feh         gmp         libffi      mutt    tokyo-cabinet
flac        gnupg       libgcrypt   nettle  unrar
fontconfig  gnuplot     libgpg-error p11-kit wget
freeciv     gnutls      libmikmod   pcre    xz
freetype    imagemagick libogg      pkg-config
gd          imlib2      libpng      readline
gettext     irssi       libtasn1    sdl
giblib      jpeg        libvorbis   sdl_image

Rather nicely, the packages are all installed under /usr/local/Cellar/, and symlinked into the right places so they show up in my path, and for building. As I really hate installing from source, because you never know what you have installed or how to uninstall it, or what you'll break if you upgrade it, I like this.

I do find that other packages mess with those careful permissions, so I keep running brew doctor so I know about the issues. I also noticed, thanks to a coworker, that the Perl community has done something similar for install Perl modules called Perlbrew. I'm going to look into that soon, as honestly, it's about damn time. Managing personal Perl modules sucks, has always sucked, and now thanks to Perlbrew will hopefully suck no longer. But I digress, more on that in another post.

I give Homebrew an A. An A+ would be a command to fix permissions problems without my help, and maybe there is one, these things elude me at times. I highly recommend it if you're trying to complete your *nix environment on OS X.

December 6, 2012

Michael P. Soulier
msoulier
But I Digress
» The Apple Support Experience

So, having jumped into Apple land with a Mac Mini recently, and having some problems, I of course took advantage of my included 90 days technical support. Being part of the Apple ecosystem includes being subject to how they support their customers, so I think it's important to write about.

Normally my idea of support is to figure out what piece of software I'm having a problem with, locating the open-source community responsible for it, and then tracking down a mailing list, IRC channel, whatever, where I can ask for help. I usually loathe help lines of any kind, as North America and most of the world handles these very poorly these days.

You know the story. You call hoping that you'll talk to a flesh and blood human, and you get a machine telling you that, "we are currently experiencing a higher than normal call volume," which is what you'll hear every damn time you call so obviously it's not higher than normal, and they're just cheap bastards who understaff their support infrastructure, but I digress.

Well, I collected a few questions together before calling Apple Canada, as I hate calling help lines so I figured once I found a human I'd hammer him/her with several questions. Better throughput. Can't help it, I think like a programmer. And, of course, I got a machine. I asked me what product I was calling about, using voice recognition to understand me as I said, "Mac Mini", and asked me for the serial number. That all worked, and I finally got a human, with little difficulty. No Indian accent either, I think it was Spanish, but I've called a few times since and there's no discernable pattern, so perhaps it's not overseas for a change. Not that I care all that much, as long as they actually help, but I've had some really bad experiences before with uninformed first-line support reading from their scripts.

Well, the guy I talked to couldn't help with the software bugs, obviously he doesn't own the code, but he can report the issues and I can hope that a developer at Apple will fix them. I already reported the issues at http://www.apple.com/feedback/ anyway, and hopefully that's not a digital black hole like most feedback pages. He did help me understand how to do a couple of things that OS X considers advanced, like launching a terminal from a keystroke. BTW Apple, that's pretty basic in any Linux window manager, so please do better.

Then we got down to the hardware issue I'm having. At least, I think it's hardware. Many other people are having the same problem. It makes me wonder if there was a bad batch of Mac Minis out of Apple recently. I called in about it, mentioned to the nice lady that wanted to sell me an Applecare package that I likely wouldn't need it, as I was thinking of taking the unit back, and suddenly I was talking to a nice man in second level support.

He took the issue on personally, gave me a direct contact number to him, and the email address, and we started swapping hardware to narrow down the potential problems. I swapped the LCD with one upstairs, no help. Got a new HDMI to DVI adaptor from Apple, no help. He arranged for a full unit exchange and data migration, which was done very professionally (albiet with some data loss that was easily made up for), unfortunately it didn't help.

Finally, we're running an app from Apple to gather up logs, and I'm told that the Engineers at Apple know what the problem is, they think it's software, and to keep my eyes on the software updates. There was one recently in fact, labelled graphics performance and reliability, so I wonder if that was it. I'm waiting to see if the issue happens again.

Thus far, I must say that Apple raises the bar for what good customer support should be. I've had plenty of bad experiences with other companies, and so far I've had mostly good ones with Apple, and now an exceptional one. The guy I'm talking to deserves a raise. Fingers crossed that the experience continues to be this good.

November 28, 2012

Michael P. Soulier
msoulier
But I Digress
» Parenting can really make you feel helpless

I've only been kicking around for a bit over 40 years now, but I've never found anything in the world to make me feel more helpless than my own child telling me that they feel bad, and knowing that I can't fix it.

My daughter was sick last night, and we monitored her fever as it rose up to a slightly worrisome 39.4C. I went out for some children's ibuprofen, as all I had in the cupboard had expired. Good thing that, the kids haven't been sick much. Meanwhile my lovely wife put her into the bath while I was gone. I brought back popsicles too, since her throat hurt and maybe the cold would help with the fever.

I ended up not using the medicine, as her temperature slowly dropped after the bath, and she feel asleep with her Mom for the night. I checked on her at about 2am and 4am, and she was definitely cooling off, so I just sighed in relief and got some sleep.

Still, it doesn't really feel like she got better because of anything that we did. We could comfort her, but in the end, her body was at war with an invader, and it was on its own. We were helpless to help. If her temperature suddenly spiked we'd be off to the hospital to let someone else try to save her. Still helpless to do anything personally. We pray that her body is stronger than the virus. We pray that the doctors know what they're doing. I hope the world will pardon me for feeling that prayer isn't particularly effective, and primarily an act of desperation. I don't like being deperate.

I had heard that nothing would terrify me as much as being a parent, but there's simply no way to prepare someone for just how scary it can be to have this perfect, beautiful little person in pain, or sick, or just crying, and not being able to do anything about it. It's even worse for a control freak like me, and like most parents, I think that the Universe would make more sense if parents could take the children's hurts on themselves. The rational part of my mind knows that some pain is necessary as part of life, and that getting sick makes your immune system stronger. The parent in me doesn't care. Child hurt. Must fix.

The little one is home today, as I want to keep an eye on her (see "Control Freak"), but the worse is past and she'll be fine. Her immune system wins another one, and comes away stronger. Me, I'm just relieved and very tired. I really should call my parents and thank them, repeatedly.

» Customizing Awesome

As I mentioned previously, I'm currently running Awesome on my netbook. While I added suspend on lid close, I still need to customize the crap out of it. I had nothing showing me my current battery status, so I searched around to find the appropriate lua code, a language that I'm still picking up.

So, I hacked my $HOME/.config/awesome/rc.lua and added this:

-- {{{ Battery state

-- Initialize widget
batwidget = widget({ type = "textbox" })
baticon = widget({ type = "imagebox" })

-- Register widget
vicious.register(batwidget, vicious.widgets.bat,
    function (widget, args)
        if args[2] == 0 then return ""
        else
            baticon.image = image(beautiful.widget_bat)
            return "<span color='white'>".. args[2] .. "%</span>"
        end
    end, 61, "BAT0"
)

-- }}}

And then added the widget to the top of my screen. I was warned on the awesome mailing list (the name does confuse the english language) that I had to be careful where to put it, since the tasklist is greedy and will take all available space.

mywibox[s].widgets = {
    {
        mylauncher,
        mytaglist[s],
        mypromptbox[s],
        layout = awful.widget.layout.horizontal.leftright
    },
    mylayoutbox[s],
    mytextclock,
    s == 1 and mysystray or nil,
    baticon.image and separator, batwidget, baticon or nil,
    separator, mytasklist[s],
    layout = awful.widget.layout.horizontal.rightleft
}

And, it works. Now I need a system load, cpu load, etc. And I should hack the keybindings, although I'm getting used to the default ones. Maybe update the default layout since I prefer maximized layout.

November 25, 2012

Michael P. Soulier
msoulier
But I Digress
» Tweeting on OS X

For some time my desktop has been an older, underpowered Linux box, so using services like Twitter via the website has been painful, as loaded with heavy javascript that would consume my CPU's limited resources.

So, like any Python programmer would do, I wrote my own client. Well, lets be honest, the hard work was done by the tweepy library, I'm just using it. I call it Twit, and I've been using it to post for some time now, lightweight from the command-line. I also keep one running, polling my account for new posts, and notifying me when there are new posts and pulling them down.

My notifications come in many forms, depending on the command-line options. I can just watch them show up in text in the shell, I can use xosd to display notifications on your X11 desktop, It can use libnotify in Gnome to display temporary notifications that drop down in the corner of the screen.

As I'm now playing with an OS X desktop, libnotify isn't supported. I could use xosd along with XQuartz, but I'd prefer more native integration with OS X, as it does have a notification system, via AppKit. Python is preinstalled on OS X, and the AppKit module is included. I found a great example of how to use it online, which made this much, much simpler, so thanks there.

I used it like so:

def notify_appkit(status, options):
    """Thanks to
    https://github.com/albertz/music-player/blob/master/notifications.py
    for how to do this."""
    global notifCenter
    if not notifCenter:
        import AppKit
        notifCenter = \
            AppKit.NSUserNotificationCenter.defaultUserNotificationCenter()
        appDelegate = AppKit.NSApplication.sharedApplication().delegate()
        notifCenter.setDelegate_(appDelegate)

    notif = AppKit.NSUserNotification.alloc().init()
    title = "Tweet by %s" % status.user.name
    notif.setTitle_(title)
    notif.setInformativeText_(status.text)
    notifCenter.deliverNotification_(notif)

Now, this does cause a little icon on the dock to jump up and down, so I'll need to look into that, and how to open the browser to twitter or the embedded link when the notification is clicked on, but it's a good start.

November 21, 2012

Michael P. Soulier
msoulier
But I Digress
» Suspend on lid close in Debian Squeeze

I recently decided that Gnome is not the best desktop for my little EeePC netbook with a little 10.5" screen. So I'm playing around with a window manager that mainly just maximizes everything. I've tried Ratpoison, I've tried wmii, and now I'm trying Awesome.

I have a lot of customizations to do, but one thing that was missing was a way to suspend the netbook when the laptop lid is closed. I could manually run acpitool -s in a shell, or pm-suspend, but it's best handled by DBus, as intended.

Now, a simple way to have DBus do the work is using dbus-send, like so:

dbus-send --print-reply \
          --system \
          --dest=org.freedesktop.UPower \
          /org/freedesktop/UPower \
          org.freedesktop.UPower.Suspend

The hard part is subscribing to the lid close event, so I'm not polling all the time, exactly what DBus was written to prevent. I had a Python script for this, but the API was changed in Squeeze to use the UPower daemon and API.

I had to do some poking around to figure out how to update it, but I just got it working, so I thought I'd share.

#!/usr/bin/python

import dbus, gobject, sys
from dbus.mainloop.glib import DBusGMainLoop

pow_prop_iface = None
pow_iface = None

def handle_lidclose(*args):
    closed = pow_prop_iface.Get('',
                                'LidIsClosed')
    if closed:
        print "lid is closed, suspending"
        pow_iface.Suspend()
    else:
        print "lid is open"

def main():
    global pow_prop_iface, pow_iface

    DBusGMainLoop(set_as_default=True)

    bus = dbus.SystemBus()

    power_proxy = bus.get_object('org.freedesktop.UPower',
                                '/org/freedesktop/UPower')

    pow_prop_iface = dbus.Interface(power_proxy,
                                    'org.freedesktop.DBus.Properties')
    pow_iface = dbus.Interface(power_proxy,
                               'org.freedesktop.UPower')

    print "Registering a signal receiver for upower events..."

    bus.add_signal_receiver(handle_lidclose,
                            dbus_interface="org.freedesktop.UPower",
                            signal_name="Changed")


    loop = gobject.MainLoop()
    loop.run()

if __name__ == '__main__':
    main()

Now I just run it in the background from my .xsession script at X11 login, and it's sitting there waiting for any change in UPower status. Works like a charm.

November 19, 2012

Michael P. Soulier
msoulier
But I Digress
» Joysticks? We don't need no stinking joysticks...

So, after unpackaging my brand-spanking new Mac Mini, I grabbed a copy of the X-Plane demo, dropped in an old usb joystick and fired it up. Oddly, X-Plane itself asked me if I wanted to calibrate the joystick, which struck me as odd, that hasn't happened in years, not since my days with plain old DOS. DOS didn't have a controller subsystem so there was no way to calibrate a joystick once, for all apps, but Microsoft fixed that long, long ago. Even Linux has jscal to allow the same thing. It's command-line, but it works.

So of course I assumed that on a Mac, with it's reputation for dead simple plug and play, everything just works, dead simple and easy, it would have the same right? Umm, no. No command-line tool, no graphical tool, no voice activated tool, no swiping on my LCD, nothing. It's left up to the application, kinda like DOS. Way to go Apple.

Now, why do I care, since X-Plane has such a utility? Well, in X-Plane, I don't. But after downloading and installing Flightgear, I do. They leave the joystick calibration up to the OS, and rightly so. So, I can't use the joystick in Flightgear, not that I'd want to use it on this machine anyway, as I've already found out that the OpenGL rendering performance on the Mac Mini is horrid. Wow, another strike over my cheap Linux box sitting next to me that was made out of spare parts from a friend's basement. That is truly, utterly, sad.

Seriously Apple, is that the best you can do? If you think I'm interested in tilting an ipad to fly a plane in any realistic simulation, you guys have been smoking way too much of what you usually smoke. I want realism, and last time I flew, the pilot didn't have to lean to bank the plane. At least, I hope not, the door was closed and locked, post-9/11 and all, but I digress.

Bottom line Apple, for what I paid for this thing, you could have done much, much better in this regard. X-Plane rocks, but Intel graphics in the mini? Pleease, put a smoking Nvidia chipset in there and give me a decent framerate for games!

Are you listening?

November 17, 2012

Michael P. Soulier
msoulier
But I Digress
» Umm, done unzipping yet?

So, the first thing I did after after wrapping my head around the cocoa interface was to start installing software like a madman. I wanted Firefox, VLC, Kobo, Gimp, Vim, etc, etc, etc. So, I had to learn how to install apps on OS X.

Apps in OS X seem to come in at least two different forms: .app files, which are self-contained application packages that you simply drop into /Applications on the system (totally awesome), and .pkg files, which are more windows-like installers.

The latter seems to be required with more complex installs, like plug-ins that need to put files in privileged places all over the system, requiring root access to install. As an admin user, I can already put .app files into /Applications without root access, the implications of which are that a virus could infect the apps in /Applications too, so that's a strike, security-wise, for OS X over Linux. Still better than Windows though, where a Power User can change anything under Program Files or Windows without escalating privileges.

The .app files or .pkg files can come in many forms, the most common of which is a .dmg file, which is a disk image that OS X will casually mount on a loopback. I've noticed that the Finder can mount .dmg and .iso files with a simple click. With an .app file, you just drag it to /Applications, and you're done, and uninstalling is just a simple. Far superior to any OS that I've used in the past. It's also common to download .zip files, which clicking on in the Finder fires-up the Archive Utility.

So there I was, clicking along and installing, and then a progress bar indicating that the .zip file was opening just kept going, and going, and going. I looked online, and found that I am not alone in having this problem. Apparently, the appleeventsd daemon requires killing under those circumstances, something that the nice guy at Apple support did not know. Looks like the issue is in OS X 10.8.2, which I am running.

Hmm. Bad timing in jumping on the Apple bandwagon? I wonder how long they'll take to fix the issue. Good test for what it will be like to live as part of Apple's ecosystem.

November 16, 2012

Michael P. Soulier
msoulier
But I Digress
» Ok, nobody panic, there's a fruit on my screen

I've needed a new desktop for a while now. As I'm primarily a Unix user, I typically buy a cheap PC and put Linux on it. This involves doing a bunch of research into available hardware to make sure that everything is going to work, like accelerated graphics for any games I'd like to play, supported network drivers, etc. Over the years it's gotten easier and easier to get Linux working on a box, but sometimes I still have to work to get something working.

The end result is a working Linux workstation that I can use for work, and my basic computing needs, often involving fighting with the fact that noone wants to support it or any software that I'm using. That part does get old year after year.

For example, I bought a Kobo ages ago, and I had to use a Windows box to run the Kobo desktop software, to patch its firmware and put books on it. Later I found out that I could just copy .epub books into it like a flash drive, which is awesome by the way Kobo, but you get the idea. For my iPod shuffle, I copy mp3s onto it and run the rebuild_db Python script to update the db on the iPod. There are workarounds for these issues, but sometimes it's nice to just plug something in and have it work. And fighting with my devices isn't as much fun as it used to be. I'm too damn busy.

So, based the recommendations of various friends and colleagues, I decided to take the plunge with my new desktop, and try a Mac Mini from Apple, running Mac OS X 10.8, "Mountain Lion". I figured that it's a lot better supported in commercial software land, while being BSD Unix underneath, so maybe it's the best of both worlds.

http://www.but-i-digress.ca/static/images/apple-logo1.jpg

I need to customize the crap out of it, and get used to the interface and keybindings, and do a lot of reading. I hope to post my experiences here, so stay tuned if you're interested.

Sure, I could have just thrown Linux on another box, and maybe I still will, but I've done that. I felt like doing something new.

November 9, 2012

Michael P. Soulier
msoulier
But I Digress
» What Curling has taught me about life

I've been curling for about 4 years now, and it's already taught me a lot about life. I thought I'd share some of what it has taught me:

  • balance is very important
  • anything worthwhile requires hard work
  • you often can't brute force your way out of a problem
  • you must think before you act
  • you can often do much more with help from others
  • shouting at people doesn't actually make them work harder
  • hope for the best and plan for the worst
  • when you finally understand the conditions, they change
  • don't take anything too seriously

and finally

  • men can still look like men while holding a broom :)

Cheers

November 8, 2012

Michael P. Soulier
msoulier
But I Digress
» I sit too much

Yeah, I really do. From my childhood I always prefered more intellectual activities than active ones, reading, watching movies, tv. Growing up asthmatic had something to do with that, but I overcame it later in life and had no excuses anymore. Still, studies to do, computers to play with, digital bad guys to beat up, and a lot of my activities involved sitting.

I played baseball as a kid, got into Tae kwon Do in my late teens, but as a University student I couldn't afford the money to continue, and I moved a lot for a while. I was never much of a runner, but for a while I was a damn good cyclist because I didn't have a car and it was how I got around.

In recent years a back injury made it clear that I sit too much, and I've tried various things to try to fix it, usually charging in too fast, determined to conquer the problem and I ended up with another injury. I needed to figure out how to go slower, targeting my problems and not doing too much, but making progress.

Recently the wife that I don't deserve found Athlete365 and Mr. John Sokolowski. He looked at my flexibility and strength, or lack thereof, and has put together an evolving program to help me fix them, at a speed that won't set me back with another injury. He also gave me some tools to help with muscles that keep me awake at night, ruining my sleep. He also thinks that I should become a Yoga master. Heh...sure John. Maybe in 20 years of practice. I'm right on that.

I've also been tweaking my diet, not that my diet was bad, but I recently found out about some of the evils of excessive carbohydrates, from an amusing documentary, Fat Head, a response to "Supersize Me". Since reducing bread (not hard, I have a gluten tolerance issue), rice and potatoes in my diet, and exercising more, I've lost about ten pounds in the past couple of months.

My issues aren't gone by any means, I have a lot of work to do, but guidance helps. There's a difference between working hard and working smart. Previously, I was just working hard.

If you've read this sitting down, get up and move around. Trust me.