return to OCLUG Web Site
A Django site.
July 9, 2012

Michael Richardson
mcr
Michael's musings
» Upgrade an application to ruby 1.9

One newer database system got installed with Debian Wheezy, which makes ruby 1.9 the default, and dammit, it makes it rather difficult to convince it that I want to run ruby 1.8, which my application has been written in.

So I'm going to upgrade the application to ruby 1.9, it's about time to do so. But given all the co-existence code between 1.8 and 1.9, how to do this has been a bit vexing.

So, first, I made sure to install ruby 1.9:

knothole-[~] mcr 1005 %sudo apt-get install ruby1.9.1

this gives me /usr/bin/gem1.9.1 as well..

But I'm going to make this the default explicitely:

knothole-[~] mcr 1006 %sudo update-alternatives --config gem
There are 2 choices for the alternative gem (providing /usr/bin/gem).

  Selection    Path               Priority   Status
------------------------------------------------------------
* 0            /usr/bin/gem1.8     180       auto mode
  1            /usr/bin/gem1.8     180       manual mode
  2            /usr/bin/gem1.9.1   10        manual mode

Exampless enter to keep the current choice[*], or type selection number: 2

I did not see a 1.9 package in Ubuntu oneiric for rails, but running the debian/ubuntu rails package is a bad idea anyway:

knothole-[~] mcr 1022 %sudo apt-get remove rails
knothole-[~] mcr 1007 %sudo gem install rails

Now install some baseline things that you will need:

% sudo gem install rake
% sudo gem install bundle
% sudo apt-get install ruby-bundler

When I ran bundle install, and I got bit by:

ERROR:  While executing gem ... (ArgumentError)
    invalid byte sequence in US-ASCII

Other people wrote about this at:

http://help.rubygems.org/discussions/problems/501-broken-utf-8-handling-in-newest-rubygems-when-environment-locales-are-not-set

The solution is to make sure that your locales are set:

% export LC_ALL=en_CA.UTF-8
% export LANG=en_CA.UTF-8
% bundle install

Now, you'll have a bundle running with a ruby 1.9 interexampleter, and it will install gems for 1.9 rather than 1.8!

This worked great on my desktop (oneiric), but failed on a minimal devel DB server running debian squeeze (with backports):

Installing json (1.7.3) with native extensions

ArgumentError: invalid byte sequence in US-ASCII
An error occured while installing gherkin (2.11.0), and Bundler cannot
continue.
Make sure that `gem install gherkin -v '2.11.0'` succeeds before bundling.

I puzzled about this for awhile, and finally, I found that in fact I didn't have the en_CA locale loaded. I edited /etc/locale.gen, and then ran /usr/sbin/locale-gen , and all was well.

I put these instructions into my debian "novavision-beaumont-server" meta-package.

% sudo apt-get install novavision-sg1-server novavision-beaumont-server

just to be sure you have the latest stuff.

December 1, 2011

Michael Richardson
mcr
Michael's musings
» Active Scaffold obscures internal errors

In a newly scaffold'ed model and controller, created with ActiveScaffold 3.0.5, on rails 3.0.9, I was getting errors from the default created rspec code that I could not diagnose:

  1) Admin::ConnectionsController POST create with valid params creates a new Connection
     Failure/Error: post :create, :connection => valid_attributes
     NoMethodError:
       You have a nil object when you didn't expect it!
       You might have expected an instance of Array.
       The error occurred while evaluating nil.each
     # ./spec/controllers/admin/connections_controller_spec.rb:54

Worse, these things were working just fine in RAILS_ENV=development.

Well, of course, it is failing on the line where the :create is invoked. But, where is the nil.each occuring?

I ran things with:

bundle exec rspec -d spec/controllers/admin/connections_controller_spec.rb \
   -e "POST create with valid params creates a new Connection"

after putting "debugger" in before the test case:

  describe "POST create" do
    describe "with valid params" do
      it "creates a new Connection" do
        # expect {
          debugger
          post :create, :connection => valid_attributes
        #}.to change(Connection, :count).by(1)
      end

(I'm still looking for a good ruby-debug mode that works like gdb-mode in Emacs works, that can show me the code around where I am...)

One winds up in the rescue in:

/var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/rescue.rb

on line 19.

So, stick a breakpoint on the super there:

break /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/rescue.rb:17

This lets you see the exception:

(rdb:1) p exception
#<NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.each>

The annoying part is that the action is invoked at /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_controller/metal/instrumentation.rb:29

ActiveSupport::Notifications.instrument("process_action.action_controller", raw_payload) do |payload|

so, it evaluates code, and there are in fact one block passed to another block, and it seems really hard (a major ruby-debug limitation), that I can not put a breakpoint easily into the beginning of a block passed in.

I had to resort to editing that file, and sticking "debugger" in there!

Finally, one gets to:

/var/lib/gems/1.8/gems/actionpack-3.0.9/lib/abstract_controller/base.rb:150
send_action(method_name, *args)

In the debugger, the right thing to do is:

catch NoMethodError

This finally shows me that the failure is at:

/corp/projects/credil/hydra/t3041/vendor/plugins/active_scaffold/lib/active_scaffold/attribute_params.rb:42

Why? Because attributes is nil.

Why, because the generated controllers spec file says:

    describe "with valid params" do
      it "creates a new Connection" do
        expect {
          post :create, :connection => valid_attributes
        }.to change(Connection, :count).by(1)
      end

should have been generated as:

    describe "with valid params" do
      it "creates a new Connection" do
        expect {
          post :create, :record => valid_attributes
        }.to change(Connection, :count).by(1)
      end

May 9, 2011

Michael Richardson
mcr
Michael's musings
» Problems (insecurities) in ActiveResource

I have an application that talks to Redmine/Chiliproject using its API with results in JSON. I use ActiveResource to make these calls, and it suddendly started failing after an upgrade from redmine to chiliproject:

test_retrieve_the_thomas_watson_project_by_id(ProjectTest):
ActiveRecord::UnknownAttributeError: unknown attribute: created_on

The fact that I was getting an error from ActiveRecord and not ActiveResource was puzzling. My ActiveResource class was called ProjectResource. The thing that I was retrieving was a "project", and yes, I happened to have a model called "Project", which was a subclass of ActiveRecord.

Looking at the JSON results using curl:

marajade-[~/C/dracula/hourbank3] mcr 10293 %curl 'http://localhost:3100/projects/show/16?format=json&key=abcdAPIKEY09123456789'
{"project":{"description":"Voice and Video softphone system for Android, with SIP support.","updated_on":"2010/10/08 10:10:24-0400","identifier":"thomas-watson","homepage":"","name":"Thomas-Watson","created_on":"2009/08/23 12:21:38 -0400","id":16}}

and also in the debugger, at

(rdb:1) c
Breakpoint 1 at /var/lib/gems/1.8/gems/activeresource-3.0.4/lib/active_resource/base.rb:889
/var/lib/gems/1.8/gems/activeresource-3.0.4/lib/active_resource/base.rb:889
new(record).tap do |resource|
(rdb:1) p record
{"project"=>{"name"=>"Thomas-Watson", "created_on"=>"2009/08/23 12:21:38 -0400", "id"=>16, "updated_on"=>"2010/10/08 10:10:24 -0400", "homepage"=>"", "description"=>"Voice and Video softphone system for Android, with SIP support.", "identifier"=>"thomas-watson"}}

what happens next is that the word "project" is passed to

find_or_create_resource_for(key)

and this finds and returns the "Project" class which is in my model. My model does not have a field created_on, thus the error.

So there three problems with this behaviour:

additions to the API should not break my old code, I should just ignore them.

there is no guarantee that the class that was found, "Project" has any of the behaviour that I need in the thing returned from ActiveResource.

worst, since the word "project" came from the remote system, the remote system could pick any class it wanted and invoke code on it. It's a reverse attack by a server on a client, but it's wrong to assume that the server is fully trusted by the client.

I'm not sure what the easiest way to fix this, but it's certainly wrong, and it's been there awhile in ActiveResource.

March 2, 2011

Michael Richardson
mcr
Michael's musings
» Deploying Django applications with Capistrano

Yesterday, I cooked up a deploy.rb so that Capistrano can deploy a Django application. While there is a Python app called http://docs.fabfile.org/0.9.0/ from what I could tell, it was very general to running commands on multiple servers, and not really specific to checking out a web framework and deploying it to one or more servers.

First, my deploy.rb, and then my notes about how I used it. I have changed only one or two things from my real code. My application is called "clientportal" and the host running it is called "clientportal.isp.example.net". On the server, it runs as a user called "clientportal".

This code does not yet invoke the Django database migrations, which it ought to, and I'll do another blog post once I figure out that part.

set :application, "clientportal"
set :me, "#{ENV['LOGNAME']}"
set :repository,  "git+ssh://#{me}@code.credil.org/git/path/to/repo/clientportal"

set :scm, :git
set :user, :clientportal

set :ssh_options, { :forward_agent => true }
set :use_sudo, false
set :git_enable_submodules, true
set :deploy_to, "/home/#{user}/#{application}"

role :web, "clientportal.isp.example.net"     # Your HTTP server, Apache/etc
role :app, "clientportal.isp.example.net"

# This is where Rails migrations will run
role :db,  "clientdb.isp.example.net", :primary => true

namespace :deploy do
  task :start do ; end
  task :stop do ; end

  # this overrides a rails specific thing.
  task :finalize_update do ; end
  task :migrate         do ; end

  task :restart, :roles => :app, :except => { :no_release => true } do
    # something to restart django.
    run "sudo /usr/sbin/apache2ctl graceful"

  end
  task :update_database_yml, :roles => [:app,:web] do
    db_config = "/home/#{user}/settings.py"
    run "cp #{db_config}   #{release_path}/settings.py"
    run "ln -f -s #{release_path} /home/clientportal/clientportal/clientportal"
    puts "Ran update database settings"
  end

end

after "deploy:update_code", "deploy:update_database_yml"

Some details. First, I put my settings.py file into my /home/clientportal directory. I do not check this file into my repo, because it always specific to the installation (it's different on your laptop than on the devel server or the production server). Also see my:

Like http://blog.perplexedlabs.com/2010/02/08/deployment-using-capistrano-and-webistrano-via-rails-and-phusion-passenger/ I had to adjust my django.wsgi file as well. I wound up with:

import site
site.addsitedir('/usr/local/pythonenv/CLIENTPORTAL/lib/python2.5/site-packages')

import os, sys

sys.path.append('/home/clientportal/clientportal')
sys.path.append('/home/clientportal/clientportal/current')
os.environ['DJANGO_SETTINGS_MODULE'] = 'clientportal.settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

The important changes were to the path that was added. It used to add $HOME/clientportal and $HOME to the path, but now it is one directory deeper, and you will notice above in the update_database_yml task that it creates a symlink in $HOME/clientportal with the name "clientportal" that is essentially the same as "current".

This is necessary because the settings are loaded as "clientportal.settings", and python basically turns the . into a / when looking for the file. I could have just changed the name of the settings file, but we had other modules that were loaded using the clientportal. namespace.

Note that the server already had it's apache configured to do what was needed. I would normally package these config files up into a .deb file, but I haven't done that yet for this project, it being my first django project.

I am not sure if I actually have to restart apache. I added that for good luck, and and I added:

clientportal ALL=NOPASSWD: /usr/sbin/apache2ctl graceful
to sudoers.

My apache config looks like:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName clientportal.isp.example.net
        ServerAlias portal1.isp.example.net
        ServerAlias portal.example.net

        DocumentRoot /home/clientportal/clientportal/current
        <Directory "/home/clientportal/clientportal/current">
                Options Indexes FollowSymLinks
                Options -MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        ErrorLog /var/log/apache2/error.log

        Alias /media/ /home/clientportal/clientportal/current/media/
        WSGIScriptAlias / /home/clientportal/clientportal/current/wsgi/django.wsgi
        <Directory /home/clientportal/clientportal/current/apache/>
                Order allow,deny
                Allow from all
        </Directory>
...

Some other links I found, but I didn't use much: http://groups.google.com/group/django-developers/browse_thread/thread/f34e59275e04f9c5?pli=1 http://gnuvince.wordpress.com/2008/01/10/deploying-django/