return to OCLUG Web Site
A Django site.
March 15, 2010

Pythian
pythian
» dpanneur – your friendly DarkPAN/CPAN proxy corner store

dépanneur
CC BY-NC-SA 2.0

There were two things I have wanted to do for some time now. The first was to come up with a way to quickly and easily set up a DarkPAN mirror so that we would have more control over our dependency chain at work. The second was to make a portable CPAN proxy service, so that I can always have access to my favorite modules, even if the machine I’m working on has no Internet access. Last week, I finally had a few ‘rount tuits’ to spend on this type of background itch, and the result is dpanneur (for dépanneur, French Canadian for convenience store).

Installation

As it stands, dpanneur is a very thin Catalyst application gluing together the goodiness of CPAN::Cache and MyCPAN::App::DPAN, and throwing in Git as the archive manager.

To get it running, first fetch it from Github

$ git clone git://github.com/yanick/dpanneur.git

then check that you have all the dependencies

$ perl Makefile.PL

and run the script that will create the module repository

$ ./script/create_repo

For now, the module repository is hard-coded to be in the subdirectory cpan of dpanneur. A branch called proxy is created and checked out. Eventually, I’ll use GitStore to push newly fetched modules to the repository, but for the time being if dpanneur is to be used as a proxy, that branch must remain as the one being checked out.

All that is left is to fire up the server in whichever mode you prefer (single-thread test server would do nicely for now)

$ ./script/dpanneur_server.pl

and there you are, running your first dpanneur. Congrats! :-)

Using it as a caching proxy

You can use the server as a caching proxy, either for its own sake, or to seed the DarkPAN branches. To do that, you just have to configure your CPAN client to use http://yourmachine:3000/proxy:

$ cpan
cpan[1]> o conf urllist = http://localhost:3000/proxy
cpan[2]> reload index
cpan[3]> install Acme::EyeDrops
Running install for module 'Acme::EyeDrops'
Running make for A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz
Fetching with LWP:
    http://localhost:3000/proxy/authors/id/A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz
etc..

As the modules are downloaded, they are also saved and committed within the repo

[dpanneur]$ cd cpan

[cpan (proxy)]$ git log -n 3
commit d065ad152f2204295334c5475104a3da517b6ae1
Author: Yanick Champoux <yanick@babyl.dyndns.org>
Date:   Wed Mar 10 20:32:52 2010 -0500

    authors/id/A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz

commit e8d2e83d1b16e2e0713d125f9a4bd2742681f859
Author: Yanick Champoux <yanick@babyl.dyndns.org>

Date:   Wed Mar 10 20:31:42 2010 -0500

    authors/id/D/DC/DCONWAY/Acme-Bleach-1.12.tar.gz

commit 7e0b4b600bac8424c519199ee96dc56ffbb177eb
Author: Yanick Champoux <yanick@babyl.dyndns.org>
Date:   Wed Mar 10 20:30:47 2010 -0500

    modules/03modlist.data.gz

Using it as a DarkPAN server

There is not much more involved to enabling DarkPAN repos. All we have to do is to create a branch with the modules we want and have the ‘dpan’ utility bundled with MyCPAN::App::DPAN generate the right files for us.

To continue with the example of the previous section, let’s say that we want a DarkPAN branch containing Acme::EyeDrops, but not Acme::Bleach. Then we’d do

                        # only necessary if you are running
                        # the server while you work on the branch
[dpanneur]$ git clone cpan cpan-work   

[dpanneur]$ cd cpan-work

                        # branch just before we imported Acme::Bleach
[cpan-work (proxy)]$ git branch pictoral 7e0b4b600bac8424c519199ee96dc56ffbb177eb

[cpan-work (proxy)]$ git checkout pictoral
Switched to branch 'pictoral'

                        # cherry-pick the Acme::EyeDrops commit
[cpan-work (pictoral)]$ git cherry-pick d065ad152f2204295334c5475104a3da517b6ae1

                        # rebuild the module list
[cpan-work (pictoral)]$ dpan

                        # commit the new 02packages.details.txt.gz
[cpan-work (pictoral)]$ git add .
[cpan-work (pictoral)]$ git commit -m "dpan processing"

                        # push back to the mothership
[cpan-work (pictoral)]$ git push origin pictoral

And that’s it. Now point the cpan client to http://yourmachine:3000/pictoral, and you’ll get the limited mirror.

cpan[1]> o conf urllist http://localhost:3000/pictoral
cpan[2]> reload index

cpan[3]> i Acme::EyeDrops
Strange distribution name [Acme::EyeDrops]
Module id = Acme::EyeDrops
    CPAN_USERID  ASAVIGE (Andrew J. Savige &lt;asavige@cpan.org>)
    CPAN_VERSION 1.55
    CPAN_FILE    A/AS/ASAVIGE/Acme-EyeDrops-1.55.tar.gz
    UPLOAD_DATE  2008-12-02
    MANPAGE      Acme::EyeDrops - Visual Programming in Perl
    INST_FILE    /usr/local/share/perl/5.10.0/Acme/EyeDrops.pm
    INST_VERSION 1.55

cpan[4]> i Acme::Bleach
Strange distribution name [Acme::Bleach]
No objects found of any type for argument Acme::Bleach

February 19, 2010

Pythian
pythian
» cpanvote: a Perl Mini-Project

The Itch

For many, CPAN is a Canadian Prairies-sized field of modules where it’s darn hard to separate the wheat from the chaff.

While the CPAN Ratings service is the principal and official way CPAN tries to rank its distributions, for me at least, it doesn’t quite scratch the itch because . . . 

  1. not all distributions have reviews.
  2. even when there are reviews, they generally don’t answer the next question: what should I use, instead?.

The Dream

Consequently, for a while now I’ve been playing with ideas on how the rating could be improved. What I came up with so far is a very minimal system going straight for the goods, where a rating would consist of:

  1. The rating proper, which can be one of three values: “thumb-up”, “thumb-down”, or “neutral”.
  2. If you give the distribution a thumb-down (or for that matter, even if you give it a thumb up), you can recommend another distribution to be used instead.
  3. An accompanying short comment (140 characters or less so that it’s Twitter-ready. Longer, proper reviews can be done via CPAN Ratings).

Aaaand . . .  that’s it. Not exactly mind-blowing, but it’s so simple it could actually work.

JFDI, you say?

And now, since I’ve had a three-day week, I decided to give the idea a try and implement a prototype. Because I had only so many hours to devote to the project (hey, it was Valentine Day, after all), I’ve built it as a REST service. That way I didn’t have to spend any time on prettiness and, if the idea does to catch on, it can easily be grafted to a web site, IRC/IM bot, phone service, search.cpan.org (well, I can dream big, can’t I?), and so on.

The cpanvote code is on Github. It’s all rather untidy, but it’s (roughly) functional. Let’s have a little tour of the application via the example REST client cpanvote.pl included in the repo, shall we?

First, we need an account, which can be created via the client:

$ cpanvote.pl --register --user max --password min

(And yes, this way of creating users is rather brain-dead, but this is only a rough prototype, so it’ll do for now.)

Once an account is created, reviews are as simple as:

$ cpanvote.pl --user max --password min XML-XPathScript --yeah

or:

$ cpanvote.pl --user yanick --password foo Games::Perlwar --meh --comment "could use a little Catalyst love"

or:

$ cpanvote.pl --user yanick --password foo Dist-Release --neah --instead Dist-Zilla --comment "nice try, but RJS is just better at it"

For the time being, I have implemented only very simple per-distribution results, which can be queried via any browser:

$ lynx -dump http://localhost:3000/dist/Dist-Release/summary
---
comments:
    - nice try, but RJS is just better at it
    - cute
instead:
    - Dist-Zilla
vote:
    meh: ~
    neah: 1
    yeah: 1
$ lynx -dump http://localhost:3000/dist/Dist-Release/detailed
---
-
    comment: nice try, but RJS is just better at it
    instead: Dist-Zilla
    vote: -1
    who: yanick
-
    comment: cute
    vote: +1
    who: max

Test Server

For the curious, I have an instance of the application running at http://babyl.dyndns.org:3000 (cpanvote.pl –host babyl.dyndns.org:3000 …). It’s running a single-thread Catalyst with debug information, using a SQLite back-end on my home machine, which has rather pathetic bandwidth throughput, so please be gentle and don’t be too too surprised if it goes down.

Whaddya think?

This is the moment where I turn to the audience and prod them (that is, you) to see if I might be on to something, or if I’d be better to stop talking now. In other words, what’s your thought on this: --yeah, --neah or --meh?

Further considerations

Random thoughts for the next steps (assuming that there will be a next step).

  • Review accounts could potentially be PAUSE-based.
  • Give peeps the opportunity to submit tags for the module alongside their review, and let the taxonomy short itself à la Deli.cio.us.
  • We could go meta all the way and vote on reviewers as well, which could give their opinion more weight.

February 3, 2010

Pythian
pythian
» Contributing to CPAN: PAUSE Id, Bug Tracking, and Code Repositories

CPAN Wants You!

CPAN Wants You!

Want to contribute to your favorite CPAN module, or maybe create your own, but don’t have the foggiest idea how to do it? Here are a few notes, tips, tricks, and links that might help you get started.

PAUSE id

While bringing awesome street cred, having a PAUSE id is strictly necessary only if you want to maintain or co-maintain a module. If you just want to contribute code, you’ll perfectly be able to do without, as it will usually be done via patches submitted to a bug tracking system, a code repository or using good ol’ email.

Becoming a co-maintainer

Becoming the co-maintainer of a module gives you the power to upload authorized releases of the modules on CPAN. To become one, the maintainer of the module simply has to promote you as such on PAUSE.

Creating a new module

You want to create your own module? Excellent! But before you do it, make sure that you . . . 

Once this is done, go ahead: log on PAUSE and register the module.

Adopting an abandoned module

There’s a module apparently abandoned by its owner that you’d love to take over? The procedure for that is given in the CPAN FAQ, and it can be boiled down to:

  • Try to get in contact with the current maintainer.
  • If you reach him or her, make puppy eyes and ask if you can take over.
  • If they are agreeable, they will flip over the maintenance of the module to you on PAUSE, and that’s all there is to it.
  • If you tried their email addresses, poked around in mailing lists, forums and other places without any luck, wait a little bit (a couple of weeks) and contact the PAUSE admins.

Module created! How do I upload releases, now?

You upload new versions of a module via the PAUSE interface or via ftp.

Alternatively, and more conveniently, you can also use the cpan-upload script provided by CPAN::Uploader.

Bug Tracking

Contributing to a module means fixing bugs or implementing new enhancements. To find those, who are you gonna call? The Bug Tracker!

rt.cpan.org

By default, every module on CPAN has a bug tracking queue on rt.cpan.org. For example, the one for Git::CPAN::Patch is at http://rt.cpan.org/Public/Dist/Display.html?Name=Git-CPAN-Patch.

Bug reporting can be done via the web interface, or by sending an email to bug-module@rt.cpan.org (e.g., bug-git-cpan-patch@rt.cpan.org).

…or somewhere else

Sometimes, the module’s maintainer uses a different bug tracking system. You’ll typically find it mentioned in the POD, or in the META.yml:

$ curl -L http://search.cpan.org/dist/Git-CPAN-Patch/META.yml | \
  perl -MYAML -0 -E'say Load(<>)->{resources}{bugtracker}'

http://rt.cpan.org/NoAuth/Bugs.html?Dist=Git-CPAN-Patch

If you don’t feel inclined to dig into META.yml for information, there is a Greasemonkey script that will do it for you, and automatically add a link to the bugtracker (along some other goodies) on the module’s CPAN page.

…or here, there and everywhere

What if the bugs are kept on rt.cpan.org, and on the local bug tracking system of Github, and a few other places? You can either follow them manually, or you can get funky and experiment with SD, a peer-to-peer ticket tracking system that can sync with and merge the information of several online bug tracking systems. SD, it goes without saying, can be found on CPAN as App::SD.

Code Repository

Most, but not all, modules have a public code repository somewhere out there. Just like for the bug tracking system look for it in the POD, or in the META.yml. It will be displayed on the module’s cpan page as well.

$ curl -L http://search.cpan.org/dist/Git-CPAN-Patch/META.yml | \
    perl -MYAML -0 -E'say Load(<>)->{resources}{repository}'

git://github.com/yanick/git-cpan-patch.git

<singing name=”Adam West”>Na na na na, Na na na na, GitPAN!</singing>

Thanks to Schwern, all CPAN distributions now have a GitHub repository tracking its releases. They can all be found under the GitHub gitpan account, and follow the pattern http://github.com/gitpan/Git-CPAN-Patch.

Git::CPAN::Patch

Speaking of Git::CPAN::Patch, you can use its set of scripts to ease the creation of CPAN-related git repositories. It also includes scripts to send patches directly from your local repository to a rt bug queue.

October 14, 2009

Pythian
pythian
» Perl Module Dependencies: how to require the latest, and nothing less

Recently, hanekomu was contemplating how to make subsequent installs of a Task::BeLike module upgrade its dependencies to their latest version.

The idea is intriguing. It’s not something you want to do for a typical module, but it makes sense in the context of Task::BeLike. If you care enough about a module to put it in your Task::BeLike, you probably care enough to want to upgrade when there’s a new version out there.

Alas, I think hanekomu’s proposed way of doing it is flawed (mind you, the debate is still going on as of the writing of this entry, and I can very well still be proven wrong). But after some pondeferous chin scratching, I might have come with a cunning alternative to it.

Let’s say that in your Build.PL (the logic would be the same for a Makefile.PL) you have your dependencies stashed in %dependencies. Something akin to:

%dependencies = (
    XML::LibXML      => 0,          # any version will do
    XML::XPathScript => '1.42',     # 1.42 or higher
    Moose            => 'latest',   # nothing but the shiniest!
);

All we want to do, really, is to switch the latest for, well, the latest version available. Surprisingly, that something that is almost as easy to do than to say:

for my $mod ( keys %dependencies ) {
    next unless $dependencies{$mod} eq 'latest';

    require CPANPLUS::Backend;
    state $cb = CPANPLUS::Backend->new;

    $dependencies{$mod} = $cb->module_tree( $mod )->package_version;
}

Yes, that’s really all there is to it. A little further hacking later, I have incorporated the functionality to my own Task::BeLike::YANICK module. The way I implemented it, installing the module the usual way will yield no surprise (i.e., dependencies already present are not going to be updated). But if the environment variable TASK_UPGRADE is set to true, like so:

TASK_UPGRADE=1 cpan -f Task::BeLike::YANICK

 . . . then the magic is going to be unleashed (the -f is to force the re-install, if the Task has already been installed before).

Alternatively, just to know which dependencies are out-of-date, one can also extract the distribution and do a

perl ./Build.PL --upgrade
./Build prereq_report