Python Programming, news on the Voidspace Python Projects and all things techie.


emoticon:file1 It looks like Fredrik Lundh has been working on my idea for a fast extension language for Python, using Python syntax :


Of course he's been working on it since long before I had the idea... Smile

Fredrik has been doing some very interesting experiments with Pyrex recently, so definitely worth staying tuned.

Actually, Pytte doesn't look like a Python extension language, so maybe my idea still has some merit. I had another wild idea, which I'm equally likely to try implementing. Wink

I've not used the Python C API, how about embedding a Python interpreter in an extension module ? It could be used instead of threading (although implementing the callback would be interesting, but you can release the GIL around calls into the embedded interpreter).

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-26 00:09:14 | |

Categories: ,

Python, Metaclasses and Overloaded Methods

emoticon:men One of the nice features of C# is that you can provide multiple overloads for methods. You provide multiple methods (including constructors of course) with the same name, but with a different type signature for the different overloads.

Because the computer knows the type of your arguments at compile time, it knows which of your overloads to call.

The equivalent in Python is to do type checking of arguments in the body of your method, which can get ugly or feel unpythonic; as Ned Batchelder reported recently.

I've been contemplating metaclasses. They felt like deep black magic, but I know that all they do is customise class creation and I've been itching for a reason to try them out. Upon further investigation the basics are actually quite straightforward.

Create a class that derives from type. The __new__ method takes the arguments (cls, classname, bases, classDict). classDict is a dictionary of the attributes (including methods) of the class.

Your __new__ method should end with a call to the type.__new__ method with similar, but possibly modified, arguments.

Once I'd worked out a way to provide method overloading using a metaclass I had to try it. It's a little rough around the edges, but it works. Smile

import inspect

class OverloadedError(Exception):

def GetMethodName(name):
    digit = name.rstrip('_')[-1]
    return ''.join(name.rsplit(digit, 1))

class MethodOverloading(type):

    def __new__(cls, classname, bases, classDict):
        newDict = {}
        overloads = {}
        for attr, item in classDict.items():
            if callable(item) and attr.rstrip('_') and attr.rstrip('_')[-1].isdigit():
                newName = GetMethodName(attr)
                if newName in classDict:
                    raise OverloadedError("Method '%s' is also overloaded." % newName)
                overloadList = overloads.setdefault(newName, [])
                overloadList.append((attr, item))
                newDict[attr] = item

        storedOverloads = {}
        newDict['_overloads'] = storedOverloads
        for methodName, overloadList in overloads.items():
            thisMethod = {}
            for entry in overloadList:
                name, method = entry
                (args, _, __, defaults) = inspect.getargspec(method)
                args = args[1:]

                if len(args) != len(defaults):
                    raise OverloadedError("Overloaded method '%s' has non-keyword arguments." % name)

                thisSignature = []
                for arg, val in zip(args, defaults):
                    if not isinstance(val, type):

                if not thisSignature:
                    raise OverloadedError("Overloaded method '%s' has no types specified." % name)

                thisSignature = tuple(thisSignature)
                if thisSignature in thisMethod:
                    raise OverloadedError("Overloaded method '%s' has a conflicting signature "
                                          "with another method." % name)
                thisMethod[thisSignature] = method

            storedOverloads[methodName] = thisMethod

            def typeCheck(self, *args, **keywargs):
                thisMethod = self._overloads[methodName]
                signature = tuple([type(arg) for arg in args])

                realMethod = thisMethod.get(signature)
                if realMethod is None:
                    raise OverloadedError("No overload matches '%s' signature for method '%s'."
                                          % (tuple(signature), attr))
                return realMethod(self, *args, **keywargs)

            newDict[methodName] = typeCheck

        return type.__new__(cls, classname, bases, newDict)

Using it is simple. Here's an example of a class with two possible constructors, one takes three integers, the other takes a string :

from methodOverloading import MethodOverloading

class SomeClass(object):

    __metaclass__ = MethodOverloading

    def __init1__(self, x=int, y=int, z=int):
        self.test = (x, y, z)

    def __init2__(self, string=str):
        self.test = string

This creates a class that has a single __init__ method, and no __init1__ or __init2__ methods. When you create an instance, the faked up __init__ does the type checking for you and calls the appropriate method under-the-hood. Smile

For example, this :

a = SomeClass(1, 2, 3)
b = SomeClass('Bright Red')
print a.test, b.test

should print (1, 2, 3) Bright Red.

Because it is a relatively quick hack (when I should be doing other things), there are a few rules and restrictions about how it works. Most of these could be relaxed by some extra work.

In your overloaded method declarations all arguments must be keyword arguments (except for self).

Those default arguments are evaluated from left to right, arguments that are types are collected as the method signature until a non-type argument is found.

For example, in the following 'example1' has a signature of (int, int, float), 'example2' has a signature of (int,). Calling 'example(3)' calls 'example2' under the hood. On the instance the method 'example' exists, but 'example1' and 'example2' don't :

def example1(self, w=int, x=int, y=float, z=3)

def example2(self, w=int)

When calling the method the keywords must not be used. Untyped arguments must be passed in as keyword arguments. Not following this pattern will cause an exception to be raised or the wrong method to be called. This restriction can be removed, but only by annoyingly fiddly code, so it will have to wait for the next revision.

Other thigns still to do include :

  • Method attributes (like docstrings etc) are not yet preserved (which of the overloaded docstrings to use ?).
  • Add name mangling to the '_overloads' attribute.
  • Currently a maximum of ten overloads per method (0-9).
  • Allow providing a default where no overloads match: a method with no typed arguments which is currently not supported.
  • Allow tuples of types.
  • You currently can't have an overloaded method which takes no arguments.

You can download this as a Python file, along with some basic unit tests :

There is a runtime cost of course. Calling an overloaded method involves an extra method call plus a bit of jiggery pokery.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-25 21:45:36 | |

Categories: , ,

IronPython, Subprocesses and Switching Off the Internet

emoticon:mirrormask Over the last couple of days, a colleague and I have been adding the final finishing touches to a user story. Part of this means sensibly informing the user when a connection to a server fails, rather than unceremoniously crashing.

The relevant line of the functional test looks like this :

# Harriet pulls out Harold's ethernet cable

So how do we simulate this from our test framework ? Smile

(Functional tests mustn't delve into the application internals to handle this sort of thing, they should be 'black box' tests that test functionality and are not dependent on implementation. That means we can't simply find the relevant connection object and close it.)

After a few abortive attempts Andrzej thinks he has come up with something that works. It also illustrates launching a sub-process from IronPython. It uses the System.Diagnostics.Process class and the ipconfig command line tool.

import clr

from System.Diagnostics import Process

def disconnect():
    runProcess = Process()
    runProcess.StartInfo.FileName = "ipconfig"
    runProcess.StartInfo.Arguments = " /release *local*"

def connect():
    runProcess = Process()
    runProcess.StartInfo.FileName = "ipconfig"
    runProcess.StartInfo.Arguments = " /renew *local*"

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-24 15:29:44 | |

Categories: , ,

Google Co-op Search for Python Programming

emoticon:computer Google have just unveiled a new service: Google Co-op.

This allows you to create and customise search engines for specific subjects, a bit like Skimpy my Python search engine.

Given that their technology might be marginally better than my homebrew CGI, I've created a Python Programming Search Engine.

This allows you to search the web for Python related topics, giving priority to results from the 1500 or so domains that I've pulled in from Skimpy. It has three 'annotations', letting you narrow your search results for pages relevant to 'algorithms', 'tutorials' or 'examples'.

If you have any suggestions for this, perhaps further annotations, or would like to help adding sites and tuning the results (adding annotation labels for specific sites) then get in touch or use the edit this search engine link to volunteer as a collaborator.

Here is a quick link to the Python Programming Search Engine :

The html to add this to your own site is :

<!-- Google CSE Search Box Begins -->
<form id="searchbox_018413290510798844940:k69bxcfofe0"
  <input type="hidden" name="cx" value="018413290510798844940:k69bxcfofe0" />
  <input name="q" type="text" size="40" />
 <input type="submit" name="sa" value="Search" />
<input type="hidden" name="cof" value="FORID:1" /
<script type="text/javascript"
<!-- Google CSE Search Box Ends -->

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-24 11:19:43 | |

Categories: ,

Movable Python Docs and Movable IDLE Homepage

emoticon:movpy2 I've finally got round to updating the Movable python Documentation. Hmmm... at least I'm part way through the update.

There are still several glaring omissions, but there is enough up there to be useful. I can complete the rest in my lunch breaks. Wink

There are also now homepages for Movable IDLE and the Mega-Pack.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-23 00:59:14 | |

Categories: ,

Selling Pair Programming

emoticon:mobile At Resolver Systems we use various development practises taken straight from mainstream Extreme Programming.

Although some of these are principles that you learn fairly quickly as a programmer anyway, like YAGNI, both Test Driven Development and Pair Programming were completely new to me.

I'm thoroughly sold on both of them. Smile

Test Driven Development produces code that is not only better tested, but in my experience better structured and with a better API. With a test framework in place, it is also easier to refactor. It does feel like harder work than not writing tests at all, but the benefits become obvious fairly quickly as soon as you embark on a bug hunt.

For the programmer, so long as you like people and the other members of your team aren't awkward s*ds [1], pair programming delivers a more obvious benefit. Pair programming is so much more fun than programming by yourself. Smile

It also feels like it delivers a much better product, but feelings wouldn't be enough to sell it to a team manager concerned that switching to pair programming is about to half his team's productivity. I've been thinking (a bit) about how to rationalise the benefits of pair programming, and whether I could sell it as a development technique to someone who is sceptical.

I certainly much prefer it, and there are some obvious benefits that it delivers. Proponents of pair programming say that net productivity is more than double, if you factor in that code produced by a pair is generally less buggy. This sort of claim needs backing up of course [2].

As well as the feel good factor, the obvious benefits that I see in practise at Resolver are :

  • Less Distraction

    Sometimes, despite your best intentions, it's hard to focus on the job in hand. Joel Spolsky discusses this in his article Fire and Motion. Smile

    Sometimes I just can't get anything done.

    Sure, I come into the office, putter around, check my email every ten seconds, read the web, even do a few brainless tasks like paying the American Express bill. But getting back into the flow of writing code just doesn't happen.

    [...] What drives me crazy is that ever since my first job I've realized that as a developer, I usually average about two or three hours a day of productive coding.

    Pair programming almost completely eradicates this problem. Ok, so if both of you are in a distracted mood then it is possible to conspire together to avoid coding, but basically if two of you are sat together to program; that's what happens.

  • Code Review

    I've not worked in a commercial programming environment other than Resolver, but I know that many companies handle checking code quality through a process of code review. This will either be done before check-in, or periodically (at least in theory).

    Pair programming means that all code has been worked on by at least two programmers, this at least reduces the chance of major problems being checked in. Test driven development, where all tests must pass before code can be checked in, and continuous integration where the tests are automatically run after every checkin, reinforce this.

    There is another way that pair programming raises overall code quality.

  • Code familiarity and ownership.

    Part of pair programming is regularly rotating programmers. This means that everyone in a team becomes familiar with the whole codebase.

    Everyone in the team knows the code, and feels a sense of ownership for it. You are less likely to get patches of the code that no-one dares touch because 'only Bob has any idea of how it works'. You also get more pairs of eyes going over the code.

    Additionally, we all have our own favourite programming tools and techniques. Some parts of the code may require particular idioms that individuals may be unaware of or unfamiliar with. Regularly changing pairs means that this knowledge is shared throughout the team.

    Until I worked at Resolver I had never used databases. By pairing with other team members I've just been the lead on a couple of User Stories allowing Resolver to interact with databases in various ways.

  • Mentoring

    Integrating new team members can be difficult in an ongoing project. Pairing means that new members can quickly be brought up to speed without having to spend months confused or working on non-productive tasks. Perhaps only weeks instead. Wink

    Within the last three weeks we've taken on two new developers at Resolver. They are both experienced programmers, but neither had worked on a similar project before. (Both had .NET familiarity, but one was primarily a C++ programmer with little Python experience, the other has used Python a lot but mainly done web development.)

    After a few weeks of pairing with other team members on different parts of our application they have just got to the point where they can pair with each other on a user story.

  • Catching Mistakes

    Whilst pairing one team member 'drives': has control of the keyboard and monitor. (Regularly switching the driver is another important part of pair programming.) Although you will both be actively engaged in creating the code, the one who isn't driving is much more likely to spot typos, mistakes and errors in logic.

However, the main reason to program in pairs is that two heads think better than one. In fact two heads think very differently to one.

Joel Spolsky, in the article mentioned above, says :

Once you get into flow it's not too hard to keep going.

You know the flow, those times when you wonder why flames aren't coming out of the keyboard, when nothing else matters but the code and time seems to disappear. Very Happy

Well you don't get the same flow when programming as a pair, but you can definitely get into a zone together, and you form ideas between you. This is a very satisfying process, and much more fun than coding in isolation.

The benefit comes from a pair looking at any problem from two slightly different viewpoints, with different experiences and approaches. The areas where your understanding is the same gives you the ability to communicate, the areas where your understanding is different gives you the ability to innovate.

Here's a simple practical example. Andrzej and I were working together on a problem last week. I metioned out loud one possibility, which Andrzej recognised as being a plausible solution. This is something that neither of us are likely to have tried working on our own.

I can't fully articulate the dynamics of working in pairs, and I'm not convinced that I'm (yet) working at one hundred percent capacity as much as I'd like to, but I'm learning a great deal and will continue to examine pair programming with a semi-critical eye in order to understand it.

This has deliberately not been an attempt to exhaustively expose the pros and cons of pair programming [3]. I'm biased, but only because my experience of pair programming has been so positive. This is my attempt to evaluate pair programming based on my own observations.

[1]In some organisations, and for some individuals, this may be a major hurdle of course...

The wikipedia entry for Pair Programming quotes this (premium content) Economist Article :

"Laurie Williams of the University of Utah in Salt Lake City has shown that paired programmers are only 15% slower than two independent individual programmers, but produce 15% fewer bugs. Since testing and debugging are often many times more costly than initial programming, this is an impressive result."


Hey, if you want to read a negative opinion of pair programming, you could read Good Agile, Bad Agile by Steve Yegge :

Take Pair Programming, for instance. It's one of the more spectacular failures of XP. None of the Agileytes likes to talk about it much, but let's face it: nobody does it.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-22 18:17:57 | |

Categories: ,

Python Jobs with the Hidden Network

emoticon:test_tubes If you've been reading The Daily WTF [1] recently, then you might have heard of The Hidden Network. This is a job advertising network using :

over fifty, high-quality on-line publications ("blogs"). These blogs are written by experts in their field and read by professionals with a passion for their career and industry.

Hey, their words not mine. Smile

In their wisdom, they've accepted me into the hidden network. If you're looking for a new job, then you should check out the Hidden Network Job Board. There should be no agencies looking for CVs advertising here, only decent employers with (hopefully) a woeful absence of pointy haired bosses.

If you're looking to hire a new programmer, then this network is the ideal way to find candidates. You can post job adverts via the job board.

The job adverts appear here on the Techie Blog, you can see them on the sidebar [2], and the other sites in the network. It would be good to start to see some Python programming jobs appear there [3]. The hidden network provides an excellent way to get your job advert in front of good candidates (hey, especially if they are readers of my blog).

[1]And if not, you should have been. Razz
[2]They are shown using geo-targeting.
[3]Python isn't currently listed in the job filters, but I've been promised it will appear in the next update, along with 'some other neat job-browsing features'.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-22 14:35:18 | |

Categories: , ,

Latest Toys

emoticon:firefox I moved house about six months ago and it wasn't until last week that I got round to unpacking my inkjet printer. Unfortunately they don't cope with being left for this long with their ink cartridges left in. Rolling Eyes

Despite not unpacking it for six months, I really do need a printer. In the UK the Computer Shopper Magazine has a good reputation for impartially reviewing computer hardware. But why is it that almost none of the models they review can actually be purchased, and none of the models available have been reviewed. sigh

In the end I found a Mono laser multi-function printer, the Samsung SCX-4200 which did have a good review and was available. The wife is out shopping, when she comes back with some paper I'll let you know if it works. Smile


It works fine so far...

My other latest toy is a Nintendo DS Lite. I know, I know.

My latest toy

There are now five of us at Resolver with these (Andrzej is the only one who hasn't succumbed to the peer pressure so far). I'm determined to beat Christian at Mario Kart in the next few days. Giles of course claims that the only reason he's bought one is for the Brain Training game. Wink

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-21 14:07:28 | |

Categories: ,

Video Podcasting and Youtube

emoticon:drive I've never got into video podcasting, I've not even got as far as podcasting. When I sit at my computer I tend to have a film or an episode of Stargate (up to season 8 so far) on in the background.

This means I have a special abhorrence that I reserve for programs or web-services which don't provide reference documentation, but instead provide screencasts. Mad

However, there is a lot of fun stuff on YouTube, despite it recently becoming part of the evil empire. Laughing

These are two of my latest 'finds', actually they've both been sent to me by friends. I just don't have the patience to trawl through it. I like the way you can embed videos in web pages though, thusly :

Stormtroopers Cop Show


White and Nerdy

Of course the best of YouTube, and lots more live content, can be found on SlippedStream. Wink

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-21 13:51:26 | |


New Releases

emoticon:target It's been another hectic week for new releases in the Python world.

  • Python 2.4.4 is now out. This contains a host of bugfixes, many of them exposed by the Coverity report on Python. A new release of Movable Python will follow soon.
  • wxPython - the GUI toolkit. This looks like a major new release, including the removal of several previously deprecated features. It's nice to see projects updated, let's hope it doesn't break too much code.
  • PythonCE for Python 2.5 - Python for mobile devices. pySQLite - is now available.
  • SPE 0.8.3b - the Python IDE. This is the first release for a long time. This release brings Python 2.5 and wxPython 2.7 compatibility, although there seem to be reported issues with wxGlade.

Of course the most important release of the week is rest2web 0.5.0. Wink

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-10-21 13:40:11 | |


Hosted by Webfaction