Python Programming, news on the Voidspace Python Projects and all things techie.

Playing with the Module Type

emoticon:torch There are a couple of times I've wanted to do things with modules that don't quite work. For example, why can't we make modules callable ?

Where a module only exports one main function or class, or needs explicitly initialising after import, it would be handy to call the module object. I've just discovered that you can subclass the ModuleType, so here is a recipe that shows how to make modules callable, and could be adapted for all sorts of fun. Smile

This works almost everywhere you use modules, but there are one or two places that expect a real module rather than a subclass.

Modules are 'new style' objects: they inherit from object. The normal rule is that if an object has a __call__ attribute, you can call it. In a module, any names (functions / classes etc) are exposed as attributes of the module. However, this approach doesn't work with modules. If you define a __call__ function in a module and import it, you still can't call the module :

>>> import testModule
>>> testModule.__call__
<function __call__ at 0x00C434F0>
>>> testModule()

Traceback (most recent call last):
  File "<pyshell#7>", line 1, in -toplevel-
TypeError: 'module' object is not callable

This is because the magic methods (ones beginning and ending with double underscores) are looked up on the class, rather than the instance.

However, by creating our own module type we can get round this. We can access everything inside an existing module by using the __dict__ attribute.

from types import ModuleType

class CallableModule(ModuleType):

    def __init__(self, module):
        ModuleType.__init__(self, module.__name__)
        self._call = None
        if hasattr(module, '__call__'):
            self._call = module.__call__

    def __call__(self, *args, **keywargs):
        if self._call is not None:
            self._call(*args, **keywargs)

if __name__ == '__main__':
    import imp
    testModule = imp.new_module('TestModule')
    def __call__(*args):
        print 'Called with the following arguments:', args
    testModule.__call__ = __call__
    callableModule = CallableModule(testModule)
    callableModule('Called', 'With', 'Some', 'Args')

Easy hey. Smile

When you run this, the test at the bottom prints :

Called with the following arguments: ('Called', 'With', 'Some', 'Args')

To use the CallableModule class, define a module with a __call__ function. Import the module and initialise a new CallableModule with it: newModule = CallableModule(module). newModule is now callable, using your __call__ function.

There is a problem with the class above though. Functions defined inside the original module are copied into the new module. This means that their enclosing scope is still the original module. If they access any module level globals, and you change those on the new module, then your functions will still use the original module scope to lookup the variables.

There are a couple of ways round this. In another post I will show how we can use this to do object oriented programming, but lose both the self argument to methods and the class statement. Wink

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-18 14:01:18 | |

Categories: ,

Movable Python for Python 2.5rc1

emoticon:movpy2 Hot on the heels of Beta 3 is Movable Python for Python 2.5 Release Candidate 1.

If you've already signed up for Movable Python 2.5, then you can download it from the group page, otherwise you can sign up for it there :

The extension modules now included are :

Plus the usual pure Python modules, and the new ones provided by Python 2.5 :

  • ctypes
  • sqlite3
  • Hashlib
  • Elementree
  • wsgiref

If you encounter problems with any of these then please let me know.

Just a brief reminder that until Python 2.5 hits final release it is available for only £3.99 instead of the usual £4.99.

For a list of all the new features in Movable Python 2.0.0, What's New in Movpy 2.

For a summary of the major new features (with screenshots) see Overview of the New Features.

There is also a free trial version for Python 2.3 at Movpy Demo Version.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-17 23:59:24 | |

Categories: ,

Flattening Lists

emoticon:apple How do you flatten a list of lists in Python ?

You can't use sum on lists. I've found two one-liners that work; both of which make my head hurt slightly when I look at them. Only because I never normally use nested list comprehensions or reduce :

nestedList = [[1, 2], [3, 4], [5, 6]]
flattenedList = [item for sublist in nestedList for item in subList]
print flattenedList
[1, 2, 3, 4, 5, 6]
nestedList = [[1, 2], [3, 4], [5, 6]]
flattenedList = reduce(lambda x, y: x + y, nestedList)
print flattenedList
[1, 2, 3, 4, 5, 6]

I think that on balance the list comprehension is more readable, but at least the addition is explicit in the reduce. The list comprehension will undoubtedly be more efficient. Any cleaner solutions ?


Thanks to all those who replied. Smile

It turns out that the built-in function sum takes an optional 'start' argument which defaults to zero.

sum(sequence[, start])

You can call it with an empty list as the initial value, which works great for us.

sum(nestedList, [])

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-17 14:24:32 | |

Categories: ,

Python Snippets

emoticon:eyeballz More random news from the world of Python.

  • pyCheesecake

    pyCheesecake has had its first release.

    Cheesecake aims to help standardise Python packaging, and encourage good code practises and testing. It includes various tools, and can produces metric on aspects of python packages available on the Cheeseshop; or your own projects.

    As a bonus, it looks like they may consider using ConfigObj. Smile

  • Python Merchandise !

    Like the new Python logo ? Now you can get it on mugs, bumper stickers or T-Shirts.

    There is a Python store on Cafepress. There is also a cheaper one (for clothing items) on the interesting looking GoodStorm.

  • Python Web Frameworks on the Small Screen

    Mickey Hadick is creating a film that features intrigue, chocolate and Python web frameworks. What's even weirder is that I get a mention somehow ! The film isn't available yet, but I'm sure it will surface eventually.

  • David Mertz and Python 3000

    David Mertz is a longstanding contributor to the Python community. I haven't heard anything from him for a while, but he has a blog on IBM developer works.

    In one of his entries he recounts a talk by Guido on the schedule for Python 3k :

    our BDFL announced a pretty concrete schedule for 3.0: An alpha should be available near the beginning of 2007, with a release version before the end of the year. Python 2.6 will almost surely be released before the final 3.0, and the Python 2.x line will continue for a good while to overlap 3.0 (because 3.0 will not run all the older Python programs unmodified). Python 2.7 will probably contain some back-ports of 3.0 features, where they can be implemented without breakage; and 2.7 will also probably contain a collection of migration tools.

  • Python Assembler/Disassembler

    I'm always on the lookout for interesting Python projects, even if I can't find an immediate use for them. Byteplay is one of these.

    It can disassemble and assemble Python bytecode. If you've had a yearning to explore what Python is doing under the hood, or try any hand compiling, now's your chance. I guess it could be invaluable in debugging, but I'll probably just play with it.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-16 00:17:56 | |

Categories: ,

The Important Stuff About London 2.0

emoticon:podcast As promised, here is a post about the stuff that really happened at the London 2.0 meetup.

The meetup kicked off with a demonstration (by a cool guy who's name escapes me) of the ThoughtWorks Buildix System. This is a custom Linux distribution (based on Knoppix) that integrates Cruise Control, Trac and Subversion. This is an ideal environment for project management with continuous integration.

At work we use Subversion and Cruise Control .NET (and Subversion) for continuous integration. It basically means that as soon as anyone checks in, the full build process is run and then all the tests are run. If the tests fail we all get an email, with the name of the pair who checked in the broken build, within fifteen minutes. Smile

It works very well for us, and I can highly recommend it. If you're looking for project management tools, you should consider Buildix. What we don't have is a link between our user stories and iterations (milestones effectively) and our subversion checkins and subversion checkins. We maintain our user stories and defect list separately in a Wiki. Trac integration would allow us to link these, including putting 'Wiki Words' in our checkin comments which would automatically link to the relevant pages on the Trac project management wiki. At the moment our wiki works fine for us, but we're considering the benefit we could get from using Trac.

After the demo, the beer continued to flow and I chatted to some really cool people. Remi the creator of CherryPy, who also runs the uber-cool web-hosting company Web Faction [1], Andrej a brazilian PHP developer for etribes, Kevin from Jamkit, Edward from somewhere and a few other good folk. Oh, not forgetting Sam Newman the leader of the motley band gathered there [2]. Smile

As usual geeks are friendly folk, and I found them really easy to talk to. Unfortunately the next one is on a Tuesday. I work at the Jesus Centre (in Northampton) on Tuesdays, so I won't be able to attend; but I'll try to get to the next ones. See you there.

[1]Formerly known as Python-Hosting, but they also support Ruby, PHP etc, so Remi has broadened the name.
[2]No sign of Simon Brunning who used to arrange the Python meetups and was at the last one I went to. He hasn't posted on his blog for a while either, I wonder what he's up to Question

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-15 23:46:50 | |

Categories: , ,

Drunk Microsoft Employees, Web 2.0 & London Python Geeks

emoticon:ir_scope This is a weird post even by my standards, but hey sometimes fact is stranger than the stuff I write.

On Monday I went to the London 2.0 geek meetup in London. This is the sporadic event that has supplanted the London Python meetups. These events are arranged sporadically by Sam Newman, and encourage geeks from various different cutting edge communities to get together for beer, chat and the odd technology demo.

This one was odder than usual, or at least so I'm assured: it is only the second one I've been to.

The next post will tell you the really interesting stuff, this one is just going to tell you the weird part. Smile

Sam had mentioned in passing that 'some lady from Microsoft might turn up to do an interview'. This had completely slipped my mind when a few females arrived, looking slightly out of place amongst our unfortunately all male company. Being ever the gentleman I spoke to the youngest of the four who was looking inquisitively at us. She was an American, perhaps in her early thirties (always risky guessing a lady's age), and also hammered drunk.

She it transpired was from Microsoft, and here to interview the London 2.0 crowd, except by now there were only a few of us left. She dragged Sam and I down to the lower part of the pub (where the lighting was better) and interviewed us.

It looks like Microsoft is trying to focus on 'community' or something, and these interviews may feature somewhere on the Microsoft website sometime. If you spot them, let me know. Laughing

Of course my chances of fame and fortune may have just been seriously diminished by this post. Perhaps this isn't a bad thing though, I was hardly prepared for an interview on the London technology community (I live in Northampton), based on an event I've attended only once before.

Despite having (probably) searched half the pubs in London before finding us, Laura Foy was able to do a remarkably lucid interview. Sam Newman had some very interesting things to say, so his interview will be worth looking out for. The best I can hope for is not to have embarrassed myself too badly. I did get a plug in for Resolver Systems, and even plug this blog: so perhaps it won't be all bad if the interview ever sees the light of day.

It seems like Laura is something of a celebrity in some circles. She has her own Fan Club, and a Microsoft Show All I have to say is that her Publicity Photo does her more favours than a night on the beer. Wink

After interviewing us, Laura returned to her beer...

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-15 23:13:43 | |

Categories: ,

SPE for Movable Python

emoticon:scanner SPE [1] is an excellent IDE for Python.

Don't take my word for it, read the article SPE on Newsforge. Smile

The default IDE included with the latest version of Movable Python is now Pythonwin from pywin32. It is very easy to get SPE working though.

Simply download SPE for Movable Python.

Then copy the '_spe' directory into the 'lib' directory of your Movable Python distribution.

You can then configure a Quick Launch button, or the IDE button, to run '' or 'spe.pyw' [2].

This will work with Python 2.3 and 2.4. It will definitely not work with Python 2.2 [3].

It appears to work fine with Python 2.5, but I haven't extensively tested it.

This file can also be download from the Movable Python group pages, where if you haven't already got Movable Python it can be purchased : Movable Python Tradebit.

For a list of all the new features in Movable Python 2.0.0, What's New in Movpy 2.

For a summary of the major new features (with screenshots) see Overview of the New Features.

[1]Stani's Python Editor, the creation of Stani Michiels.
[2]There is an earlier version that does work. I can make this available if anyone wants it.
[3]Inside the '_spe' directory.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-15 22:49:12 | |

Categories: , ,

The Python Black Hats

emoticon:black_hat Well, we thought it would never happen. Confused

Up until recently I at least, thought that writing a virus in Python was a non-starter. That was until earlier this morning when emails started appearing on the py2exe mailing about AVG reporting py2exe, and any executables it generates, as a Trojan horse.

In case you're wondering, py2exe turns Python scripts into Windows executables that will run on a computer without needing Python installed. A trojan horse is a program intended run a user's computer without their knowledge, and with malicious intent.

The culprit is Backdoor.Rajump, which opens up a backdoor on computers it runs on; and apparently also acts as a keylogger. It is a Python program, created with py2exe. Surprised

The problem is now sorted with AVG: it no longer produces the spurious warnings.

This sounds terrible, but as Harald Armin Massa points out :

on the bright side: Python now gets used for serious low level system hacking. Yes, it's virus, malware, bad bad bad.

But even the bad guys cannot ignore the productivity gains of Python anymore.


Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-15 18:39:10 | |

Categories: ,

rest2web Changes

emoticon:cross There have been several fairly major changes to rest2web in subversion.

  • The restindex and uservalues for a page (the page metadata basically) can now be in a ReST comment. This means your website source documents can now be valid ReST documents as well.
  • The macro system has been changed to make it easier to use. (All the standard macros are now built-in).
  • In force mode (where rest2web will render documents with no metadata) rest2web no longer requires a config file. Smile

More details below.

There are also (at least) three more sites online that have been created with rest2web :

restindex inside comment

This change is quite simple, but it means that all your website pages can also be valid ReST documents.

You can now put the restindex and uservalues inside a ReST comment :

        # values

        # values

This means that your pages can still contain the rest2web metadata, but also be rendered by docutils individually.

Changes to the Macro System

The restweb Macro System has been simplified.

All the standard macros have been moved into the rest2web package, along with the support modules they need.

You can still supply your own macros, in the same way as before, by specifying a macros file in your site Config File. This can override any of the built in macros if you wish. Smile

Two of the standard macros (smiley and emoticon) need to be configured with the URL path to the directory containing the images. If you are using a smiley set other than the one supplied with rest2web, the smiley macro will also need to know the path to the directory containing the smiley images.

You can configure these by supplying them in a Macro Paths section in your site config file. Shown below with the defaults which are used if you don't include these values :

[Macro Paths]
smiley_directory  = ''
smiley_url = '<% path_to_root %>images/smilies/'
emoticon_url = '<% path_to_root %>images/'

A blank value for smiley_directory means use the standard smiley set. Smile

This change is fully 'backwards compatible' with the old system though. Config files and macro files from before the change will continue to work in the same way.

Config File and Force Mode

If rest2web is used in Force Mode, it can create a website from a directory (tree) of ReST documents with no metadata. It will even auto-build index pages for directories.

It now no longer requires a site config file to specify the source and target directories. Simply run -f, with no r2w.ini file.

rest2web will start looking for ReST source documents in the current directory. It will create the HTML files in a subdirectory 'html'. (Which means that text files in this directory won't be scanned.)

You might not like the default template for force mode. (On the other hand you might like it very much, it is based on the template used to create the rest2web site.)

If you want something more minimalistic, you can replace the file rest2web/defaults/template.txt with something like :

    <title><% title %></title>
    <meta http-equiv="Content-Type" content="text/html; charset=<% final_encoding %>">
    <link rel="stylesheet" href="/stylesheets/docutils.css" type="text/css">
    <% body %>

This won't include the breadcrumbs or navigation sidebar, but these are easy to add. Very Happy

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-14 10:48:36 | |

Categories: ,

Hex-Dump, IronPython & GDATA

emoticon:ir_scope The next installment of the Hex-Dump [1] series on programming with IronPython is available :

Windows Forms GUI for GDATA Reader

He is going through implementing an RSS reader, using Windows Forms and google GDATA.

[1]The creation of Mark Rees.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-12 15:58:24 | |

Categories: , ,

Two to the Power Five

emoticon:black_hat Today is my two to the power of five birthday. Smile

(My boss has pointed out that this makes it my 100 000'th birthday. I'll leave it as an exercise for you to work out for which number base this is true...)

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-12 15:49:27 | |

Categories: ,

Tests as Specification

emoticon:mirrormask At Resolver Systems we practise TDD: Test Driven Development. This is part of a programming doctrine known as eXtreme Programming, XP.

Other aspects include User Stories [1], Pair Programming and Iterative Programming. We also embrace the XP principles of YAGNI and DTSTTCPW. Smile

The pattern for Test Driven Development is: write the tests first. This is a lot less demotivating than retro-fitting tests to code you have already written and has several other advantages.

A user story will specify a new feature. When implementing that feature, we go through roughly the following steps :

  1. Write a functional test that (when it passes) will prove that all the specified aspects of the feature work. We have our own functional test framework that automates clicking on GUI components, entering text etc.
  2. Pick the simplest (or most fundamental) part of the user story, and work out where in the code this will fit in. It may be in an existing class or require a new one. Then decide on the best API for using the code you are about to create, and write a unit test for it.
  3. Write the code, fixing it until the unit test passes.
  4. Repeat steps two and three until the user story is complete.

Between steps one and two you have a failing functional test. This drives you to write the appropriate unit test that will fix the failing functional test. You know what shape the unit test will have, because your functional test has specified it.

Between steps two and three you have a failing unit test. You know what shape the code will have because the unit test specifies it.

The crucial part is decide on the best API for using the code you are about to create. Writing tests makes you think about how the code ought to work, not just growing it organically. Often this will involve changing your existing code to accommodate the new feature in the best way possible, rather than just bolting it on. Writing unit tests strongle encourages you to write loosely-couple code, and by forcing you to think about testing it causes you to write code in smaller modular units. This changes the way you write code. In my opinion, drastically for the better.

The major win though, is that you have full test coverage, both with unit tests and functional tests. I can't tell you how pleasant it is to be able to make major changes to a project internals, yet know that because you have hundreds of passing tests; the chances are that you have managed to fix everything you broke. Smile

As a consequence our code is completely specified by our unit tests. Every unit of functionality of our code should have a corresponding unit tests. Usually one test file per class, and usually one class per file, but we're not anal about this.

Andrzej is a member of our team who has had precious experience of extreme programming, and is a great enthusiast of agile programming methodologies in general but with an unfortunate predilection for Ruby [2].

He recently discovered this :

Ruby Testing Techniques (pdf)

The important parts are slides 3, 4 & 5.

It describes how, by using a naming convention for your tests, you can derive an API and class specification from your tests. As well as being useful, you can then evaluate your code coverage and specification.

The example they used on the presentation is :

def test_should_supply_constraint_indicating_whether_review_is_awaiting_retailer_action
    assert_equal "Status IN ('A')",

The specification from this yields :

review_status should:

  • supply constraint indicating whether review is awaiting retailer action
  • submit review and store submission date

We're evaluating this to see whether this will be useful for us, and if so we will create a tool to do this from Python unit tests.

The hardest part is creating test naming conventions that will provide useful information.

[1]Resolver is a new program, so we have no customer. Instead of an onsite customer we 'use' one of our non-executive directors (who is an expert in the field we work in) as a customer representative. He prioritises our user stories and defects for our iterations.
[2]Hopefully I'll soon be able to publish a review of the Manning Ruby for Rails book.

Like this post? Digg it or it.

Posted by Fuzzyman on 2006-08-12 15:10:38 | |

Categories: , ,

Hosted by Webfaction