IronPython Tutorial and Testing Desktop Applications at PyCon (US) 2009

The decisions are in from the hardworking, much loved and generally good looking program committee for PyCon 2009.

Actually, we're not all that hard working: this year I helped with the talk reviews. I'll steal Brett's disclaimer though "I am on the program committee, but I was not allowed to vote on my own proposal nor know its status". [1]

The esteemed Mr Tartley and I will be giving an IronPython tutorial. This will be based on the one that Menno, Christian and I gave at PyCon UK this year - but I'll rename Twatter (because apparently it's much ruder in US English than it is in UK English) and we also learned a lot whilst giving the tutorial the first time round. It covers Windows Forms, databases, network and web requests, threading plus using .NET classes and should be a lot of fun [2].

I'll also be giving my talk on the Functional Testing of Desktop Applications. This is a 45 minutes talk about testing techniques and development practises for testing GUI applications. There is a lot of discussion around the functional testing of web applications; but much less around testing traditional user interfaces.

Being on the review committee was interesting (and it was great to see so many submissions on testing related subjects - the Python community is definitely going in the right direction with its focus on automated testing). Although the review system is not perfect, and very susceptible to the human flaws and biases of the reviewers (myself included), we had a very good bunch of reviewers - with enough wildly different opinions to balance out individual biases. Despite our different ways of thinking we were all committed to creating the best conference that most represents the spirit and interests of the Python community. The software infrastructure created by Doug and team is actually very good and was a very useful tool in making the difficult decisions about how to narrow down the number of talks to fit the schedule.

After the first couple of rounds through we had about sixty talks to fit into just over fifty slots (the invited talks, panels and forty-five minute talks had already been dealt with by that point). By this stage all the talks in the system were very good, and so we had to make the judgements based on reducing duplication of topics and maximising the range of subjects covered. It took over two hours to do! In the end the total acceptance rate was over 60%. There were several great talk proposals that didn't make it purely because we had other good proposals that covered the same ground.

Special thanks go to Ivan Kristic who acted as chairman, herded the rest of us cats, and made some of the toughest decisions.

I'm really looking forward to PyCon - see you there!

[1]

I tried my hardest to hack Doug's system for preventing reviewers seeing the status of their own talks, but was pleasantly surprised to not be able to find my way around it. You could infer its general state by counting up totals for the different categories (if there were 50 talks in the 'all positive' category but only 49 showing then you can infer that your talk only has positive votes) but not see any specifics.

[2]	Although tutorial givers have been notified of acceptance a final list has not been announced. I think there are still some issues around room allocations - so it is still possible that this tutorial might be pulled.

---

EuroPython 2009: PyCon UK on Steroids

I am on the committee for organising PyCon UK again this year (yes - I was very bad in a past life), but it will be a bit different this time (and 2010 as well). We are hosting EuroPython, so PyCon UK has morphed into something a bit bigger...

EuroPython will be held in Birmingham, UK from Monday 29th June to Saturday 4th July 2009.

We start with two days of tutorials, will have some very interesting keynote speakers (more details to come) and finish with sprints. We are already announcing the call for papers - come and speak at EuroPython 2009!

Talk & Themes

Do you have something you wish to present at EuroPython? Go to http://www.europython.eu/talks/cfp/ for this years themes and submissions criteria, the deadline is on 5th April 2009.

Other Talks, Activities and Events

Have you got something which does not fit the above? Visit http://www.europython.eu/talks/

Help Us Out

We could use a hand - any contribution is welcome. Please take a look at http://www.europython.eu/contact/ .

Sponsors

An unique opportunity to affiliate with the prestigious EuroPython conference! http://www.europython.eu/sponsors/

Spread the Word

Improve our publicity by distributing this announcement in your corner of the community, please coordinate this with the organizers: http://www.europython.eu/contact/

General Information

For more information about the conference, please visit http://www.europython.eu/

Looking forward to seeing you!

---

The Resolver One Spreadsheet Challenge - win $17000

Resolver One is the Python powered spreadsheet created by Resolver Systems, and the project I've been working on for nearly three years now.

Resolver One is a highly programmable spreadsheet program built with IronPython. It is capable of creating powerful spreadsheet systems, but is easy to program with Python and .NET libraries.

We're convinced that Resolver One allows people to create astonishing things that simply aren't possible in a traditional spreadsheet environment. And we want to prove it. Enter the Resolver One Spreadsheet Challenge.

The Resolver One Challenge

We're so confident about the revolutionary potential of Resolver One that we've set up the $25,000 Resolver One Challenge. Every month between now and May, we will be giving away $2,000 for the best spreadsheet we receive. And in late May, we'll be handing over $15,000 for the best of the best. Let your imagination run wild

Build a blogging engine directly in Resolver One. Hook Resolver One up to existing .NET or Python libraries in unusual ways. Build the game of life, or a Mandelbrot viewer directly into the grid. Get Infocom adventure games running inside a spreadsheet; or for that matter, have a conversation with Eliza. Make a music player that does visualisations in the cells.

Or something more businesslike?

Use the sophisticated web integration to pull of stock prices, or integrate your spreadsheet with Google Maps. (Perhaps you could build a spreadsheet that plots a map, showing in which part of the country stock or house prices are rising or falling the most.) Build an election predictor (and use a combination of Monte Carlo analysis and the web front end to make it really special).

In other words: Resolver One gives you the tools, you just need to use your imagination, and your Python and spreadsheet skills!

Getting Started with Resolver One

Resolver One is free to try and for non-commercial and Open Source uses.

• Download Resolver One

To get you started with Resolver One we have a new tutorial. It takes you through all the major features, with examples to try:

• Resolver One Tutorial

When you install Resolver One you get a bunch of sample spreadsheets that also illustrate the core features.

In other Resolver One news we've just announced the major features that will be coming in Resolver One 1.4:

• The preliminary Resolver One 1.4 feature list

The main feature of course is that we're switching to IronPython 2. There are some other interesting additions though, including a first cut of being able to use numpy inside your spreadsheets!

• Alpha support for numpy! Our Ironclad project has been working toward this for over a year now, and in Resolver One 1.4 the work will start to pay off.
• The first steps towards what we call "model-side scripting". In Resolver One 1.4, you will be able to set a cell's formula from your button click handlers. While this is a small change in itself, it has big consequences - and should be a big help for people trying to write CRUD applications in Resolver One.
• A major upgrade to our support for statistical calculations, with 24 new statistics functions, from AVEDEV to VAPR.

We also have exciting news about Ironclad itself. Almost a 1000 numpy tests now pass when used on IronPython and we are working on performance and getting PIL (the Python Imaging Library) to work. More details soon...

---

Python 3: An end to Unicode Problems?

One of the major changes in Python 3 is that there are now only Unicode strings. There is a bytes type for when you are dealing with binary data, but when you are dealing with text it is always Unicode. There is a corresponding change to the Python I/O system so that files opened in text mode return decoded strings and in binary mode they return bytes.

Compared to many languages the support for Unicode in Python 2 is pretty good. So long as you obey the straightforward rules that you always decode data to Unicode when you receive it, and encode when you need to write it (always specifying the encoding of course) then Unicode strings in Python 2 work very well. The problem is that by having the default string type (str) as a byte string the language is virtually begging you to write code that falls over as soon as it has to cope with non-ascii characters or multi-byte encodings. Even if you are careful it is very easy for byte strings to slip through (particularly when dealing with third party libraries that aren't as fastidious as you). As soon as you have a mix of Unicode and byte strings you risk an implicit decoding operation, which will use the default encoding (probably ascii) and possibly failing loudly.

What Python 3 brings to Python is internal consistency. You can no longer cause encoding related errors by adding Unicode and byte-strings. Whenever you have text you have strings that are represented internally as Unicode. No need to worry about that any more. What it doesn't mean is that you don't have to worry about encodings, or even cross-platform issues. In fact it is likely to be a cause of some fun cross-platform issues.

What Python 3 does by default is read and write using the platform default encoding. On a Mac that will be 'mac-roman' and on Windows in the UK it will be something like CP1252. Very different encodings. This means that if you have program files stored as text that were created on one system and then read them from another system, by default they will be decoded with a completely different encoding.

Take a file like this created and then read on the Mac:

If we take the same file and then read it with Python 3 on Windows:

And this is because:

The answer to this is that in fact now that we are always decoding when we read a file in as text (or writing text out to a file), it is even more important that we are aware of the encoding being used. If we are only ever using this file on the current system then maybe we don't need to worry, but if we ever need to read data that might have been created on another system then we had better know what encoding was used.

The solution: the open function takes an optional encoding parameter:

---

Private Members in Python and C#: You Don't Really Need Them and You Can't Really Have Them

A while ago I wrote a mildly controversial blog entry about the way the Python community responds to certain questions: You Don't Really Need It

One of the issues I talked about was the fact that Python has no concept of private members. It does provide name mangling when your member names start with a double underscore. This is intended to be used for avoiding name collisions in subclasses. In Python the strong convention is that member names that start with a single underscore are private and shouldn't be used outside the class.

This worries developers who come from a background in statically typed languages, where the default is usually for everything to be private. As well as the nervousness that naturally comes from having your privates exposed, some feel that this violates encapsulation and means that Python isn't truly object-oriented. Of course Python encapsulates data and methods as objects, it is merely a translucent encapsulation. In practise this just doesn't seem to be a problem.

In Python you can (ab)use name mangling (or other tricks like data access through closures) that make it inconvenient for developers to access private data and methods. However, Python has such simple to use introspection capabilities that these are almost always trivial to get around.

However, most static languages also provide introspection. In C# this is called reflection, and though it is damn inconvenient to use, it is there and can be used to access / modify private members. In fact, not only is it available, it is also the recommended way to overcome certain difficulties.

Take the following situation that Christian and Orestis encountered at work the other day. By default when making web requests with WebClient.DownloadString it raises an exception if the server sends non RFC compliant headers. Unfortunately such rarely-encountered sites as the Yahoo finance sites do this. You can specify in your app.config file that you are happy to accept non-compliant headers (useUnsafeHeaderParsing), but there is no direct API to set this. If you need to change this setting at runtime you use something similar to this wonderfully horrible piece of reflection (shown as IronPython code [1]):

This modifies the same hidden setting that would be set from app.config.

So in C# your privates are also exposed, all you do is make life more painful for the developer who really does need access to them. Here's to not making life painful for us poor beleaguered developers.

To end with, here is a quote from an excellent article (not at all about Python) on how a strong focus on testing (essential for any developer with a genuine commitment to quality) challenges your conventions:

"3. Private makes less sense than it used to... Get used to living in a more public world."

---

[1]	Adapted from here.

---

Integers Can't Handle Floats

This surprised Glenn and me at work the other day:

Python integers don't know how to subtract floats from themselves! This is because the result needs to be a float. Instead, when you subtract a float from an integer, the integer returns NotImplemented, which causes the Python interpreter to call the __rsub__ method on the float - that then does the subtraction and returns a new float.

The reason that it caught us out is that in Resolver One we have some spreadsheet objects that need to behave like numeric objects under certain circumstances (for compatibility with Excel you need to be able to treat cell ranges that only cover a single cell as if they were the value of the cell they contain).

We were taking a shortcut to achieve this. Rather than implementing all the numeric protocol methods by hand we attached a bunch of functions that delegate to the corresponding bound method of the object they hold as values. In IronPython 1 this worked fine, but IronPython 2 has got more compatible with CPython - and when we call the bound subtraction method of an integer with a float it calls the float __rsub__ with our cellrange. Unsurprisingly floats don't know how to do arithmetic with cell ranges...

The nice and simple (and cleaner) solution was to stop delegating to bound methods and instead use the numeric functions from the operator module that provide full Python semantics for numerical operations.

---

Python 2.6 and Executable Zipfiles

A new feature that was quietly sneaked into Python 2.6, without the fanfare it deserves, is the ability to distribute Python applications as executable zipfiles. Python has long had support for importing modules and packages from zipfiles - through the oh-so-badly-needed-in-IronPython [1] zipimport.

What is new is the ability to make zip archives executable. If you call the Python 2.6+ (or 3.0+) interpreter passing in a zip file instead of a Python file - the interpreter looks inside the zip file for a Python file named __main__.py (at the top-level) and executes it. The zip file can also contain all the (pure-Python only) modules and packages your app depends on.

This is a great way of distributing applications as a single file. The nice thing is that the Python interpreter doesn't depend on the extension to recognise zipfiles, instead recognising them automagically. This means that on Windoze you can give these archives a new extension (perhaps '.pyz') and associate them with Python 2.6 - allowing Windows to execute them automatically when you double click on them in explorer. I think, but am not 100% certain, that the zipfile specification is flexible enough that you could also prepend a pound-bang ('#!') interpreter line to make them executable under Mac OS X and Lunix type platforms.

michael$ echo > __main__.py "print 'Hello world'"
michael$ python __main__.py
Hello world
michael$ zip test.zip __main__.py
  adding: __main__.py (stored 0%)
michael$ python test.zip
Hello world

UPDATE: Floris Bruynooghe notes in the comments that you can add a hash-bang line to a zipfile and make it executable:

$ cat > __main__.py
print('hi there')
^D
$ zip test.zip __main__.py
adding: __main__.py (stored 0%)
$ cat > hashbang.txt
#!/usr/bin/env python3.0
^D
$ cat hashbang.txt test.zip > my_exec
$ chmod +x my_exec
$ ./my_exec
hi there
$

---

[1]

Both zipimport and ultimately zlib module support are needed for setuptools compatibility in IronPython. setuptools is ever more becoming the defacto way to package large Python applications (like Turbogears for example) and without it they fall at the first hurdle when attempting to use them from IronPython. There are encouraging signs that the Microsoft Dynamic Language guys are on the case.

---

Join the Glorious Python Programming Revolution

Ars Technica has a new article on learning Python:

• Getting a grip on Python, 6 ways to learn online

The article itself is kind of meh, merely linking to six of the most well known online Python tutorials, but what is interesting is their rationale for promoting Python:

"Recently, Google has stepped up its presence in the cloud computing arena. Google's new App Engine (aka "AppSpot") lets you design and run web applications using Google's existing infrastructure."

"At this time, App Engine uses Python as its primary programming language. Although Google is investigating other languages for future releases, if you want to get started with App Engine, you'll need to first master the Python scripting language."

Overwhelmingly the best part of the article is the image they use...

Ars Technica have a real problem with puns when doing articles on Python. Take their article introducing Python 3 for example:

• Python 3 Slithers onto the Scene, Sheds Backwards Compatibility

Thissss article is pretty good, but as expected Python 3 is causing controversy both inside and outside the Python community. People wonder if the benefits of breaking backwards compatibility are worth the potential confusion it causes. The very best response I've seen is the following blog entry by James Bennett. It kicks off with a fantastic introduction that has nothing to do with Python but should be read by programmers and business managers of every strain.

• Let's Talk About Python 3.0

Back to Ars Technica, they will always have a place in my heart. They've done a fantastic series of articles on the history and technology of the Amiga computer (the first 'desktop' computer with a pre-emptive multitasking operating system being one of the jewels in its crown of glory). If, like me [1], you are nostalgic for the beautiful usability of this machine (both as a user and a developer) then you'll love this series.

[1]

I had a great time playing with 68k assembly language on the Amiga. Building C structures and calling into the operating system. Both the Amiga OS and Motorola 68000 instruction sets were very elegant. It was Amiga OSes inability to support memory protection (and thus virtual memory) that was (and is) a big cause of its downfall. The latest release of the beautiful-looking-and-not-quite-dead-yet Amiga OS 4.1 still doesn't have default support for virtual memory and only runs on Power PC architectures - no x86 support (and no supported capability to run under virtualisation). Good luck with that.

---

Falling in Love with Sphinx (and redeeming doctest)

Sphinx 0.5 (half a Sphinx) has just been released. This is timely as one of the most valuable things that I learned at PyWorks (in the hallway track) was how to use Sphinx. Chris Perkins (who has some patches that will be integrated into the next release) showed me how to get started. My conclusion - Sphinx is frickin awesome!

Sphinx is the documentation system used to build the Python documentation. It takes reStructured Text source documents and can build HTML / Latex output, with reST directives for classes / functions / methods, cross-referencing between pages, auto-generation of indexes, online and offline search (very clever) for the HTML, syntax highlighting and more.

I think we can get a lot of benefit from Sphinx at Resolver Systems. We already use a homegrown tool (originally written by me one Christmas) that imports our spreadsheet object model from IronPython and builds reST documents from docstrings, which get turned into our API Documentation. Although Sphinx can do docstring extraction we'll probably still need to use our ApiDoc tool - both because we need it to work with IronPython code (that imports .NET classes) and because it gives us complete control over which methods and properties are documented as part of our public API (plus I don't think Sphinx extracts docstrings from properties).

Syntax highlighting, search and indexing are all things we'd like for our API docs though, we just need to change our tool to generate reST suitable for Sphinx. In order to get a feel for it, I've ported the Mock docs over in preparation for the forthcoming 0.5.0 release (still a few things to do before it's ready):

• Docs for Mock 0.5.0 Built with Sphinx
• PDF Documentation for Mock

Getting started with Sphinx was made even easier by using virtualenv for creating clean development environments:

easy_install virtualenv
virtual_env some_name
cd some_name
source bin/activate
easy_install sphinx

By activating the new virtualenv environment, sphinx is then just installed for that environment (you will also need docutils installed which isn't picked up as a dependency by easy_install).

You can then get started with the sphinx-quickstart command that will create source and build directories for your docs along with a configuration file based on your answers to its questions. After that you just need to learn the Sphinx specific directives! The source reST files for the Python docs are a good place to start for examples (or you can browse the sources of the mock docs).

Building the HTML output is amazingly easy: sphinx-build source output

Once I'd installed MacTex (nearly 1.2 GIGS to download - for goodness sake) then building the PDF was also trivially easy:

sphinx-build -b latex source latex-dir
cd latex-dir
make all-pdf

The -b switch to the sphinx-build tells Sphinx to use a builder other than the default HTML builder. As well as generating Latex output there is another really useful builder - the doctest builder.

A few weeks ago I wrote an ode to doctest: Doctest How I Loathe Thee. In my opinion doctest is a terrible unit testing tool, it does however excel at testing examples in documentation. Sphinx makes it brain-dead simple to do. Any literal blocks you include in your docs Sphinx will try to infer if they contain Python code, and if they do it provides syntax highlighting with Pygments (fully configurable of course). You can mark a block of code as a doctest instead with the following directive:

.. doctest::

    >>> items = [1, 2, 3]
    >>> mock = Mock(magics='getitem contains', items=items)
    >>> 2 in mock
    True
    >>> 4 in mock
    False
    >>> mock[-1]
    3

Not only will this be syntax highlighted, but you can run all of the doctests in your documentation with: sphinx-build -b doctest source output

Mock has been developed using Test Driven Development, so it has pretty full tests. Nonetheless, making sure that all the examples in the documentation are up to date and still work can be a major pain. Now it is trivially easy. I have a renewed love and respect for doctest.

Of course it would be nice to have Sphinx running under IronPython. That means that the components of Sphinx needs to run on IronPython. Jinja 2 has no C extension dependencies, so after easy installing it (and copying the package to somewhere importable for IronPython since it doesn't have fecking zlib or zipimport and so doesn't support easy install...) the simple example works fine:

What about docutils?

OK, so that's a very shallow test, but its a good start.

---

Fun at PyWorks

Last week I returned from a fun time at the PyWorks Conference in Atlanta. PyWorks was a conference organised by the folks behind the Python Magazine (happy birthday by the way!), and was bolted alongside the already established PHP Works conference.

In total there were about 250 people present, the vast majority of whom were there for the PHP content. Whilst I'm interested in programming languages in general, there is very little about PHP that interests me (except perhaps how easy it is to get started with PHP), so I'm afraid to confess I didn't attend any of the PHP talks. There were some fantastic Python talks given, and it is a shame that so few people were there to hear them (perhaps 30-50 total people there primarily for the Python content). This was the first year for PyWorks and it will be interesting to see if they do it again.

In this picture from Saturday evening, Mark Ramm assists Chris Perkins in a visual demonstration of WSGI middleware...

The more refined gentleman to the right is Shane Caraveo from Activestate who gave a talk on GUI Applications with XULRunner and PyXPCOM from Python. For guaranteed fun at any conference invite Mark Ramm and ply him with tequila...

It was great to catch up with the Python folk I know from conferences and the blogosphere (Kevin Dangoor, Ed Leafe, Holger Krekel, Noah Gift, Doug Napoleone, Matthew Wilson, Jesse Noller and others). It was nice to meet new faces too, like Brandon Rhodes who roped a few of us into speaking at the local PyAtlanta User Group. There were about 30 people or so there, which was a great attendance for a regional user group. It was all the more poignant because that evening Noah officially handed the reigns for the PyAtlanta group over to Brandon, as Noah is leaving for a cushy new job in New Zealand. I gave a talk on Mock for testing. As I didn't prepare (well - I had three slides) I did it all with live coding and it didn't go horribly wrong!

Noteworthy talks (just the ones I can remember I'm afraid):

• Matthew Wilson on decorators. Decorators are way cool.

• Jesse Noller on multiprocessing

  The API is really nice and it has some cool features. It will be nice when I do some CPython work and get a chance to use it.

• Ed showing off some of the new features in the Dabo Framework.

  Dabo is a framework for writing desktop applications using the wxPython UI toolkit [1]. Ed was showing off two of the new features - a seriously cool looking forms designer and a way of delivering desktop applications across the internet (with features like auto-update built-in). As any desktop application ought to whip a web application in terms of usability and features this is a nice way of getting some of the benefits of web applications for desktop apps.

• Holger Krekel on py.test

  I also spoke to Chris Perkins quite a bit about Nose. The more I see talks on these two unit testing frameworks, the more convinced I am that I like the unittest way of structuring tests (object oriented tests with assert methods). I'm also impressed with features like integrated coverage testing and particularly test auto-discovery. These should be easy to add to unittest (including getting them into the standard library version) and it is something that is on my list.

• Noah Gift and Python for System Administration

  Even better I managed to get the copy of his book Python for Unix and Linux System Administration that he gave away (just beating Jesse to it). I'm really enjoying it and have found the section on IPython the most useful so far.

Outside of the talks I learned about virtualenv, Sphinx and a bit about Paver. Once Paver supports 'sub-pavers' then we may be able to use it to replace MSBuild for some of our build scripts. Currently we call out into a lot of small CPython scripts from MSBuild - so Paver could bring them all together for us.

My talk was on IronPython and Silverlight. All the material is on my IronPython Pages, but I still desperately need to update the Silverlight Pages for the latest release of Silverlight.

Many thanks to those who organised PyWorks, and the people I spent time with who ensured I had a good time!

[1]	And unlike other cross-platform toolkits it produces apps that look good on Mac OS X.

---

My First Screencast: Column Level Formulae in Resolver One

I've finally created a screencast. It's 2mins36 seconds and is about using column level formulae (one of the new features of version 1.3) in Resolver One. (Bonus points for working out where the names come from - but not many because it's easy.)

Column level formulas are cool to avoid duplicating formulae, but you can use them to do some interesting things like data aggregation in combination with list comprehensions; which is the subject of this screencast.

It actually took me two days of work to create this screencast! About half a day to create the spreadsheets, half a day to write and practise the script, and then a day of recording the audio and video plus editing. The next one I do will take considerably less time. Surprisingly (to me) recording the audio and video separately, and then editing them to fit together, worked very well. The trick is to know the script well enough that you don't have to read it (and therefore don't sound like you're reading a script).

For the audio I used Audactiy, and for recording the video Camtasia. Camtasia is extremely simple to use, but pretty limited in the editing that you can do.

Needless to say, it isn't 'studio quality', but it was fun to make and hopefully informative. The docs have more information on Column Level Formulae, and Jonathan did a great screencast on the other new features in Resolver One 1.3.

Ian Ozsvald of Showmedo is also doing some screencasts for us. They're a lot better than the one I did! The one on Python Objects in the Spreadsheet Grid is particularly good.

---