Tao of the Machine

Changes

The generator proposal mentioned yesterday didn't cause a stir on the newsgroup. The gist is, that just about everybody likes it. I am one of the very few exceptions.

I wonder if the years working with Python have spoiled me. When I used Pascal, in the middle 90s, I would have welcomed any syntax change. I marvelled at Ada because it seemed like Pascal on steroids. (I never actually used it, aside...:-) Right now I would probably welcome syntax changes in Delphi (Object Pascal). Then why not Python?

It's not that Python is perfect. I think it's more that since I've been using it, I've become more aware of readability and maintainability issues. To be able to read your own (or somebody else's) code, with little effort, even months or years later, is a great boon. When using other languages -- Pascal, C, Basic, etc -- I didn't worry much about it; readability was, and still is, a matter of proper programming and coding styles, not something that was built into the language. Reading other people's code, or your own months later, took some effort; I considered this normal and to be expected. That all has changed with Python. If Python code is hard to read, it's because it was coded that way, or because it deals with a complex problem, not because the language makes it hard to read.

So, as a result, any change that, in my perception, negatively affects Python's readability and clarity, causes a knee-jerk reaction. I wouldn't have the same problem with other languages, since I don't consider them very readable to begin with.

It doesn't help that this latest proposal is part of a disturbing trend, to enhance the language with more and more features, at the cost of some of its core values -- readability, simplicity, ease of understanding, for example. Of course, it still does a lot better in this respect than most other languages... but I'm not sure I like where it's going. Will the future Python still be Pythonic?

New generator proposal

PEP 289 has seen a revival. It was announced on c.l.py today:

In brief, the PEP proposes a list comprehension style syntax for creating fast, memory efficient generator expressions on the fly:
 
sum(x*x for x in roots)
min(d.temperature()*9/5 + 32 for d in days)
Set(word.lower() for word in text.split() if len(word) < 5)
dict( (k, somefunc(k)) for k in keylist )
dotproduct = sum(x*y for x,y in itertools.izip(xvec, yvec))
bestplayer, bestscore = max( (p.score, p.name) for p in players )
 
Each of the above runs without creating a full list in memory, which saves allocation time, conserves resources, and exploits cache locality.
 
The new form is highly expressive and greatly enhances the utility of the many python functions and methods that accept iterable arguments.

I don't like it. I don't have any good arguments against it, except that it's another little step away from readability and simplicity, and another little step closer to clogging the syntax. Like with the introduction of sum, you get little in return. OK, so it's "fast and memory efficient". Big deal; I don't use Python because it's fast or memory efficient, I use it for other reasons, like readability, maintainability, dynamic features, and the fact that you can hack up a working program real quick.

Also, even though it's only two brackets short of a list comprehension, I don't find expressions like

y = x*x for x in roots

very readable. y is a generator. Somehow, I find the lack of distinguishing syntax disturbing. (As said, I don't have really good arguments against it.) Also, what about atrocities like:

for x in x*x for x in roots:
    ...

or:

[x for x in S]    # This is a list comprehension.
[(x for x in S)]  # This is a list containing one generator
                  # expression.

The meaning depends on the presence or absence of parentheses... That tuples use this is one thing, but I don't think it's a good idea to add more of this to the language.

Rather than gratuitous features that are not really necessary (bool and sum come to mind), I'd like to see stuff that enhances Python's "dynamism" and flexibility. Lately I've been running into restrictions, e.g. not everything can be pickled, the global state of the interpreter cannot be stored and restored later, there are no "free" code blocks, a function/class/etc's source cannot be edited on the fly, etc. It might be interesting to see what can be done to fix these problems, in a Pythonic way.

Putting testing to the test

An interesting thread on c.l.py talks about the limitations of unittest. Basically the complaint is the following. Let's say you have a bunch of data, that you want to loop over and do certain tests with. You can write something like this:

data = (
    (1, 2),
    (10, 4),
    (2j, 3j),
    ...etc...
)
for a, b in data:
    self.assertEquals(f(a), b)

But this has the drawback that when an error or failure occurs, the rest of the data in the loop is not tested.

Finding a good solution is not easy. I tried to add something to Testipy. My idea was, that if you have methods Assert, AssertEquals, etc, then there should be wrappers LazyAssert, LazyAssertEquals, and so on, that work like (and in fact call) their counterparts, but they catch any errors that might be raised, and add them to the TestCase object's result list. So I rewrote the example above to look like:

data = (
    (1, 2),
    (10, 4),
    (2j, 3j),
    ...etc...
)
for a, b in data:
    self.LazyAssert(a > b)

This works until the third element of the list is reached. 1 > 2 fails and is added to the result list as a failure. 10 > 4 passed and is added as OK. But 2j > 3j raises an exception before LazyAssert is called, so there's no way it can handle this error.

I suspect that a class-based approach may not be the way to go. I also suspect that some notion of "free" code blocks (like Ruby has) would be useful here. But I'm not sure. The notion "methods are tests" is a powerful automation mechanism; it's also restrictive in some cases, though. Surely something else must be possible, besides wrapping every test in the loop in a new method? But what?

Update. Further reading: Self Stocking Test Suites: Dynamic Testing Made Easy by Garth Kidd.

Some links

• IRIX tips
• Helpful IRIX commands
• tricks and fun with vi

Thought of the day

If everything was red, you wouldn't even see that it was red.

Visions of apocalypse

The Trusted Computing FAQ. Scary, scary stuff.

L'ordinateur, c'est moi

Yes, more O2 stuff. There's not much else to write about anyway, except that maybe in a few months or so I will visit the Netherlands again.

• I successfully compiled Python 2.3.2. Although it depends on how you define success, really... I get a clean compile out of the box, but the resulting Python doesn't have the socket module, as I found out when I tried to write a simple mail checking program (just for testing purposes). Uncommenting _socket in Modules/Setup yields errors. Some other modules, like grp and pwd, did compile after uncommenting. So this is something I need to take a closer look at. By the way, the precompiled Python 2.1 on freeware.sgi.com doesn't have _socket either.

• elastiC 0.0.37, the development version, does compile. Yay! I can now experiment with this interesting language. (0.0.35 didn't work for some reason, the configure file seemed messed up, thinking gcc was a cross-compiler, etc.)

• What doesn't compile yet is Common Lisp. I tried versions of CLISP and CMUCL, to no avail. Then again, I don't really know what I'm doing. :-) Maybe there are binaries for IRIX somewhere.

• I have no clue how to use samba. Reading some documentation is in order.

• I installed a bunch of packages from freeware.sgi.com, mostly as auxiliary software for stuff I wanted... a newsreader, samba, cvs...

• Once cvs is set up properly (which I have done before, but don't find easy), I can actually hack on some of my projects from the O2. The adventure engine comes to mind, and other stuff that doesn't require fancy GUIs and graphics (or sockets :-).

• So far working on this machine has been fun. Sure, not everything I try works, but it's a good way to learn Unix, and as long as I don't stretch my expectations too much (like I did with the Mac) it will hopefully remain fun. I wonder how long the O2 has been out. I'm guessing it came out around 1996. It must have been a monster at that time, with 256 Mb of memory. Its performance is still surprisingly good; yesterday I compiled Python in one shell window (a memory-intensive task) while browsing the Net and running gvim in others. On my Windows box, this would have been a definite slow-down, even though it has 256 Mb as well and a 900 MHz processor (O2 has 250 MHz RISC).

Now with 10% less content

More SGI stuff. I realize this will not be very interesting for most readers, so won't keep posting about it much longer, unless some peculiar features/questions come up. Hopefully I will have some real content again one of these days. :-)

Maybe I can use some shorthand, some kind of "ticker" syntax, appended at the end of a post, to keep people informed about my SGI progress, or lack of it. With simple syntax:

+: indicates progress, success
-: failure, no progress (yet)
!: observation
!+: observation, positive
!-: observation, negative
?: question, something to find out
>: to-do

[+ installed bash, set it to default shell. + installed vim. !− IRIX vim in text mode isn't very good. !+ but gvim is OK. + installed gcc. + compiled hello.c. :-) + compiled and installed scheme48. − elastiC doesn't compile here either. + edited bash .profile. ? still don't know how to access PC. > install samba.]

Yes, elastiC doesn't compile on IRIX either. (Earlier attempts failed on Mac OS X and Cygwin.) It makes me wonder where it *does* compile. I'd like to try the language, but the Windows binaries are old, and I obviously haven't had any success compiling so far. Suck.

Exploring Unix is both a rewarding and a frustrating experience. Rewarding, because there's always something to learn. Frustrating, because it's not always easy to find out things. It's not bad if you're just tinkering, but if you need to find out something quickly, then you might be in for a world of pain. At least, until you know what you're doing. :-) But this is what I got the SGI O2 for in the first place... so I have a lot to learn, to find out, to tinker, etc. Just like back in the days when using a computer was fun...

Second impressions

Progress: Hooked it up to the network. SGI box can go online now. Downloaded software; Python 2.1 from the SGI freeware page. After some poking around, it appeared to be installed in /usr/freeware/bin.

The IRIX version is 6.5.11f. About 2.5 years old; maybe newer versions support more options, including such essentials as, setting the desktop background image, and storing console settings. I am currently looking for an upgrade of some kind. SGI SupportFolio seems to have them, but they didn't send me an email with my registration confirmation.

Some more software needs to be installed to make the machine usable. The shell (sh) doesn't support readline, for example, and will be replaced by bash. It also only has vi and not vim or emacs. At least vim will need to be installed. Caveat: the harddisk is only 4 Gb, with 1+ Gb free, so I cannot afford to install lots of stuff. Just the bare necessities. This includes gcc as well. There *is* a C compiler, though, that I have to investigate first. A new browser would be cool as well, but Mozilla Firebird is 50+ Mb. I really need something better than Netscape 4 though, which is the default.

Oh, and the machine is very much capable of displaying high resolution graphics in lots of colors. It's just hard to see without the right monitor, and (this version of) the OS doesn't make good use of the features.

Open issues: How to access files on the PC? How to get and install an IRIX update? How to free harddisk space without messing everything up?