Tao of the Machine

Laszlo Presentation Server

Laszlo Presentation Server is a system to develop rich web-based applications. It works with Flash, but because of its system-independent .lzx file format (which is based on XML), it will be capable of supporting other display formats as well.

For a quick guide, see Laszlo in Ten Minutes. In the meantime, here are my first impressions:

• Writing applications using XML is straightforward and simple. I'm not really a fan of XML, but in this case it seems a natural fit. Your components are represented by XML tags. Child controls are easily added by child tags. For example, here's "hello world" in LZX format.

<canvas height="100" width="500" >
   <text>
      Hello, World!
   </text>
</canvas>

• Running an .lzx application is easy. Simply drop the file where LPS can see it, and point your browser to it. (Of course the application server needs to be running.)

• There are some nice features, like floating windows, easy drag & drop, easy editing, debugger, etc.

• There are data-aware controls, but most of the examples use XML datasets. Attaching them to a "real" database is possible, but not very straightforward. I would like to connect to a database directly; Laszlo however seems to depend on server-side techniques, like JSP, ASP or CGI. Also, whatever tool or technique is used, it still needs to pass data as XML.

• For more programming power, you can define classes and methods (in XML). You can also mix LZX code with JavaScript etc.

• While most of the GUI stuff is as you would expect it, there are some unusual features. For example, let's say you have a list of addresses. You edit the address by clicking it, after which an edit view appears under the original line, moving all the following addresses down. To see what I mean, see the section about data-driven applications.

• The application server output window sometimes gives helpful hints. For example, I tried modified the "hello world" example so it would show a bigger font. Without looking up the reference for the text tag, I changed it to <text size="10">. When running it, the program ran as usual (without errors), but this appeared in the output window:

hello.lzx:2:20: attribute "size" not allowed at this point; ignored
hello.lzx:2:20: found an unknown attribute named "size" on element "text", however there is an attribute named "fontsize" on class "text", did you mean to use that?

• It's just too slow for my computer. Granted, I don't have a high-end box (900 MHz Intel Celeron, 256 Mb), but still. Apparently a lot of horsepower is necessary to run this smoothly, which you would expect from a server, I guess. Running it with IE seems faster and more stable than with Firebird, but loading times are often very slow with both browsers. Not to mention the fact that it hogs memory. (These problems might be related to the fact that it uses Java.)

All in all, a promising system. It's easy to write powerful programs that are pleasing to the eye. You'll need something better than my computer, though. :-)

Brainstorm in een glas water

A few more thoughts about the "document database" (1, 2).

Performance problems, like slow packing and committing, are caused by the fact that the database is too large. It is too large because it contains many large documents. (Remember, 400 Mb of data, and that's only to start with.) Large databases and pickling don't match very well.

One solution might be, to keep the index and the document metadata in the database, and keep the documents (files) out of it. In other words, the repository would have a ZODB database, plus a not-quite-magic directory with files. When necessary, e.g. when editing, the program will open the desired file(s).

Drawback #1: Searching the actual data of each document will be very slow, since the whole directory tree needs to be traversed, and every file opened. While I might include this option, I hope this won't be necessary at all. Most common searches should be covered by the index and metadata search (keywords, size, date, etc). Otherwise, you can always do a grep or find on the actual files.

Drawback #2: The file structure can be changed externally. You could move files around, delete them, add new ones, all from outside the program, so the database would not be up-to-date anymore. The obvious solution is "so don't do that". There should also be a way to recheck (parts of) the directory, and update the database accordingly. If it's fast enough, users could easily import new files in bulk by dropping them into the right directory. This would be a good way to start with an existing collection of files.

Like somebody suggested (Ian Bicking?), it would also be possible to store URLs this way. The document's location would then be an URL rather than a local file. Obviously, such a document cannot be edited, but the GUI could do other things, like opening a web browser with the desired page.

Prototyping

Another person experimenting with Self-like prototypes for Python. Just remember that I was there first <wink>.

Of course, my code was just an experiment as well... I wonder if a "serious" prototypes package would be useful. I mean, would people actually use it? The prototype approach is very different from the regular Python class/instance model, and allows for very different constructs.

Also interesting is Prothon, and the c.l.py discussion about it. I agree that the choice of tabs for indentation is unfortunate. I also wish there were more code samples available. From what I see, I'm not sure it's all that close to Self's object model, but then again, I haven't actually programmed in Self...

Ancient Chinese programming secrets

And now for something completely different...

Taoism is about being in harmony with nature. As the Wikipedia page says, "Do not try to force things, for nature is overpowering. In particular one must act in accordance with how things are, not how one wants things to be." Also, there's the principle of wu wei (action through inaction). "It is the practice of working with the stream rather than against it; one progresses the most not by struggling against the stream and thrashing about, but by remaining still and letting the stream do all the work."

Confucianism is about rules and rituals. About duties, etiquette and what is proper. (There's a lot more to be said about this (see the Wikipedia item), but it mainly refers to social and interpersonal relations.)

Clearly, dynamic ¹) programming languages follow the Taoist way. Due to the flexibility of those languages, the program and language can adapt to the problem, rather than the other way around. Thus, a solution can be reached that does not violate the problem's nature, with a minimum of effort.

Statically typed languages, on the other hand, are more Confucianist. They come with a set of rules and restrictions. Like with Confucianism, these rules comfort those who use these languages and give them a sense of security. When implementing a programming solution, the problem must be expressed in terms of the language, adhering to its restrictions. The result naturally complies with the language's rules, but is not necessarily the best solution.

HHOS. OK, I'm just babbling here, but I think it's an interesting idea.

¹) "Dynamic" meaning "dynamically typed", or something similar to it. Thus it covers Lisp, Smalltalk, Python, Perl, Ruby, Groovy and more. Where OCaml fits into the picture is unclear, being both strongly, statically typed and dynamic...

Revolution OS

I just saw this movie, that describes the open source/free software movement, and some of the people who play an important role in it. It is basically a collection of short interviews and snippets, glued together to tell the story of (mostly) Linux.

The movie is interesting, but doesn't really tell anything new. I'm not sure who the target audience is; open source developers probably already know all this, and I doubt this material will be very interesting to non-programmers. That leaves closed-source programmers. Hmm.

It's also worth noting that the visionaries of the movement seem also to be the most annoying flamboyant. They do what we have come to expect from them... ESR stomps on Microsoft and communism. RMS rides his GNU/Linux hobbyhorse, still sounding like that irritating kid in school that corrects the teacher. Linus is refreshingly "normal" in comparison. Most of the other guys also seem more down to earth and (IMHO) much more sympathetic.

The movie is worth a look, if only to see and hear these people (rather than just read about them). Aside from that, it does not seem to have much lasting value, especially not to those who are already open source developers.

QOTD: How to extract Windows "file properties"?

In Windows 2000 and XP, files have a "summary", which can be viewed by right-clicking a file, selecting "Properties", then the "Summary" tab. These properties include Title, Subject, Category, Comments, Author and more.

How to extract these properties from code? Sounds like a simple question; you'd think there must be some API function for that, or something. But apparently that is not the case; various google searches found lots of people asking the same question, but no real reply.

Well, that isn't entirely true. Here's a Delphi solution (untested), and here's some information about the "Horrible Property Set Format". What I am interested in though, is extracting these metadata from Python. Looking at the low-level Delphi code, that seems a bit difficult.

The best solution I can come up with right now is to wrap said Delphi code in a DLL (if possible) and call the DLL from Python. Is there a better way?

More C

Another lesson... behold the dreaded -mno-cygwin switch.

Yes, lately I've been busy trying to compile C code. Existing (open source) packages, sometimes with Python extension modules already written, sometimes not. Windows isn't exactly an ideal platform for this, but something's got to give. If the end result works (i.e. a working .pyd file that can be imported from Python without crashing ;-), then it can be very gratifying. But until that moment is reached, I feel much like I'm stumbling around in a dark room, not really knowing what I'm doing or what's ahead, running into lots of things, sometimes grabbing the right library or using the right switch by accident.

Compiling 101

A lesson learned in compiling C code... Just because the configure script and makefiles work, doesn't mean they're up to snuff. Sometimes you just have to throw them away and compile by hand.

Also, don't expect the (third-party) code you're compiling to be without errors or up-to-date.

Basic rules, but easily forgotten when you haven't compiled C code in a while.

It's just one of those days...

It seems all I do lately is trying to pound unwilling programs into submission, with little success. Java programs that I have no clue how to start or what they require to run. C libraries that won't compile. Or Python extension modules that do compile, but don't actually work.

Maybe I have more luck tomorrow. If nothing else, I learned that compiling Python extensions with Cygwin (or MinGW) is not a walk in the park. :-/

Programming and creativity (2)

Hmm, interesting points were raised in the comments to yesterday's post. A few comments of my own:

Bob Ippolito: "Languages like C and Pascal don't really do less than a language like Python (because you can write Python in C).. they just make you type more. I don't think it's creative at all, just redundant."

Yes, but let's say we're tackling a high-level problem in C (or Pascal, etc). Say, writing a library for XML parsing. Assuming that I cannot or don't want to use third-party libraries, most likely I will have to write my own linked lists, dictionaries, etc. This is a form of creativity, since I'm making something that wasn't there before and is useful. When you only know C, this may indeed be the case. But when you know a high-level language like Python, all that work seems redundant, and not creative whatsoever. In fact, in Python this problem wouldn't come up at all, because you would just use the built-in list and dict.

John Eikenberry: "When I think of restrictive programming languages I think of things like recursion being the only loop construct and no state variables. In my experience languages like Prolog, Scheme or Haskell can force you to be creative. Whereas other standard imperative like Java just make you be more verbose (as compared to Python)."

This is a good point, and it illustrates why learning these languages is worthwile, even if you don't actually use them for real world programming projects. A language with recursion but no loops, for example, forces you to tackle problems differently, and to *think* about them differently. Hence ESR's claim that learning Lisp makes you a better programmer, even if you don't use it. This kind of creative learning is useful even (or maybe especially) if you use other languages.

This is one of the reasons why I keep looking at different languages... not to replace Python (which is my current language of choice), but to learn different concepts. In that respect, Self is interesting, and so are Forth and Logo, to name a few. And even obscure languages like Q-BAL.

Programming and creativity

Some thoughts:

1. Restrictions don't hamper creativity... they stimulate it.

2. But how do restrictive programming languages (e.g. statically typed) fit into this picture?

Would programming in Java or Pascal be more creative than programming in Python? It certainly doesn't *feel* like it... the restrictive languages get in my way, Python does not.

But in a certain way, yes -- Java and Pascal force me to find creative solutions for things that are no-brainers in Python. Most design patterns, for example. Or iterating over a list with values of arbitrary types. Unfortunately, this introduces creativity where I don't want it... solving a problem is one thing, working around the restrictions that a language imposes on you is another. So this seems like a waste of time, especially since the end result isn't any better or more interesting.

Strange. In art, restrictions fuel creativity, because it forces you to think in ways you haven't thought before. The type of restriction hardly matters -- try making a decent-looking painting with only yellow and blue, or writing a story without using the letter 'e'. Exercises like these will reshape the way you think about painting, writing, etc.

Why doesn't it work that way with programming? Using Java, Pascal, etc, certainly affect the way one thinks about programming, but is it for the better? Some may think so, but I beg to differ. Using these languages doesn't make me a better programmer, nor do they provide a better end result (everything else being equal). With a dynamic language like Python (or Lisp, etc), I get the job done, it's done faster, it's more flexible and maintainable, more open to change, and on top of that, it's fun.

Now, this is not another "Python is cool and Java sucks" rant. For Python, you can substitute your favorite dynamic/non-restrictive language, ditto for Java/Pascal and restrictive languages. No, what I'm wondering is, why don't restrictive languages fuel creativity? Or do they?

XL

Here's another programming language to look out for: XL (part of the Mozart project). The compiler is not done yet, but people are working on it. Could be interesting.

Groovy (3)

A bit more Groovy stuff...

From the groovy-dev mailing list: "You can now define functions in scripts which can be very useful in shell-scripty type use cases. Here's an example test case..."

def foo(list, value) {
     println "Calling function foo() with param ${value}"
     list << value
}

x = []
foo(x, 1)
foo(x, 2)
assert x == [1, 2]

println "Creating list ${x}"

Nice, this makes things more Pythonic and scriptable. I don't like the new << operator... lists already have an add method that does the same; what's the point of adding a redundant operator that looks like line noise?. Then again, let's not forget that Groovy doesn't have the same design philosophy as Python...

The expression 1..3 creates a list [1, 2]. In other words, the upper bound is not inclusive. This is kind of counter-intuitive, methinx... Haskell has a similar construct that does the right thing, if I recall correctly. Of course, Python's range(1, 3) returns the same list, but for some reason the .. operator "feels" to me like it should include the upper bound. Hmm.

Also, code like

for (i in [1..10]) {
        println(i)
}

doesn't do what you might expect... [1..10] is a list with one element, viz. a list containing the numbers 1 through 9. When you want to loop over such a list, use for (i in 1..10) instead.

That's it for now. I'm glad to see that the language is in steady development, and that new, useful features are added regularly. I do get the feeling that the language somewhat lacks a coherent design philosophy, but it could also be that the creator is still looking for that. Time will tell. But with its dynamic features, code blocks, and ties to the Java libraries, there's enough reason to start using it.

Update. Apparently the CVS version of Groovy has different behavior for the .. operator. 1..3 is now [1, 2, 3], and 1...3 is [1, 2]. I'm not sure how often changes like this happen... I'm aware the language is under heavy development, but this makes me a bit worried about using it for production code.

Groovy (2)

More on Groovy. This language turns out to be an interesting alternative to Python (maybe), although I don't necessarily like all its features. For example, there seems to be a lack of functions. Granted, these can be emulated with "closures" (I'd much rather call these "code blocks", but it seems to be a part of the official terminology).

add = { a, b | return a + b }

This defines a closure that acts much like a function. It takes two arguments and returns their sum. Using it is unsurprising:

a = add(3, 4)
println(a)

The manual claims that a function call's parentheses can be omitted in certain cases (if there's no ambiguity), but that doesn't always seem to be the case:

b = add 5 6
println(b)

This prints something like calls1$1@12d7a10... strange. And add 5, 6 is an error. It doesn't (always) work with one parameter either:

id = { x | x }

a = id(42)
println(a)
# prints 42

b = id 43
println(b)  # same here... doesn't work as intended

Aside from that, closures can be really nice. This is how a list of numbers can be mapped to a list with double the values:

list = [1, 2, 3, 4]
a = list.map { it * 2 }
println(a)
# [2, 4, 6, 8]

The "official" syntax would be list.map({ it * 2}), but in this case the parens can indeed be omitted. I'm not sure I like that feature, although it does seem to make this code a bit more readable.

it is an implicit variable. By default, closures take one argument, called it. A different name can be specified, and so can multiple arguments. For example, the line with list.map could be written like this:

a = list.map { num | num * 2}

Python could do this with a lambda, of course, or by defining a named function first. Neither solution seems ideal, though, when the code block gets a bit more complex. This is one area where Groovy seems better than Python. (And maybe Ruby, considering it has code blocks as well.)

A strange quirk:

b = list.map({ 0-it })
println(b)

This works, but using -it does not, for some reason. I have yet to figure out why not.

A with (like in VB and Pascal) can be specified... more or less. The parentheses are in the way, kind of. Maybe there's a better syntax for this.

with = { obj, closure |
         closure.call(obj)
}

with(lst, {
    println(it)
    println("list size: " + it.size())
})

# prints:
# [1, 2, 3, 4]
# "list size: 4"

/* cannot be written as:
with list {
    println(it)
    println("list size: " + it.size())
}
*/

Note the power of closures here... in Python, we could write a with that takes a list and a function, but not a code block. Apparently closures are used a lot in Groovy, maybe a bit too much... it doesn't always make things clearer.

More later. Too bad there's not too much documentation on these closures. The mailing lists seem to be active, though, maybe I can pick up some more information there.

Update. When fleshing out new Groovy features, it seems that many people like a syntax that mimicks (or is close to) Java's. I'm not sure that's a valid argument. Sure, similarity of syntax may be nice for people coming from Java. But languages like Groovy (and BeanShell, etc) also attract a different crowd: those who don't like Java, but who wouldn't say no to a dynamic language that uses the Java libraries. ¹)

I don't know the proportial sizes of these groups (Java-background vs non-Java-background), but it may be a good idea to go for the "best" syntax (whatever that means), rather than going for a Java-like syntax, disregarding all other considerations.

¹) This includes Jython too, but like I pointed out in my previous Groovy post, I cannot use Python for this upcoming project.

Groovy

I am currently looking at Groovy. It's a hybrid of Java, Python and Ruby, that runs on the JVM. Much like Jython, it gives programmers the power of a dynamic language, while retaining access to the vast Java libraries.

I could just use Jython, but for the project we have in mind (for work), we cannot use Python or Delphi, for non-technical reasons. (We cannot reuse any code from existing Python/Delphi projects, and the safest way to do so is to use a different language.)

Groovy seems nice. Close enough to Python for me to like it. :-) It has lists and dictionaries (although the syntax of the latter seems a bit awkward). More importantly, it has code blocks, known as "closures" (which name is a bit unfortunate, IMHO... they're not like Lisp/Scheme closures).

Much of the new constructs and syntax are like Ruby, but seem a bit less hairy. It also seems a bit more readable.

Installing it wasn't so difficult, even for someone who doesn't have much experience with Java, like me. I had to tweak a batch file or two, because they called the command shell to verify something, which caused odd effects on my console screen. After setting the correct environment variables, everything worked. groovysh is an interactive interpreter, not as powerful as Python's, but usable nonetheless.

At the moment, I have a hard time understanding this code:

class Foo {
  myGenerator(Closure yield) {
    yield.call("A")
    yield.call("B")
    yield.call("C")
  }
}

foo = new Foo()
for (x in foo.myGenerator) {
  print("${x}-")
}

That is, I can see what it does... but it's unclear how it does it. Foo.myGenerator looks like a method. But when for (x in foo.myGenerator), that method is not called. Or is it? If so, what happens to the yield parameter? Is it implicit? I would like to know more about the inner workings of this construct.

Hopefully this is explained somewhere else. More later.

Art

David Brown: Art.

This may be a bit of a stretch, but I wonder if the above (read the post first :-) can be applied to agile vs design-everything-first programming. With agile development, you write code, evaluate, refactor, write some more, etc. It's a continuous process where design and actual coding go hand in hand, and you often learn along the way, about the problem at hand, and about programming in general. With the other method (design everything first, then implement without changing the design), you don't have this learn-as-you-go benefit.

Appetite for construction

Lots of stuff to do, less than a week before I leave. Besides work etc, I've been working on a website, which might be online any day now. Not unrelated, Firedrop has been updated; it now sports a HTML import feature, among other things. Early next year there will also be a Py article about Firedrop/Kaa. Also, Wax was bumped to version 0.1.44.

I will probably have some serious programming withdrawal when I'm in the Netherlands... <0.3 wink>

The case for reinventing the wheel

Software reuse is cool, but sometimes it's better to write something from scratch.

I once overheard somebody say that X was the company's best programmer because rather than writing something himself, he would look online for components to reuse, thus saving a lot of work and time. (It wasn't a programmer who said this, by the way.) While this is generally true, especially for application programming, it is not an absolute truth. Of course it's a waste of time to write something yourself if perfectly good code is already available. But there are other cases.

1. Sometimes it's easier/faster to write new code, rather than trying to understand existing code. I encountered this today... for my work, I am writing a module that talks to an FTP server in a certain way. There was already (old) existing code, that we never actually used, and that pretty much did what we wanted. Pretty much. It turned out that trying to understand the old code, and trying to make it work conforming to the new specifications, was more work than I thought. I now think that writing something new would have taken less time.

2. The new code will (or should) do exactly what you want, while the existing code might not. For example: Python first had xmllib, but that didn't stop people from writing a more sophisticated XML framework, because xmllib didn't do what they wanted. And even now that xml.dom and friends exist, people are still rolling their own parsers, because they are not entirely satisfied with the xml package, for various reasons.

3. The existing code may be lacking in certain ways. It might be too simple or too complex. It might be buggy (so you either have to fix it or write your own). Maybe it isn't really meant to do the task you have in mind. Maybe it has licensing issues. It may not perform as well as you think it should. It might use a brain-dead algorithm. And so on.

4. When you write your own version, you learn a lot about the issue at hand. If I write my own XML parser, I necessarily need to know or learn a lot about XML, and probably about parsers, in order to do it right. When I just use an existing parser, often in the form of calling a few methods on an object, then I won't learn much at all.

Granted, sometimes you don't want to learn, because you're not interested in it, or because you don't have the time. But everything else being equal, having the opportunity to learn something is not unimportant.

That said, I often successfully reuse code, especially in Python where it's often easy to find a workaround if existing code has limitations. But I wouldn't go as far as saying that the best programmer is the one who reuses the most. Sometimes reuse is obvious, sometimes it's not possible, and there's a large gray area where the benefits of both options have to be considered, rather than just go for the reuse because "some" code is already available.

Hey, Goodyear didn't go out of business making wheels...

Hum...

Lots of stuff going on before my vacation. Some work... I am designing a website... the site will possibly be maintained with Firedrop, which needs to be improved... another project is the "cloning" of a Delphi app, using Wax... some changes to Antilog are necessary... etc.

Currently reading: Robin Hobb: Assassin's Apprentice
Currently listening to: Haagse Mark, B-52s, Spyder-D
Currently hacking on: Wax, Antilog, Tarantulon

OK, so I don't really know anything to write about. Maybe tomorrow...

De kritiek van Hans, part 2

(via Jarno Virtanen -> Glyph Lefkowitz) In this Artima interview, Bertrand Meyer mentions the importance of reducing complexity. "I think we build in software some of the most complex artifacts that have ever been envisioned by humankind, and in some cases they just overwhelm us. The only way we can build really big and satisfactory systems is to put a hold on complexity, to maintain a grasp on complexity. Something like Windows XP, which is 45 million lines of code or so, is really beyond any single person's ability to comprehend or even imagine. The only way to keep on top of things, the only way to have any hope for a modicum of reliability, is to get rid of unnecessary complexity and tame the remaining complexity through all means possible."

I prepared a long post, discussing the meaning of simplicity and such, but I wasn't content with the result. So I'll just say this: certain Eiffel features, like requiring getters and setters to access attributes, and static typing, don't reduce complexity... they increase it. The same goes for being anal, reducing flexibility, and preventing programmers from doing what they think is best. Reducing the number of rules makes things less complex, adding rules does the opposite.