Fun with faulty benchmarks :-)

» In which we take a critical look at a flawed speed contest, and improve Python code in spite of that.

This page has a comparison of list-processing scripts in various languages. Apparently the idea is to create a list (or vector, or table, or array) of 100,000 elements, populate them with random values between 0 and 1, add all those values, and compute the square root. [via]

Now... I don't know if the author intended all examples to be exactly the same, or that it's just filler to demonstrate his software. I am far from a benchmarking expert, but off the top of my head, I can think of a few things that are wrong here:

some languages get to use special objects optimized for the task at hand (e.g. Io's Vector), others do not. In any case, comparing a Python list to a C++ array is apples and pears, IMHO.

output is done between starting and stopping the clock, which muddles the results.

Anyway, of course I had to look at the Python script and see if I could make it run faster. Here's the original:

import time
import random
import math
t=time.clock()

a=[random.random() for i in range(1,100000)]
def add(x,y): return x+y
print(reduce(add, a))
def square(x): return x*x
print(math.sqrt(reduce(add, map(square, a))))

print(time.clock()-t)

Aside from the fact that the Python version creates a list of 99,999 items rather than 100,000, there are many things that can be improved. This is what I came up with without trying very hard (it's about half the time of the original script):

import time
import random
import math

t=time.clock()

a = [random.random() for i in range(100000)]
print sum(a)
print math.sqrt(sum(x*x for x in a))

print time.clock() - t

The obvious improvements being, using sum rather than reduce and a custom add function; and using a generator expression for the second part. I'm sure it can be made faster, possibly quite a bit so -- but that's not the point of this post. :-)

In the site's results, Io clocks in at a cool 0.04s. Even though the benchmark as a whole is flawed, it's nice to see that a very dynamic language can also be very fast. Then again, in normal circumstances you probably would use List rather than Vector. This post shows the two approaches. (List is quite a bit slower and doesn't have special-case methods like sum and square.)