For these numbers, lower is better. The fastest optimiser gets a 1.00, all the others are scaled relative to it. The most red value is the slowest. ghc2 is GHC 6.8.1 with -O2. supero-* is running through the Yhc front end, then through Supero, then through GHC 6.8.1 -O2. none corresponds to the supero optimiser doing no inlining or optimisation, essentially giving GHC the whole program in one go. The other supero-* results are all based on Positive Supercompilation with homeomorphic embedding as the termination criteria and a given generalisation function. whistle is no generalisation criteria, msg is the most specific generalisation, and general is my newly created generalisation function. All the details will be in the paper I'm submitting later today.
I know some places where these results have fallen a bit flat of what they should, but even so, I'm very happy with this!
| ghc2 | supero-general | supero-msg | supero-none | supero-whistle | |
| bernouilli | 1.00 | 1.41 | 1.53 | 1.18 | 1.58 |
| digits-of-e1 | 1.00 | 1.03 | 1.16 | 1.06 | 1.03 |
| digits-of-e2 | 1.39 | 1.00 | 1.00 | 2.58 | 1.00 |
| exp3_8 | 1.00 | 1.00 | 1.00 | 1.01 | 1.00 |
| integrate | 2.19 | 1.00 | 1.03 | 8.79 | 1.00 |
| primes | 1.76 | 1.00 | 1.00 | 1.69 | 1.54 |
| queens | 1.26 | 1.00 | 1.21 | 1.52 | 1.05 |
| rfib | 1.03 | 1.00 | 1.00 | 1.03 | 1.00 |
| tak | 1.38 | 1.00 | 1.92 | 1.92 | 1.92 |
| wheel-sieve1 | 1.02 | 1.00 | 1.13 | 5.33 | |
| wheel-sieve2 | 1.58 | 1.37 | 1.00 | 1.00 | 1.40 |
| x2n1 | 1.71 | 1.00 | 1.09 | 5.19 | 2.76 |
Interesting results! But given that you used such a powerful technique as supercompilation I would have hoped for even better results. And I think you could get even better results if you use different benchmarks. I've also used nofib for benchmarking but in my experience these programs are not very high level and they have an oldish feel to them. For instance, none of them use monad transformers (at least afaicr). It would be interesting to see what results you would get from a program which is highly parameterized or uses highly generic libraries like monad transformers, syb or something along those lines. That will give some indication of what kind of improvements we might expect when writing very high level programs. The hope is of course that the optimizer will make the high level programs as efficient as the low level ones which does the same thing, thereby eliminating any redundancy we've introduced when abstracting the program.
ReplyDeletejosef: If I could choose the benchmarks, I'd get absolutely wonderful results :-) I realise the microbenchmark style problems aren't that representative, so hopefully larger programs will offer more benefit. Nofib is a bit weird, while none have monad transformers, loads have CAF's and quite a lot have circular programming - things which are quite rare in real programs.
ReplyDeleteMore benchmarks are to come!