As usually I am focused mostly on the compiler comparison as optimizing compilers on major platform x86/x86-64. I don't consider other aspects of the compilers as quality of debug information (especially in optimizations modes), supported languages, standards and extensions (e.g. OMP), supported targets and ABI, support of just-in-time compilation etc.
This year I did the comparison using following major options equivalent with my point of view:
I tried to decrease the number of graphs which are still too many. Therefore I removed data for -O0 -g and -Os from the graphs but still I post some data about these modes below. If you need exact numbers you should look at the tables from which the graphs were generated.
I had to use -O0 for compilation of SPECInt2000 254.gap for both compilers as LLVM3.2 can not generate correct code in any optimization mode for this test.
Here are my conclusions from analysing the data:
Therefore I had to use Dragonegg (a GCC plugin which uses LLVM backend instead of GCC backend) for generation of Fortran benchmarks by LLVM.
Although CLANG made LLVM less dependent on GCC, still LLVM is heavily dependent on GCC and more generally on other GNU projects (GOLD, binutils etc). Industrial compilers (including Intel compilers, SUN studio compilers, OPEN64, Pathscale) usually support triad of languages C, C++, and Fortran. It is a pretty big investment to implement Fortran front-end especially with language-dependent optimizations.
To see a perspective, the performance difference between LLVM2.9 and GCC4.7 reached 20% (on SPECFP2000 in 32- and 64-bit modes for -O3). So LLVM made a significant progress with the performance point of view since 2.9 version.
I believe such progress is achieved mostly because of a new RA introduced in LLVM 3.0. By the way, although new LLVM RA is much better than the old one, I think it is a mistake that the new RA still does not use graph-coloring based RA which has a potential to improve performance even more
It is still close to true for LLVM with CLANG front-end. For example, in case of 32-bit SPECInt2000 the code quality generated by GCC4.8 in -O1 mode is 16% better than one generated by LLVM3.2 in -O1 mode and 1% better than code generated by LLVM3.2 in -O2 mode, but GCC compiler in -O1 mode is 2% and 10% faster than LLVM3.2 correspondingly in -O1 and -O2 mode. It means that GCC -O1 is closer to CLANG LLVM3.2 -O2 with the performance and compiler speed point of view.
Where GCC is really slower (2.5 times) than CLANG LLVM3.2 is in LTO mode.
One is that dragonegg supports only the two mention targets. You even can not benchmark SPECFP for LLVM on other targets as you can not use LLVM to compile Fortran programs.
Another one is that the quality of generated code generated by LLVM for other targets is not so good as one generated by GCC. For Power7 example (second most important server architecture), LLVM rate for SPECInt2000 or part of SPECFP2000 (4 benchmarks on C) is about 20% worse than for GCC.
For ARM (PandaBoard, OMAP4430), SPECInt2000 code performance generated by LLVM with -O3 is >10% lower than one generated by GCC4.8 with -O3. Also GCC generates >5% smaller code with -Os than LLVM with -Oz (it is even more strict than -Os) which is quite important for this embedded processor. The results for ARM has been gotten for 9 of 12 benchmarks (unfortunately I did not manage to compile eon and perlbmk by LLVM and as usually LLVM3.2 generates a wrong code for gap in any optimization mode).
So I would not recommend switching to LLVM for any Linux distribution because other targets are not refined as x86/x86-64 with performance point of view (there are a lot of other aspects besides generated code performance which make such switching unreasonable). By the way, only LLVM-3.2 binaries for MACOS provided by LLVM site are compiled by LLVM itself. For Linux and FreeBSD (this project officially switched from GCC to LLVM because of new version GNU license for GCC), the binaries are still compiled by GCC (correspondingly by GCC 4.6 and GCC 4.2).
Still I think that GCC community should pay more attention to improving code quality for x86/x86-64 as LLVM is catching us up.
Last modified: 02/04/2013 - vmakarov at redhat dot com
Return to index page.