At my work, we are trying to switch to Linear Algebra Solvers that out performs Lapack due to the size of problems we are interested in.
I just successfully installed and tested Magma1.3 along with CUDA 5.0 on an Ubuntu 11.10 server. I hope my note here would be helpful for whom just switched to CUDA 5.0 and want to use Magma 1.3 with the improved CUBLAS library.
1. I downloaded Magma 1.3 from http://icl.cs.utk.edu/magma/software/index.html and extracted it.
2. To compile Magma, I had to provide a make.inc file. Thanks to Magma Team, they'd provided sample make.inc files for different environment. There are four options,
- Intel's MKL
- AMD's ACML
- Netlib's LAPACK + ATLAS
- Netlib's LAPACK + GotoBLAS2
For each option, the sample make.inc.$(LIB) file are provided. Following the advice and instructions from http://www.pavanky.com/installing-magma-with-gotoblas2/, I picked LAPACK and BLAS2 as well. Therefore, we have to install GotoBLAS2 first.
3. GotoBlas2 can be downloaded at Taxes Advance Computing Center. The GOTOBLAS2 is not in active development, so after downloading and untaring, it is not surprising I had to do some miner change. Thankfully, the link I was referencing provided the trick. In f_check file add a new line after line 237,
$link =~ s/\ -rpath\ s+/\ -rpath\@/g; # line 237
$link =~ s/\-l\ /\-l/g; # Add this.
I'm using gcc4.6, it doesn't like empty linking string after "-l". The patch fixed the problem.
After compilation, copy libgoto2_barcelonap-r1.13.a to the library folder
Note: For Intel Xeon-based architecture, we need to uncomment the Nehalem line in file getarch.c.
Change /* #define FORCE_NEHALEN */ to #define FORCE_NEHALEN
4. I referenced
a post on Magma forum for modifying make.inc, to compile Magma test files which uses gcc-4.5 and gfortran-4.5, I had to make soft links from gcc-4.6 to gcc-4.5 and gfortran-4.6 to gfortran. I also had to change -lgoto to -lgoto2 and add flags -lblas, -lgfortran, -ldl and -lstdc++. Ofcourse, other environment variables are set to match my machine as well.
My make.inc is as the following:
GPU_TARGET = Tesla
CC = gcc-4.5
NVCC = nvcc
FORT = gfortran-4.5
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
OPTS = -O3 -DADD_
F77OPTS = -O3 -DADD_
FOPTS = -O3 -DADD_ -x f95-cpp-input
NVOPTS = -O3 -DADD_ --compiler-options -fno-strict-aliasing -DUNIX
LDOPTS = -fPIC -Xlinker -zmuldefs
LIB = -lblas -lgfortran -lgoto2 -lpthread -lcublas -lcudart -llapack -lm -ldl -lstdc++
CUDADIR = /usr/local/cuda
LIBDIR = -L/usr/local/cuda/lib64 -L/usr/lib64
INC = -I$(CUDADIR)/include -I/usr/include
5. The last things is that for some reason, the /usr/local/cuda-5.0/src/fortran_common.h skipped
defining micro CUBLAS_GFORTRAN for linux based machine so it breaks the compilation for
test files. I added few lines at lines 61 to define the micro CUBLAS_GFORTRAN as the following,
#if defined(_WIN32)
#ifndef CUBLAS_INTEL_FORTRAN
#define CUBLAS_INTEL_FORTRAN
#endif
#elif defined(__linux)
#ifndef CUBLAS_GFORTRAN
#define CUBLAS_GFORTRAN
#endif
#endif
I'm not sure if everybody needs this step, because I suspect myself might havedone something to lead the CUDA 5 installation run script to skip Fortran option a while ago.
If you are following my steps, you should be now good to go. I'm leaving work now to ...hopefully see Stanford get the Pack12 Championship, but this is the the
speed up from a Tesla C1060 and nForce 980a/780a equipped old machine (my playground :p),
N CPU Time(s) GPU Time(s)
===================================
1024 2.12 0.64
2048 17.00 1.22
3072 56.24 3.31
4032 126.69 6.43
5184 279.86 12.52
I'm already excited about the speedup. Good luck!