How does SAS performance compare with Julia?

Be Sociable, Share!

    I have been a SAS programmer and architect for years:  I did my first professional work in the 1990s.  I also have made friends among people who focus on SAS.  I had mentioned in a SAS Consultants forum in Google Groups that the open source Julia language was delivering impressive results compared with the R language.    The testing involves several math problems which could be translated into most any language.  Previous results (and code) has been already shared on http://julialang.org/

    I had wanted to try SAS to see how it does against Julia, but I did not have a machine where I could load both software applications.  However, Dom Pazzula not only tested it on his machine, but also provided the comparison code.  This blog post has the results of the test and the SAS code for you to try the comparison yourself.

    SASvsJulia How does SAS performance compare with Julia?

    The times in the following table are in milliseconds (lower is better). SAS was tested with version 9.3 on 32-bit Windows.  Dom believed that 64-bit would run slower on the same hardware; he also claimed that there would be no advantage to SAS Grid or Connect spawning due to the communications overhead and the sessions spawning times which would overwhelm these comparatively small math problems.  Put into other terms, the fixed cost of high-capacity server solutions does not demonstrate performance benefits on small problems.  Python is using NumPy complied with the Intel MKL for the matrix multiplication. Dom commented that he was not sure what Julia is using behind the scenes for matrix multiplication, but it’s impressive.

    Dom rewrote all the code to report the average of several loops instead of a one-time single minimum value (when I did a comparison without SAS, I added more iterations to the loop since my computer was more speedy than the one used for the Julia metrics).  Because of this rewrite, the SAS code (later) is not what produced this table, but rather a modification to test for averages (since some of the initial reports were reporting zeros).  You may have to make one of these two types of modifications.

     

    Julia

    R

    SAS

    Python

    Fib

    0.08

    76

    0.8

    3.2

    Parse Int

    12.67

    10

    0.2

    3

    Quicksort

    0.71

    252

    5.4

    15.6

    Mandel

    0.55

    30

     N/A

    6.2

    Pi Sum

    42

    750

    45.2

    675

    Rand Mat Stat

    41.2

    146

    35.1

    197

    Rand Mat Mul

    131.8

    1012

    1198.2

    109

    I did not ask Dom about his machine specs because the point of this post is to provide a magnitude benchmark of the results.  Also, this type of comparison needs to be done on your own machine and environment, and therefore any number of tests in another environment may not match the results you might get.  These benchmarks are among the most straight-forward anyone could do when comparing the speed of mathematical languages.

    Dom did not perform a mandlebrot calculation in SAS because there are no complex numbers.  Dom commented that SAS could benefit from multithreading the libraries underlying IML.  Also, the SAS sort code is a Quick Sort algorithm implemented using PROC FCMP instead of the SAS native PROC SORT.

    Though Dom sent me the SAS code, I decided to also double-check the code by running it on my available installation of SAS 9.3.  Again, this SAS code is a translation of the original code to compare Julia, and NOT the rewritten code which produced the table above.  I like his use of PROC FCMP, and I added the name of the two macros with the %mend statement (which is a best practice). Thus, this SAS code is what you can use to test speeds.

    options nonotes nosource;
    options cmplib=work.cmp;
    %macro start_timer(n=5);
    	el = 1e32;
    	do _counter=1 to &n;
    		s = datetime();
    %mend start_timer;
    %macro end_timer(test,put=put);
    		e = datetime();
    		el = min(el,(e-s)*1000);
    	end;
    	&put "&test " el;
    %mend end_timer;
    proc fcmp outlib=work.cmp.fns;
    	function fib(n);
    		if (n < 2) then
    		return(n);
    		else
    		return(fib(n-1) + fib(n-2));
    	endsub;
    	subroutine qsort(arr[*],lo,hi);
    		outargs arr;
    			i = lo;
    			j = hi;
    			do while (i < hi);
    				pivot = arr[floor((lo+hi)/2)];
    				do while (i<=j);
    					do while (arr[i] < pivot); 						i = i + 1; 					end; 					do while (arr[j] > pivot);
    						j = j - 1;
    					end;
    					if (i<=j) then do;
    						t = arr[i];
    						arr[i] = arr[j];
    						arr[j] = t;
    						i = i + 1;
    						j = j - 1;
    					end;
    				end;
    				if (lo < j) then
    					call qsort(arr,lo,j);
    				lo = i;
    				j = hi;
    			end;
    		endsub;
    	run;
    quit;
    
    data _null_;
    	%start_timer();
    	t = 0;
    	do j=1 to 500;
    		t = 0;
    		do k=1 to 10000;
    			t = t + 1/(k*k);
    		end;
    	end;
    	%end_timer(PI SUM);
    	run;
    data _null_;
    	%start_timer(n=5);
    	do i=1 to 1000;
    		n = floor(2**31-1*uniform(1));
    		str = put(n,hex.);
    		in = input(str,hex8.);
    	end;
    	%end_timer(PARSE INT);
    	run;
    data _null_;
    	%start_timer(n=5);
    	f = fib(20);
    	%end_timer(FIB);
    	run;
    data _null_;
    	array arr[5000] _temporary_;
    	%start_timer(n=5);
    	do i=1 to 5000;
    		arr[i] = uniform(1);
    	end;
    	call qsort(arr,1,5000);
    	%end_timer(QSORT);
    	run;
    proc iml;
    	t = 1000;
    	file log;
    	%start_timer(n=5);
    	n = 5;
    	v = J(1,t,0);
    	w = v;
    	do i=1 to t;
    		a = normal(J(n,n,123));
    		b = normal(J(n,n,123));
    		c = normal(J(n,n,123));
    		d = normal(J(n,n,123));
    		P = a || b || c || d;
    		Q = (a || b) // (c || d);
    		v[i] = vecdiag((P`*P)**4)[+];
    		w[i] = vecdiag((Q`*Q)**4)[+];
    	end;
    	s1 = std(v)/v[:];
    	s2 = std(w)/v[:];
    	%end_timer(MAT STAT);
    	%start_timer(n=5);
    	a = normal(J(t,t,123));
    	b = normal(J(t,t,123));
    	c = a*b;
    	%end_timer(MAT MULT);
    quit;

    The Julia language continues to be under construction.  Its goals are intended to improve performance for a wide variety of calculations which people commonly do with R and Matlab.  As such, some of the Julia developers have also worked on R and other projects too.  I have mentioned in several presentations that if I were in my early analytics career, I would take some time away from video gaming and put it into the Julia language.  Open source (not just Julia) provides a pathway to establish your own expertise in promising new technologies.

    Because I already have my own list of academic and professional achievements, I will not likely participate in Julia development.  Nevertheless, I have already been presenting and continue to want to present on the features as Julia grows up. I enjoy sharing such benchmarks because I want people to be aware of how they can test mathematical languages on their own.  Again, I repeat that the benchmark results I published may or may not represent your environment, and I do not expect you to get identical results.  Instead, you can use this SAS code and the other language code to see what results you achieve on your own hardware.

     

    Be Sociable, Share!