Raman Iyer Interview — Microsoft Analysis Services

Be Sociable, Share!

    Hi Everyone

    I’m excited to present the first interview that I am posting to this blog. In these interviews, I’m generally looking for leaders and influential voices in business intelligence in general and data mining in particular.

    This first interview is with Raman Iyer, and experienced leader and developer for Microsoft. Raman chose to write responses to my questions, and he included links too (knowing that the product would be web-based). You can find Raman supporting http://www.sqlserverdatamining.com and often responding to technical questions on MSDN Forums.

    Quick bio:
    Raman Iyer is Principal Development Manager for the Analysis Services Engine development team, responsible for building the server that powers Microsoft’s core Business Intelligence offerings in SQL Server, including OLAP, PowerPivot (In-Memory BI) and Data Mining. He was a founding member of the SQL Server Data Mining team, developing early prototypes and core DM engine features in the 2000 and 2005 releases before going on to lead the Data Mining development team through the SQL Server 2008 and DM Add-in releases.

    How did you come to work for Microsoft?

    A long time ago, in a galaxy far, far away (now part of the SAP universe), I was happily geeking out on their Replication Server product while doing a side-project porting client libraries to Linux, when an old school friend of mine called excitedly about how he had just been hired by Microsoft, and how terrific it was, and that I should at least consider interviewing. After some initial resistance, I grudgingly gave in and flew out to Redmond. I ended up loving the work environment, the fantastic campus and the super-smart folks I met (not necessarily in that order). I got offers from all the three groups I interviewed with and decided to sign up with the SQL Server Storage Engine team – even though, at that time, I had no clue what those “options” were that the recruiter was enthusiastically pitching. I actually wound up starting work at Microsoft before my friend. The rest, as they say, is history.

    How did you come to work with SQL Server Data Mining?

    While SQL Server 2000 was under development, one of the senior Program Managers on the SQL Storage Engine team was talking with the machine learning folks at Microsoft Research about productizing some of their innovative work. Our paths met when he was at the cusp of forming a new group to build a new Data Mining platform that married the Microsoft Research work in decision trees and scalable clustering with the database group’s expertise in building robust server technologies and data access APIs. His passion was infectious and I caught the Data Mining bug (and haven’t managed to rid my system of it ever since), moving to his team as the first DM developer (after a round of interviews with the researchers and some OLAP developers, including Mosha).

    What types of roles have you played in developing SQL Server Data Mining?

    I have worn many hats on the SQL Server Data Mining team since its inception, and it’s been a long, strange (but wonderful) trip.

    In the earliest days, I was the only product team developer in a team made up of mostly of Microsoft Research members – in fact, I was physically located in the Microsoft Research building for the first few months of the project for that reason – and built some early prototypes while we were working on the core of the DMX language and the OLE DB for Data Mining specification. I still remember the first “AutoFill” Excel add-in I wrote using some research code for decision trees – a precursor to the “Fill From Example” Table Analysis Tool that’s part of our popular SQL Server DM Add-ins for Excel. A little after I got started, our fearless leader suddenly decided to join a seminary and become a priest, and our fledgling Data Mining [DM] effort officially got merged into the OLAP Services team led by Amir Netz, becoming the reason it got renamed to “Analysis Services”. The OLAP team had already shipped their SQL Server 2000 beta, so we had to scramble – hard and fast – to put the DM engine, the DMX language, the OLE DB for Data Mining APIs, and the decision trees and clustering algorithms into the SQL Server 2000 release.

    After successfully delivering on that short timeline for SQL 2000, we worked on significantly extending the SQL Server Data Mining platform for the long release that was SQL 2005, adding several new algorithms in collaboration with Microsoft Research (MSR) – we actually had a couple of MSR devs working full-time on the product team – and other core concepts like the mining structure, OLAP mining models and DM dimensions, which I played a key role in designing and developing. We were also rewriting the Analysis Services Engine from scratch, and myself and the other DM developers helped build core infrastructure pieces as well. As we started wrapping up the development work for SQL Server 2005, I also took on the role of evangelist, writing articles and white papers, helping put up the and run http://www.sqlserverdatamining.com/, and authoring the periodic “Data Miner” newsletter. After SQL 2005, I took over as the development lead for the DM team, helping ship new DM features in SQL 2008 and two releases of the DM Add-ins for Office, while continuing the evangelism.

    I am currently the development manager for the Analysis Services Engine team, which builds the server behind OLAP, PowerPivot and Data Mining.

    How do you believe your training and education helped you for the work you have today?

    Surprisingly enough, I did not have a Data Mining, statistics or machine learning background when I started working on SQL Server Data Mining. I did have a degree in computer science and engineering, and I had experience with database systems and data access APIs. Those were the foundational skills that helped me contribute in significant ways to SQL Server Data Mining and the Analysis Services Engine infrastructure. And it makes sense when you consider the fact that SQL Server Data Mining is not just about the DM algorithms; it’s about the server infrastructure, security, scalability, and about making the DM technology accessible to SQL developers and easy to deploy – in short, all things that myself and the rest of the team (which also includes PhDs in Data Mining, by the way) were reasonably proficient at. The core engineering skills of building robust, scalable server software are also ones that are portable to new domains as both your interests and your work requirements change.

    I also spend a significant chunk of my time managing and mentoring people, and helping make the right business and engineering decisions. The time I took to earn a mid-career technology management MBA has certainly helped change how I think about leadership, negotiations, business strategy and decision-making.

    What do you enjoy about the team you work with?

    Beyond the fact that the team is composed of top-notch technical folks, I really like the fact that team members from very diverse cultures and backgrounds can work together very well – there are strong opinions, a lot of passion, and some yelling (though things have mellowed down a bit in recent times) but no one takes it personally and it’s always about doing the right thing for the product and for the customer. We are definitely a team that embraces diversity tension.

    What positive responses have you heard from people using SQL Server Data Mining?

    My favorite response is when an executive at a large European bank exclaimed that he could put a copy of SQL Server Enterprise Edition on the desktop of every data mining analyst in the organization and get the same quality solutions for less than the price they were paying for licensing their current data mining software.

    Some of the most frequent user comments are about how it is possible to build and put a solution into production in days, rather than months, with SQL Server Data Mining. This is particularly gratifying because ease of development and deployment was a core tenet when we designed the product. Of course, I always like to hear about the big boosts in sales that customers have achieved with SQL Server Data Mining solutions.

    A recent interaction I had with a healthcare research group got me thrilled to no end – not just about SQL Server Data Mining but also about the promise of data mining in general. The research head related how he was building models to predict hospital admissions and discharge rates and was astounded by the fact that SQL Server Data Mining indicated that a mostly-ignored lab test was the single most important determining factor for admission. He was skeptical at first, but when he investigated further, he found very recent research that confirmed the hypothesis. He is now an enthusiastic convert and cannot stop talking about new insights that Data Mining is capable of delivering in real-time to healthcare professionals.

    Why do you believe technical people want to work for Microsoft?

    To me, Microsoft is unique in (a) the breadth and depth of the technology areas it covers in software – you pick an area of interest from anywhere on the wide spectrum from gaming to operating systems to search to databases and it’s quite likely that Microsoft is working in that space – and (b) providing you with the ability to directly impact the work and lives of millions. If you couple that with the work environment, the benefits and the opportunity to work with some of the smartest folks on the planet, the answer is obvious.

    What encouragement do you have for students considering technical careers?

    The industry is in the midst of a huge transformation in terms of how software is developed, licensed, deployed and serviced, and it is a great time to launch a technical career and participate in this upheaval. My advice for students is to focus on getting their CS [Computer Science] fundamentals right, and getting real-world experience shipping commercial software through internships at companies where you are allowed to do that. Software is not only a key driver of economic growth; it’s also now an essential part of our personal lives. So – though I am personally biased towards business software – keep in mind that the consumer space is wide open as well, if that is the area that excites you. Passion is very important for success as well as for sustaining your energy in the long-term and keeping you from burning out. So is the interest (and the ability) to continue learning all your life.

    Beyond the technical, are there any personal passions or interests you want to share?

    At the moment my hands are full outside of work balancing the demands (and quite frankly, the magnetic cuddliness) of a new baby with making sure that my four year-old gets quality time so he doesn’t feel neglected (he’s been coping very well so far). Music of all stripes has been a longtime passion and I hope to be able to spend more time at least listening to old and new music as my second son gets a little older, though actually getting back to practicing my long-lost (and dubious) vocal skills seems like a distant dream. My other interests include reading (I’m looking for an excuse to buy the new Kindle – alas, my Kindle 2 is in perfect working order), audio equipment, and new gadgets.

    What would you like people to know about the future of SQL Server Data Mining?

    SQL Server Data Mining has always been about bringing the esoteric world of data mining to new audiences and helping them solve practical problems with it. I believe we have built a fairly comprehensive platform that addresses the needs of the broad user base we targeted: developers and business users, including end users who spend a lot of time manipulating data (or what we call “information workers” at Microsoft). Our continuing challenge is around how to make more of these folks aware of these capabilities, including the fact that predictive analytics is readily available to them (at no extra cost) if they have a SQL Server license, and coming up with new ways to showcase the power of the product and simplify it even more. At this point, it’s too early in the release cycle to publicly talk about specific details. You will hear more about product plans at PASS Summit 2010 coming up in November.

    Can you share your experiences building PowerPivot?

    PowerPivot was a foray into a completely new space for the Analysis Services team – we were used to producing server/platform technology and a development experience for the BI professional, and suddenly we were pushing ourselves to create a data mashup/reporting experience for the end-user that would put a lot of power into their hands while keeping it eminently approachable. To support the simplified, relational-style model, we had to build an all-new in-memory storage engine and a new expression language (DAX) from the ground up, and integrate it not only into the Excel environment but also into SharePoint on the server side. To top it all, we had to do it in the relatively short time period allocated to an incremental “R2″ release. The team pushed incredibly hard and we had to make some tough engineering choices, but in the end, we delivered something that’s causing a fundamental shift in how Business Intelligence is produced and consumed, and I am personally proud of having been (and continuing to be) part of that.

    One of the vital pieces that I helped design for PowerPivot was embedding our data into the Excel workbook and the complex internal dance with Excel and Excel Services that enables the user to seamlessly work with and switch between native Excel data and the millions of rows of PowerPivot data. I vividly recall our initial meetings arguing about whether the “one file” approach was feasible, and the “Aha!” moment for the team months later when we demoed the first single workbook with Excel and PowerPivot data living harmoniously side-by-side.

    After that epic effort to get the first version of PowerPivot for Excel and SharePoint out, obviously we could not sit still. We had to go fix all the rough edges inherent in a v1, and of course we had to make it better in new ways. That’s exactly what we are busy working on right now. Again, you will need to wait for PASS Summit 2010 to get all the details. All I can say for now is that I am super-excited by the direction we’re taking and how things are shaping up so far.

    Be Sociable, Share!