When Building for Big Data, Remember to Think Small
October 21, 2013 Leave a comment
- With a continued focus on top down, company-wide all-encompassing projects, big data is in danger of turning into the next service oriented architecture (SOA) – a good idea that simply cannot be realized.
- Conversely, Microsoft’s diminutive self service business intelligence solution, Power BI for Office 365, highlights the potential in thinking small with big data.
I never win anything. For that reason I never gamble and have never, ever entered the Publishers Clearing House sweepstakes. But last week on a whim I requested a beta invitation for Microsoft’s forthcoming self service business intelligence (BI) service (Power BI for Office 365 preview). Lucky me, I won an invite and immediately began pawing through the available documentation and downloading a few samples. What did I find? Sometimes the biggest insights can be found in the smallest of packages, even the seemingly unpretentious spreadsheet itself.
Now I must confess that prior to this experience, I held a hefty bias against spreadsheets (by Microsoft and others). In the past, I had attempted to use them to divine insight from some of our primary research projects, which are made up of maybe 600 rows (one per survey respondent). In every instance, I came away with the feeling that spreadsheets were better suited to managing expense reports rather than delivering data visualization and discovery. Where do you begin? How do you convert a set of tables and rows into some form of insight?
What I discovered very quickly in working through this product preview is that I was working without the benefit of a guide and without the support of any analytical tools. Simple, I know. But that’s not what made the experience valuable. It was the familiarity and “smallness” of working in Microsoft Excel that made the biggest impact. You could work with some pretty sizable data sets, but you could just as easily derive knowledge from something as small as a brief questionnaire.
In my travels as an analyst, I see many solutions that are extremely well thought out and very capable of working with multiple terabytes of data in motion and in memory. But what has worried me about these offerings and about the way big data itself is being positioned within the enterprise, is a sense of “bigness,” that in order to succeed with data and analytics, enterprise users must basically re-invent the notion of the data warehouse, gathering and massaging all corporate data regardless of where it lives.
With this focus on top down, company-wide all-encompassing projects, I think big data is in danger of turning into the next SOA – a good idea that simply cannot be realized, at least not in its entirety. Conversely, Microsoft’s diminutive self service business intelligence solution, Power BI for Office 365, highlights the potential that rests within the familiar from working with smaller tools from the bottom up.
Mind you, Microsoft is not alone in approaching big data a bit at a time. There are a host of cloud-based, self-service BI solutions coming down the pike from smaller players such as BIME and Birst, and even from established entrants GoodData and MicroStrategy. In all cases, these products aim not to redefine or rebuild the data center but instead to equip everyday users with the tools necessary to make use of the data they have at hand. That’s a scenario where everyone wins, IT professionals, users and CxOs.