Two weeks ago, Matt Coventon, ISE’s Big Data Practice Lead, and I went to New York City for three days to attend the annual Strata + Hadoop Big Data Conference. If you are not familiar with it, Strata + Hadoop World is the largest Big Data conference of its kind. The conference brings together data scientists, software engineers, innovators, executives, and software providers who want to explore and learn more about the latest emerging data technologies. I am a relative newbie when it comes to Big Data, so my objective was to go and learn as much as I could about the market and the technologies.
The three days were filled with non-stop tutorials, keynote addresses, breakout sessions and time to wander the expo center floor talking to technology vendors. So, after all of the non-stop learning, these are the three big ideas that I took away:
1. There are no ‘Out of the Box’ solutions
Despite the plethora of open source tools and languages, proprietary analytical tools, frameworks, and Business Intelligence packages, there is no easy, out of the box solution. Big Data is still in its infancy as a discipline and the technologies are changing faster than most can keep up with. What is lacking is an ‘off the shelf’ solution that allows you to pick one supplier to solve all of your big data needs. The challenge of architecting a Big Data solution is picking from the vast array of tools in the Big Data stack. What is ‘cool’ and ‘hot’ today, may be left behind in just two years. Companies that want to implement a successful Big Data project will need to understand that without a single technology provider to turn to in the marketplace, they will need to partner with a firm that will be able to develop a custom solution to meet their unique needs.
2. There must be a pony here somewhere…
Everyone knows that there will be a big payoff for companies that successfully leverage their Big Data capabilities. One of the more interesting sessions that I attended was a panel discussion of Venture Capitalists that are currently specializing in the data market space. The key takeaway came when one of the VCs said that they are focusing on investing in technologies that support industries that already have the data. If your industry does not already have a robust data collection process in place and a significant amount of data already collected, then you will most likely not be part of the early wave of companies capitalizing on this technology trend.
The VC went on to say that her firm was targeting two industries that are ripe for leveraging their data, healthcare and manufacturing. Healthcare is obvious. Because of insurance processing, all of our healthcare records are currently being collected. She went on to say that the Healthcare industry is fraught with risk, due to HIPAA regulations and privacy concerns. Manufacturing, on the other hand, has been putting sensors on the factory floor for over 25 years and has been collecting tons of data. This is a market ripe to leverage Big Data.
3. The demand for Big Data talent has far outpaced supply
As I talk to clients and attend various industry meetings and conferences, I keep hearing two themes over and over again. “Big Data is going to be a game changer,” and “We can’t find people with strong data skills to support our data growth.” In Automated Vehicles, Precision Farming, and Manufacturing, everyone is focused on the data. How to collect it, how to organize it, how to analyze it, and how to display it. I have read several news articles recently stating that there are over 80,000 unfilled data scientist, data analysts, or data engineer positions in the U.S. right now. The demand is high, but there are nowhere near enough experienced data technologists to support the rapid growth.
At ISE we are growing our Big Data practice. We are committed to providing the right value added skills our clients need. Many of our business-minded engineers are evolving to become business minded data engineers. This is an exciting time to be in the Big Data market, providing custom software professional services.