Note: Separate multiple email address with a comma or semicolon.
Note: Separate multiple email address with a comma or semicolon.
Your Email Address:
Internet-scale consumer companies like Facebook,1 Google and Yahoo have inspired and directly created the underpinnings of the big data infrastructure software available today. While the rest of the world can now leverage offerings like Hadoop or BiqQuery, few companies have the internal resources required to build in-house “last-mile” applications (i.e., applications able to make big data useful for the masses) on top of this infrastructure in order to actually make big data useful to them. Thus, startups emerged to fill this gap.
The first wave of big data application startups offered data tooling to technical users to help reduce manual coding and improve the efficiency of analytics at scale. More recently, we’ve seen entrepreneurs directly attacking core enterprise software applications, such as customer relationship management, helpdesk, and security, as well as industry-specific applications such as farming, healthcare, etc.. creating a wave of Data-Driven Software (DDS).
DDS is end-user rather than manager-centric. It leverages an organization’s data footprint in new ways to help users do their jobs more efficiently and effectively. For instance, Origami Logic collects data from myriad sources to empower marketers to answer the question “What happened today?” by measuring the marketing signals that matter. Marketers can now react and respond on a daily basis instead of monthly.
Ultimately, we believe that all software should be data-driven, and incumbent vendors in every category of user-facing software must either adapt or risk being disrupted.
The greatest misconception is that big data and DDS are only for large companies. Every company at any size needs to think about becoming a more data-driven organization in order to provide the best experiences for customers and end users. Businesses that fail to focus on their data and extract insights from it will inevitably be disrupted.
There are two interrelated challenges facing the big data industry: programming interfaces and hiring talent. Soon after the first modern databases were developed in the 1970s, SQL (American National Standards Institute’s Structured Query Language) emerged as a lingua franca, or common language, used to store and retrieve data. Today, we live in a sea of interfaces and tradeoffs created by different big data platform layers attempting to serve different needs. Compounding this problem is the supposed need for the mythical, all-knowing data scientist who understands math, statistical modeling, code and the intricacies of data pipeline engineering, as well as the business domain and storytelling. Classroom education may be part of the solution, but we absolutely need software to help bridge this gap; however, with companies built from the ground up to work on big data, such as Trifacta1 for data preparation and Zoomdata1 for visual analytics, we believe the industry is on the right path.
We’ve seen a lot of focus on developing advanced analytic methods and frameworks, such as natural language processing, e.g., Apple’s Siri, machine and deep learning, e.g., IBM’s Watson, and now artificial intelligence. While horizontal platforms to apply these algorithms at massive scale are interesting for the most sophisticated of companies, few have the requisite data volume and skill sets to apply these techniques effectively. Thus, in the short term, it will likely be the verticalized players, such as IBM or Apple, who bring the power of these technologies to the masses.
A somewhat related recent tech meme is the rise of chatbots—programs designed to simulate intelligent conversation with human users. With large enough pools of training data to learn responses from as well as recent advances in natural language processing, we are on the cusp of machine-driven conversational interfaces that are palatable to end users. While a chat interface is far more constraining from a design perspective than a traditional graphical user interface, there is an inescapable comfort and familiarity it offers end users. It’s a magic curtain behind which a company can offer any blend of human- and machine-created responses they desire.
For example, Demisto1 Enterprises’ Security Operations platform builds upon both these trends. It’s the first intelligent automation and chatOps platform for security operations centers. Unlike most security vendors that analyze machine data, Demisto’s platform learns from human data in part by offering an interface through which almost every action taken to resolve an incident is recorded. This allows DBot, the Demisto security chatbot, to determine what it can help expedite or even automate for incident responders.
This depends on the lens through which one defines “large” big data players. If it’s the “holders” of big data, such as Amazon, Apple, Facebook and Google, then it’s clear that they enjoy huge data network effects and learning feedback loops—advantages that make it very hard for industry upstarts to compete. Google’s decision to open source its deep learning platform, TensorFlow, is a clear indication that Google believes that its data is the basis of its competitive advantage, even more so than its technology. In order for a startup to compete on a given application, it will need to find some way to create its own data advantage.
However, if you look at the containers for big data—the database management platforms—then the answer is different. Traditional database management vendors and new purveyors like Cloudera1 are not in the same market. Platforms like Hadoop are, by and large, augmenting and not replacing current infrastructure. It’s classic new-market disruption. Hadoop has lower performance in certain traditional attributes but has improved performance in new attributes—namely scalability and the ability to handle unstructured or semi-structured data. The business model here is different as well; from our perspective, to be successful over the long term, it must make money at a lower price per unit sold, which in this case is the amount of data.
Value slowly moves up the stack. Technologies always get commoditized, but there are always new layers of innovation. More importantly, we believe it will be the enterprises that don't embrace big data technologies that become commoditized and irrelevant, whereas enterprises that appreciate and leverage their data assets will be the ultimate winners in their respective categories.
That being said, we are in the very early innings of this big data revolution. While there has been a lot of hype, there is still plenty of ongoing innovation and room for massive growth of new ideas. It will take years for these technologies to become mainstream. In order for average businesses to benefit from their investments in big data analytics, they will require software to deliver client solutions constructed specifically for the industries in which they operate.