A Breakthrough in Big Data Visualization from MIT

(US and Canada) The scale and pace of data collection have driven a constant need for innovation in how data is presented and analyzed. The massive data sets of today require a new visualization paradigm that allows more users to dynamically engage with data in an immersive environment without requiring special skills or training. Hopara, an innovation recently devised at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), delivers more data into the hands of average users, accelerating the promise of creating data-driven organizations and maximizing ROI of investments in data collection.

 Does your current dashboard paint the screen black or overwhelm the user with too much data on important big data datasets like the picture here so users can’t tell what’s going on? Does your current system allow users to see the big picture as well as drill into the details, enabling an empowered person to take action on the data? Does your current system allow the average user to fly over data? In other words, can it act like a 'data drone'?

On small data sets, none of this matters. However, most enterprises are drowning in data and struggling to convert investments in data collection into value for the company. Before the user of a data visualization app can perform specific analytics, like run simple analysis between data element X and data element Y, they need to “see the lay of the land.” They need to fly over the data and drill into areas of interest. That is what Hopara does for data scientists and their enterprise customers.

Big data and small data

It is obvious to everybody that the amount of data collected is going through the roof, with 90% of all data collected in just the past two years (Source: US Chamber of Commerce). This is driven by cheap internet of things (IoT) sensors, exploding transaction volumes and the substantial value of using such data in decision-making. This data deluge is breaking traditional techniques of visualizing and analyzing data, which were designed for traditional small data sets of thousands of records. Current data sets may have millions or even billions of records. In this new world, users and analysts must adopt a two-step approach.

Analysis is a two-step process

When seeing a big data set, the first step has to be one of exploration/discovery so average users can find their bearings in the massive data set and not get overwhelmed. The general question is “tell me something interesting” and “help me find the context.” In other words, one wants to fly over the data to see what it looks like and to find things of interest. When something interesting is found, one wants to drill down into the area of interest to see more detail and discover what is really going on. For example, once a user has discovered that suspicious activity is occurring in the half hour before closing a store, then she can move to the second phase, which is to analyze the subset of the detailed data. As such, the pattern is exploration/discovery followed by analysis, and most importantly informed action!

Discovery and exploration leading to action

At scale, conventional toolkits such as Tableau, Spotfire and Excel don’t give users of data the required features for discovery & exploration needed to enable decision-making and action, such as:

• Fly over

Ideally, I want a data drone whereby I can fly over my data looking for interesting features. This requires the ability to place my data on a large canvas and fly over this canvas. Technically, I need the ability to pan over a large data set.

• Zoom

When something interesting is observed, I need the ability to zoom into areas of interest to see more detail. Sometimes this involved increasing the resolution of the display to see more detail. Google Maps is a good example of zooming by lowering the area in the window and increasing map resolution. However, sometimes, zoom entails changing the window semantics, such as moving from a map of IoT sensors to a bar chart of a particular sensor output over time. In summary, sometimes users need syntactic zoom and sometimes semantic zoom.

• Teleport

Sometimes I want to teleport from my office address to my home address or from my home address to the school of my children. This entails moving to a completely different display.  

A good discovery system is one that lets users fly over, zoom and teleport on any kind of enterprise data. That is exactly what Hopara does.

Data Drone Navigation

As data and user needs evolve, modern visualization tools can now add exploration at scale. Creating an easier, more immersive data experience for average users is critical to fulfilling the promise of a data-driven organization. If people can’t--or won’t--engage with the data because it’s too hard, complex or confusing, their decisions will not be driven by data.

Imagine letting any user fly around a set of data -- just like you would in a virtual environment like an online video game. A Google Maps-like paradigm makes it possible to zoom in for detail on demand, or to pan across a custom canvas, find adjacent data points, or click to

Data Drone Navigation is a spatial indexing and caching technology developed at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) under the leadership of Prof. Mike Stonebraker and commercialized as a component of Hopara. It provides next-generation data visualization navigation that lets viewers pan across large sets of data, zoom in for hyper detail, and teleport across vastly different canvases. Beyond traditional navigation, it gives viewers a fully engaged immersive experience without requiring special skills or training, making data more approachable, less complex, and more actionable. Just grab your mouse and start flying through the data.

This is what we’ve built at Hopara. It’s a next-generation data exploration system designed to put more data into the hands of more users. Our Google Maps-like Data Drone Navigation enables a more immersive experience that provides easier detail discovery, faster enlightenment, and more intelligent decisions by anyone/everyone in your organization (no computer science degree required).

For example: users can “fly” from a scatterplot of millions of data points to a multi-floor custom floor plan, then drill down to individual sensors revealing detail of underlying data sets, then pan across time series or jump to another floor in sub-seconds. It lets you travel to where you want to go, without having to know the “how” of the underlying data that gets you there.

"Data Drone Navigation" is a Service Mark of Hopara, Inc. All Rights Reserved.