Big Data is feeling the heat. Among others, the heat of IoT and the heat of Big Data Real User Monitoring and Application Performance Management. In this post, we’re going to look at how Big Data is revolutionizing how it deals with IoT data as well as the effects of Big Data in APM and RUM.

Big Data and IoT: Connecting the MANY Dots

As more and more devices get connected, the term big data might not be useful anymore. As it is already bigger than just big, there will be no word to describe the amount of data in a world where billions of items are connected, each with their own sets of data. Medical devices, heart monitors, children’s toys, cars, and even…Yes. The beds we sleep on. All of this data will have to be analyzed in one form or another with the goal being able to gain knowledge about patterns and future trends that will be used to enhance everything from the air we breathe to conserving energy, from monitoring our health to driverless cars driving us to work. But, it isn’t the data that will yield the information we need, but its analysis.

There aren’t too many conversations in the IT arena nowadays that don’t mention both. They feed off each other and are tied to one another.

Here, we’ll take a look at both practices and then see how they are connected.

Big Data has been around for quite some time. Data coming from many sources at mind-blowing bursts of speed. Data of all types, of all formats. But, as mentioned before it isn’t the data but the analysis of data that makes up the Big Data X-factor.

IoT is rapidly becoming the path of choice for new device startups and even old-time makers of devices such as Fisher Price. Microwaves, refrigerators, air conditioners. You get the idea. They’ll all be smart. What means smart to one device will be different to another. But, when it comes down to it, they’ll all be equipped with data gathering chips that’ll hold the data itself. This data will later be analyzed to report such information as customer usage and product performance.

You can look at Big Data as the funnel that is the passage way of this deluge of data from so many sources. The world is already operating on this data to some degree. Traffic lights, train signals, and so on. Get ready for more…

IoT needs big data. Big data doesn’t need IoT as much, as there is an overwhelming amount of data coming from already massive number of sources, as needed. There’ll be more of a need for new advanced infrastructures, operating systems, and a host of new software and hardware applications. More so, it’ll be the companies who’ll need to either build big data warehouses themselves to deal with the continuous stream of data flowing in to analyze it all and provide relevant insights, or they will need to outsource part or the entire process.

The Big Data RUM/APM Challenge

Well, we have no real slowdown coming for Big Data. Real User Monitoring (RUM) and Application Performance Management (APM) have its own share of data to analyze. The massive amounts of data coming from every point of an APM tool, from software infrastructure to the virtualization layer to underlying hardware, the mass amounts of data that needs to be analyzed (and rather quickly I might add) is putting an added strain on the practice of big data and IT Operations Management.

APM tools, especially a lot of the traditional ones, may not have seen this coming and are struggling to catch up. So, if big data doesn’t seem to have enough challenges, APM wearily steps up to provide yet another. Environment management has become a Big Data problem, so what’s needed is some Data Analytics.

APM tools today, unlike first generation tools are constantly encroaching upon the Big Data threshold. Environments typically generate several terabytes of operations data each day. Storage for such vast quantities of data has become quite a big IT expense and that is just one facet of the Big Data puzzle.

Putting aside the sheer amount of data that the APM tools of today need to process, the questions analytics must answer is:

If there is a problem, what is it?

How do we fix it?

The problem is two-fold. Identifying that there is a problem happens in the NOW. APM tools must be able to monitor real-time data and identify anomalies, that’ll tell you if there is something amiss.

However, in order to get to the root cause, APM tools need not only be able to analyze data in real-time, they need to look at historical data as well to use as a benchmark of sorts. This means APM tools can’t cut it, if there is only one or the other. And it must do this within seconds after identifying an anomaly.

Big Data Application Performance Management

Because of this, it is time for Big Data APM. One of the primary goals of Big Data analysis is the ability to track a timeline-like series of anomalies to pinpoint the root cause. The only time IT ops has been successful in honing in and identifying the root cause is when it tracked anomaly events.

Wrapping it up, in the digital businesses of today to be truly successful you need to be driven by two things. Software and your customer base. There is no way of getting around it. Because of this fact, digital businesses of all stripes and colors must be able to connect the dots between the experience they are providing and the underlying technology that is running it all. Timely detection of real time anomalies along with root cause analysis are paramount to making this connection.

4 Recommendations for Big Data and APM that can drive success:

1) The amount or quantity of data to be monitored will only increase. Avoid investing in a tool that isn’t scalable. Develop a plan for the retirement of any tools that can’t hack it.

2) Keep in mind that problems can occur at any point in your infrastructure and off in the cloud, APIs gone awry. You need to have your finger on the pulse of any environment you use to run your business. Don’t think just monitoring your own will keep you safe.

3) Machine learning is key to building and improving on anomaly detection models.

4) Don’t sell scalability short. Big data APM have challenges that goes beyond just fixing the problems of here and now. As IoT brings in more data and connects more devices, detecting anomalies at scale will only become more important.

Whether it’s IoT related or Big Data APM, you will need some type of Big Data Analytics tool(s) if you don’t already have something in place.