In the perfectly elastic, brand new, and unfortunately mythical enterprise that we mentioned earlier, all applications were developed using Java, .Net, PHP, Ruby, or another modern environment. It is certainly true that there are many Enterprise Application Performance and Real User Monitoring tools that work well for new or existing applications written on one of these stacks.
What about everything else?

Many business processes and the transactions that implement them touch many applications. These may have been developed in house, outside by a consultant, inherited from a long past era, or purchased from a commercial software company. Some processes are unique to one application. Some touch two or more applications. Some are part of a Service-oriented Architecture and use an enterprise service bus or other middleware component to touch everything.

How do you do end-to-end monitoring when there are so many ends? There is really only one way to do this and it is to see everything. Tracking transaction flows from the user desktop, across multiple servers, through an enterprise service bus, and ultimately to the final call to a database or mainframe, is not for the faint of heart. There are typically many hops in this kind of transaction and each of these must be stitched together to create a comprehensive view of what is happening. Of course, during the stitching or composition process it is necessary to maintain the user or business context of the overall transaction.
Looking at the problem this way, one thing is clear. It is not an in house development project.

Some other things are also quickly apparent.
• It is a Big Data problem. There are many data points to be captured, correlated, indexed, and analyzed
• It requires more than “sniffing” network traffic. The business value of the process can only be found in complete transactions
• Periodic pinging or synthetic transaction monitoring isn’t enough. This approach may indicate that an application is not available or slow, but it doesn’t tell why or help diagnose problems
• Samples and averages of transaction traffic provide useful information, but don’t capture the complete picture. Quite a bit of important information gets lost in an average.
• It is important to monitor continuously, not just when a problem has been identified, otherwise the initial cause and preconditions of the issue are lost
• There is an IT variation of the Hippocratic Oath involved: First do no harm. The monitoring process itself should not create performance problems

Users on disparate devices, connected to an application, linked to other applications, integrated with shared middleware, accessing one or more databases, on multiple networks – life is complex.

About Elad Katav:

Elad Katav has over 18 years of leadership experience in the IT industry. He specializes in software deployment, operations and customer service from the vendor, partner and customer sides. Elad also lectures on “Inventive Thinking in the Technology Environment” at many educational institutions. In his spare time, Elad enjoys playing the guitar, reading tech blogs and playing with his gadgets.