I am attempting to put together a complete list of End User Experience Monitoring tools since I could not find one anywhere on the web. I need your help in order to complete this list – there are surely going to be tools that I miss – please leave me comments with tools you think I should add to the list.
What Qualifies as an End User Experience Monitoring Tool?
In order to count as an End User Experience Monitoring tool it must be able to track the response times that real users experience when visiting the site – not a robot which is synthetically pinging the site. Specifically I am referring to tools that would enable IT operations to ensure that the real end users of an application or website are experiencing good performance. As I have alluded to in a previous post “speed solves a lot of problems” – claiming that even if your usability is not perfect – if it runs fast – people are less likely to notice.
Hmm. I suppose other free options like OWA, Piwik (with plugins) and Google PageSpeed module, as well as Analytics’ SiteSpeed http://www.google.com/support/analyticshelp/bin/answer.py?answer=1205784 … aren’t really considered “enterprise” or “complete” … but I’m finding it harder and harder to get at a solution that “just works” in terms of stats/analytics, user experience pings/errors, as well as path tracking/analysis, mouse tracking and survey integration. I’m beginning to think that complicated setups like this require more than one tool with a lot of customization/configuration. Even the default usage of Google Analytics doesn’t go far enough in letting me inspect individuals’ paths in segmented ways, just aggregate data as a whole, by default. So I’m thinking starting with an open source product like OWA+PageSpeed and extending it to suit my needs might be best, when combined with data mining efforts.
Louis,
I am sure you are not referring to “Outlook Web Access” when you say OWA – please educate me.
Piwik looks like an analytics tool – can you refer me to where the end user experience monitoring feature is?
PageSpeed is a great free tool for improving the performance of your web pages – but does it measure the real end user experience of visitors to your site?
SiteSpeed looks cool – it is only a sampling – and I am sure that it keeps all user data anonymous – so it would be great for a simple web site – but not so useful when it comes to heavy duty enterprise applications as you say.
I agree that it is difficult to find a single tool out there that can do both the performance monitoring – down to the single user level – for any kind of application as well as all of the analytics that you would want to see.
Thanks for the reply, Alon. If it didn’t, it’d be nice if the comment system emailed me your reply, perhaps take a look at Disqus or other such comment systems, they really can improve community-building on blogs and news sites 😉
OWA, indeed commonly means Outlook, so it’s kind of unfortunate naming clash, but in this case I’m referring to Open Web Analytics from http://www.openwebanalytics.com/ which has what it calls, “domstream recording”. This lets you see exactly what a user sees in their browser and where their mouse goes, though I’m not sure how embedded it is for performance timing or error catching. A similar feature is in piwik as a plugin, as I understand it.
PageSpeed is not just a Firefox/Chrome extension 😉 It’s a web service now too! Seriously though, what I was referring to was mod_pagespeed, the Apache module, which beyond all its other fun goodies has an “instrumentation” feature that — you guessed it: measures “the time the client spends loading and rendering the page, and report that measurement back to the server” http://code.google.com/speed/page-speed/docs/filter-instrumentation-add.html
Google helpfully mentions that their instrumentation feature is not needed if you already have one of your own and cites Boomerang, Episodes or Jiffy:
– http://yahoo.github.com/boomerang/doc/
– http://stevesouders.com/episodes/paper.php
– http://code.google.com/p/jiffy-web/
Sampling is indeed the worst part about Google Analytics … sigh.
Also, I would suggest that “real end user experience” tracking is something you’d need to be Google Chrome to get. After all, as Google notes about its own filter: “Note that the data reported by this filter is only approximate and does not include time the client spends resolving your domain, opening a connection, and waiting for the first few bytes of HTML.”
Of course, that’s where the highly technical Speed Tracer comes in 😉 http://code.google.com/webtoolkit/speedtracer/
Louis,
Do you mind if I use your comment as a blog post (-: it would be very helpful for our readers.
Thanks for the insight on OWA – I now do remember coming across it – the DOM stream recording feature is something I did not see and looks like a very helpful tool. To me this looks like more of an analytics tool that marketers and product designers would use in order to improve the usability of their product. I should probably clarify that I am referring to end user experience operations tools which allow for the scalable measurement of performance.
mod_pagespeed definitely looks like something I would put on the list as an open source alternative. I do remember hearing about it at Velocity. The other ones listed also look cool.
Thanks again!
ps – I am familiar with disqus – and I like it too – so I will look into installing it.
Sure, feel free to use whatever you like, just mention my name or link to my site (currently points at my G+ account).
To clarify, CC 3.0 BY – http://creativecommons.org/licenses/by/3.0/
Abbas – Thank you for all of the valuable additions to the list – quite a bit of work! I will make sure to update the list shortly – I have some questions:
Regarding NetQos – I already have Customer Experience Manager as Computer Associate’s end user monitoring tool – and when I go to the NetQos site and click on “CA NetQoS Super Agent End-to-end application response time monitoring” I get a page not found. Are you sure that NetQos still has an end user experience monitoring tool on the market?
Regarding Webmetrics – I am unable to identify which product does REAL end user experience monitoring – their “website monitoring” service does synthetics.
Regarding Keynote – do you know if their “Real Browser Monitoring” tool actually picks up on the end user experience of all users that arrive at a monitored site? Is that their end user experience monitoring tool?
For Netscout – are you referring to the “nGenius Performance Manager“?
New Relic recently launched their RUM (Real User Monitoring) with Real End User Metrics. We currently capture page load time for over 1.8 Billion pages every week, and capture 20 Billion unique web transactions everyday. The data is based on real end users – no synthetic data. The end user performance is presented in-line with core APM data for full visibility of the application’s experience. http://newrelic.com/rum
Thanks for the sales pitch Patrick (-:
I will add it to the list. BTW – I love that new infographic that you guys created.
Whoops – sorry for the salespitch version – It was a cut and paste.
Thanks – look forward to being on the list!
Ooh, New Relic, the Rails folks! Neat. I’m kind of confused how RUM relates to that of the pricing page … is there a free version, or is it a feature of one of the paid plans, or something else entirely?
We’re more than just Rails! We started in Rails but now support Java, .NET, PHP, Ruby & Python (Beta). RUM is included for free when you deploy our service in your web app. You have the option to auto-inject the async JS into your app’s header & footer. Its pretty sweet. (end sales pitch).
How are you compare your tools with Dell Foglight? Can you capture Ajax calls? Can you replay the end user session?
Hi – Good idea!
I think it might be good to break the list down somehow to highlight the different functionality available in each tool e.g robot/synthetic vs real user experience and also to highlight the enterprise tools that can map any sort of transaction across the enterprise.
There is a big difference between a point product that does robotic transactions for website performance management and a full end-to-end monitoring tool capable of pinpointing bottlenecks and problems inside complex applications and via thin client etc.
FireScope.com also have end user experience management as part of their BSM tool
Cheers
Mike
Mike,
Thanks for the input. I was thinking of doing a separate list for robotic transactions. It looks like this list is going to be rather long even though it is very focused.
Can you please send me the link to the FireScope End User Experience Management tool? I was not able to find it.
Regarding the “end-to-end” and transaction tracing tools – I think that may deserve s separate list as well (even though most of enterprise grade tools listed are part of a larger suite). Once I am done with listing everything out – I may add some commentary on each. I was thinking of even just simply segmenting them into software based vs. hardware appliance based tools.
i’m surprised knoa wasn’t on anyone’s list. I’m researching this space as well and would assume knoa (http://www.knoa.com) would be on it. (I’m not affiliated with them and have never used their software so don’t know how good they are, just been researching the different providers)
Dan – I am very familiar with Knoa – I even visited their beautiful offices once in New York’s Union Square. I will add them on – they probably have one of the most solid solutions for monitoring SAP applications.
Precise is another end-to-end monitoring tool that intorduced TPM for virtual appliance.
Chetana – Thank you – I guess TPM is the new name of the i3 product?
I haven’t heard from Precise in a while…I will add them on to the list though – you are right.
NOT to be intentionally confrontational, but how can you claim this is the “COMPLETE LIST” of anything?
IMHO, a here is a GLARING omission from the list:
Here is a mobile, ON-DEVICE end-user performance measuring tool. NOT a simulator (like many other companies who shall go nameless), it actually resides on an actual mobile device, NOT Smoke-and-mirrors, but real-time performance measuring, in addition to a variety of other features & benefits for a mobile USER: 3P Mobile; Privacy, Performance & Personalization are the 3 pillars of a mobile users experience!
My apologies, I read your initial post but DIDN’T comprehend!
You can add SolarWinds “Synthetic End User Monitor” to your list. We used the trial version and it seems easy enough to get started with for all your web based needs.
Thank you for your list. One of the challenges with “End User Experience” monitoring is the misuse of the term “end-user experience.” While doing research, I’ve been finding that most solutions monitor traffic (from end-users) once it hits the datacenter, which really is not an accurate reflection of the *true* experience the end user has while using business applications on their computing device (desktop/laptop/mobile). The disparity between the datacenter perspective and the *true* end user experience seems to be getting even broader with the increasing complexity of app and network architectures. Does anyone else see this as a growing issue?
Exactly.. I keep spending time researching an ‘end-user experience’ application/device only to see that it is either like a network sniffer with very unfriendly UI, or it has a nice UI but lacks the ability to drill down into user sessions.
I have deployed the Gomez RUM product (formerly Vantage Agentless or Adlex) at two sites. It does report Real User experience by sniffing the packets at the datacentre. It is difficult to explain this in a short blog such as this – but deep packet inspection can detect when the client has received all components by the TCP/IP acks that are sent back from the client. The only part of client latency not reflecting is page rendering time by the browser once all the packets have arrived – but this is tiny.
I loved the Adlex product. However – at that time it did not handle Ajax well (or at all) – but this was a few years ago. There are technical issues with handling AJAX – so I am not sure if Compuware solved it. Other solutions use Script Injection (Javascript) in the web page – in a similar fashion to Google Analytics.
Hi,
very interesting list.
Please, consider also ip-label (French competitor) offer (cloudobserver), a RUM with powerful OLAP reporting,
allowing people make deep analysis.
More informations at http://www.ip-label.co.uk/index.php/en/End-User-Monitoring-QoE-Solutions/Cloud-Observer
Atomic Labs Pion
Application Performance WebTuna
Site Confidence Real User Monitoring
Extrahop
OpTier Experience Manager
It’s a confusing list. I am curious which ones can rival Truesight (Coradiant) for being able to drill down to each user session, read cookies, see a list of every object and page downloaded, having that split into ssl, network, host, user wait etc.
Most of these just gives big picture information.. Sometimes we want to drill down to an actual users session.
Hi – nice list.
You should also include Performance GUARD (PG) from CapaSystems.
PG measures on a wider range than most other tools – network latency, response time measurements on IP level, i.e. NIC-2-NIC for client/server applications as well as through Internet Explorer for web applications.
Furthermore PG also measures the health of the end user workstation, i.e. CPU, Memory, Disk, network etc. – not only for the PC as a whole but for each individual process running on the workstation.
Even further, PG also measures the startup time and the login time from all workstations in the organisation. This will allow for detecting slow starting PC’s as well as detecting what causes the most hang time across all user…
And, of course, there is my company – AppDynamics, which has a very powerful EUM solution for .NET, Java and PHP: http://www.appdynamics.com/eum.php
Qualica technologies has a tool that uses onsite nodes to compare performance with real last mile nodes. http://www.qualica.com/performance
RUM and EUM is quite a confusing subject for many people. End User Monitoring includes a variety of techniques such as: (a) the use of synthetic transactions; (b) Datacentre appliance that sniff network packets; (c) Embedded Javascript at the Browser. I think you need to be a little clearer about the method used by each product and the Pros and Cons.
Most IT people consider EUM simply to be technique (a) shown above and are not properly aware of either (b) or (c). Most Vendors consider RUM to be techniques (b) or (c).
I too have started to research the subject and have outline the pros and cons of each method (see link below). I can’t guarantee that this information is completely accurate yet. The info is work in progress.
http://www.sde0.com/index.php/best-practice/end-user-monitoring
Hi! You’ve done a great job compiling the list.
I think, Shopping Cart Diagnostics http://www.shopping-cart-diagnostics.com would make a good addition.
This is the tool to monitor end-user experience for ecommerce websites – it detects any problems customers are facing throughout the shopping process – from login to the payment stage.
Similarly, it possesses the functionality to measure performance characteristics and a number of other essential parameters.
http://www.CloudClicks.co.uk offers synthetic transactions as a service. This removes some of the cons Doug correctly points out in sythetic monitoring.
Because it is software as a service there is no maintenance and they are cheap as this is their core business.
We at Pingdom have just opened up our new RUM service for beta testing. You can apply for beta access on our Real user monitoring info page at: https://www.pingdom.com/rum/
Hi Alon,
Did you ever create a list for robotic transactions? My two -cents is the performance monitoring industry as a whole would be more honest it would be easier to create categories for buys if the whole “end user experience” terminology was dropped. As any good dictionary will tell you “experience is subjective” so we’re not monitoring the experience…we’re at best providing a “user’s view”, point of view or a perspective. Ask yourself this, if you take a snapshot of a sunset with the world’s best camera and send it to to me are you sending me your experience of the sunset? No. Its your view of the sunset. That, in effect is the best any monitoring solution can provide, a view. The rest is marketing. It would be easier for buyers in this marketplace to parse out what technology they need if the marketing-speak was more specific.
Hi Alon,
This is a great list you have Created,I don’t have anything to add to list yet but really wanted to praise your work so that’s why leaving a comment you have really done a nice job it really helped me a lot in knowing how many tools are available for the purpose of RUM and i didn’t knew there were so many.
I would like to know which of these tools require “an agent” to be installed on end users desktop. The reason I ask is if I need to install a user monitor agent on a users desktop I will be defeating the purpose. Basically if I see a user issue it could be due to the badly designed agent causing havoc with users desktop
Another thing is how can Mobile App (Which connects to the company network but not via a browser) be monitored or does it really need monitoring.?
Hopefully the above “silly questions” could be answered
Additionally I believe Synthetic transactions are a necessary evil…
Reason being any user monitoring tool cannot really monitoring what happens say when I submit an order in Siebel and it goes to downstream system to check the balance (Loose example so please pardon). But running these transactions add to the overhead of the system being monitored.
There are perhaps only a couple of vendors that use agents. one is pretty much out-of-the-box and the other requires scripting. See my recent post in that Forrester covers this in a specific video.
Net net agents generally allow precise measurement, to the moment the browser is actually filled with returning data, by desktop. Merck has 70,000 users or so using this approach for sap.
most large IT shops download lots of stuff to each desktop. large meaning over 1000 users on an enterprise app like crm or erp (financials, manufacturing, hr, etc.) small slices for various compliance, performance, security and related reasons. I talked to one financial company that pushed 42 small and separate software components to each desktop. I’m not kidding folks. they are all pretty small, lightweight and generally don’t effect local performance. In the case of the “42” I’m still baffled but there you go.
the other benefit of an agent based approach is that you don’t need back end connections. you can implement, if no scripting is required, in 4 to 6 weeks. You don’t need to wait for a release cycle with the back end software since you don’t connect to it.
That said, uem is all about what data your vendor can collect and then what you can do with it. accurate desktop level performance is just the beginning. some collect user error data and/or workflow which enables additional use cases, often outside of IT.
This is a very recent webinar on user experience management – Forrester did this in the past month or so:
http://www.youtube.com/watch?v=TQwEoC9uuSo&feature=youtu.be
Listen to this webinar by Forrester Research, JP Garbani, Vice President which covers Service Level Agreements SLA. Understand the cause of the gap between the line of business users and the information technology IT teams. Learn about new technologies, such as agent based user experience management which can potentially help reduce this gap.
—————————————————–
This is a forrester webinar on user experience management – about six months old:
http://www.youtube.com/watch?v=cSC1duu_fVI&feature=youtu.be
Jean-Pierre Garbani presented this webinar on User Experience Management/Monitoring (UEM) tools and how they can profoundly and positively impact Change Management and User Acceptance Testing (UAT). J.P. speaks about basic User Experience Management/Monitoring and also touches upon capabilities in areas such as User Performance Monitoring/Management (user error, workflow) which enable use cases such as Change Management.