“Understand the Secret Sauce” and Finer Points About Business Transactions and Evolving APM

On this planet, water is essential. Air is essential. And I could argue that for APM, Thomas Tammann’s “10 Tips for Every New AppDynamics Customer” is essential. While his “Top 10” doesn’t carry the same life-dependent intensity as sun to earth, it is core to success for modern APM.

I’ll set the scene: London, AppD Summit Europe, May 2017. The stage is dark. The audience is in high-spirits. Thomas bounds on stage. His tall, lanky form leaning into the experience. Genuine confidence into the insights he’s about to share are based on years of customer partnership as an AppDynamics Solutions Architect. His knowledge is lifted from the context of our customers, their customers, and the backdrop of evolving digital transformation trends. We’re set for a worthy session that any application team can learn from.

Thomas shows us that simple ideas are tough ideas. Some of the key points he covers may seem obvious, but they’re not initially observed by a majority of customers. Only after challenges do customers see how essential it is to invest in each of the tips described.

Here Are My “Top 4”, Paraphrased from Thomas’s “10 Tips”:

 #1. Set an APM strategy.

If APM is just another tool, IT is doomed to service the business. Our most successful customers have adopted a business-first IT mindset. They view APM as a strategy, not a tool.  This is because application performance drives business performance. A tool adjusts performance, but a strategy ensures you’re using performance to gain advantage. Thomas advocates that it’s critical to invest time in setting a strategy and most importantly, to align your team to it. Teams are made up of people, and when people have different understandings about the direction the team is headed, you’re all bound to get lost. To illustrate this point, Thomas had the audience stand up, close their eyes, and point to “true north”. When he asked us to open our eyes, we witnessed the reality that we were all pointing in different directions — our individual ideas of “true north” were entirely scattered. So, strategy is powerful and communicating that strategy has a unifying effect, increasing chances for success.

#2. Go with the flow.

We give visibility to your Business Transactions — every detail, every dependency, complex or not. Don’t sacrifice knowledge for a pretty picture. All too often customers superficially edit the reality of their application flow-map because they believe it’s easier to understand. “Not true,” suggests Thomas. By “understanding the secret sauce” (that is, how and why AppDynamics does what it does), you can embrace the complexity of the true-flow and benefit in the long term.

#3. Automate.

Thomas uses unconventional wisdom (AKA, “When lazy is good”) to show us the beauty within the AppDynamics platform. Simply put, as soon you do something twice, you should automate it.  AppD helps you do this. For example, dynamic performance baselines are easily created to automate alerts that show you when performance deviates from norm. This helps prevent IT alert storms. AppD also enables customization of automation. For example, you can automate agent deployment, monitoring for on-prem controllers, configuration deployment (the new scope configuration brings you additional power), and integration with event systems.

#4. Configure signals from the noise.

Business Transactions are the critical first step. But know that data ingestion is just the beginning and that when you gather indiscriminately, you end up not knowing and not using what you have. Data isn’t knowledge — it’s how you process it that makes it useful. As Thomas suggests, often times, “getting everything and having nothing” is the result of a data deluge. Collecting data which serves as foundation for information and then knowledge is the goal.

Consider learning more about best practices for AppDynamics customers by reading our eBook, scheduled for delivery during the summer of 2017.

General Note: The title, “Ten Tips for Every New AppDynamics Customer” comes from Thomas’s experience working with customers that had around two years use of the tool. The “10 Tips” are not necessary for new customers, but also immature customers (in using the platform of course; they are ripe with wisdom in every other sense). That’s why Thomas originally called his talk the “10 Commandments”, as every customer, new or old, has to consider them. I would simply think of them as best practices.


Making Dev and Ops First Class Citizens At Standard Bank

Now that AppSphere 2016 is over, I have been reflecting on the track sessions that I managed and
the ones that I watched at the event. While all of them were very good, there are a couple which really stand out for me. One of these was ‘Standard Bank’s: Agile, DevOps & the Role of first-classAppDynamics’ case study. It was delivered by Andrew Kretschmer, Executive Head, Digital Channel Development and Lenro Meyer, Application Development Manager. What really stood out for me was their story (great slide quality also), honesty, and the benefits derived from AppDynamics’ software. So let me say, thank you guys! Here is a quick breakdown of their session in case you missed the presentation.

A Bit of Background into Standard Bank

Standard Bank is one of South Africa’s largest financial services groups, with 14.9 million retail banking customers, 1,221 branches and 8,000 ATMs. Established 154 years ago, it’s no surprise that today’s IT environment is a mix of new and legacy systems. Andrew explained that their strategy is driven through digitalisation of business processes and a mobile, self-service-first mentality. Ultimately, their enterprise vision is to be the first leading financial services organization in, for, and across Africa, delivering exceptional client experiences and superior value. From the stats highlighted in the slides this vision is certainly being realized with the  number of transactions rising significantly in regards to online banking and their SBG mobile banking app (in fact transactions on these platforms are up 387% according to the last company report!).

What Digital Challenges Did They Face?

In short, a number of digital challenges, but in summary:

  • IT spend was ~30% of operating costs, when best in class is 15%. This spend made it difficult to allocate budget towards digital innovation. Standard Bank’s unit cost per a feature on an application or service was 3X higher than that of their peers.
  • Digital acceleration changing customer expectations. Mobile and web were changing the way customers wanted to bank, but unfortunately it took 300+ days to deliver features into production with many outages. This resulted in low customer satisfaction.
  • Developers treated as second-class citizens. This was a pertinent point for me. In February 2015, Lenro explained that their mobile app suffered a severe outage and while his development team were pulled into a crisis room to find the root cause, they were treated as second-class citizens. It actually took them two hours to convince stakeholders that they needed access to the production environment to find the root cause. This coupled with a lack of adequate monitoring capability, led to the outage lasting a day and as Lenro explained to their CIO, “If we continue to treat developers as second-class citizens then this will happen again.”

Was Agile Software Development the Silver Bullet?

In order to solve these digital challenges, the first strategic initiative was to move towards Agile software development practices. But as Andrew explained, while they did try to adopt processes such as having daily standups, etc. this initiative ultimately failed. The reason was simple: You can’t just implement a new process or operating model without changing the culture.

Therefore the solution was to transform to a DevOps operating model, breaking down silos between teams and ramping the number of developers from 12 to 250 over the course of 3 ½ years.

AppDynamics Chosen as a Strategic APM Solution, Key to Driving DevOps Adoption

On the back of the mobile app outage and as part of their DevOps adoption, Lenro began to evaluate their monitoring strategy back in 2015. Initially this was hampered, because as a developer (a second-class citizen) he only was given limited access to existing monitoring tools but by doing research online he identified that a modern, enterprise APM solution was required.

His seven key requirements can be seen below:

After trialing a couple of solutions Lenro and the team at Standard Bank concluded that AppDynamics APM was the only solution on the market to address their key requirements. AppDynamics offered an enterprise, intelligent APM solution showing end-to-end visibility of performance in real-time, automatically and easily.

Three Major AppDynamics APM Benefits

Standard Bank have only been an AppDynamics customer for a year but Lenro outlined three major benefits from their adoption:

  1. Remove quality impediments in software rapidly. Lenro explained that AppDynamics has been used on Standard Bank’s internet banking platform in Nigeria to remove quality impediments. The situation was that logins were taking 30 seconds and because the internet banking platform follows a layered or tiered architecture approach, it made it extremely difficult to identify the root cause. Within minutes of deploying AppDynamics it gave developers immediate visibility into the layer causing the issue (in this case the core banking platform) and the exact line of code that was the root cause. This means that logins now take less than three seconds. Lenro explained that this rapid visibility means that developers can now focus on building new features rather than hunting and scouring logs to find the culprit of performance problems.
  1. Prevent outages and reduce the time to fix production issues. In this use case, Lenro explained that they had made changes to the core banking implementation. As soon as the different channels that rely on core banking were started up, AppDynamics started triggering alerts in regards to slow transactions. Using AppDynamics, Lenro and the team were able to immediately identify that it was one node in a four node cluster that was experiencing a 25% call failure. The impact was that potentially 25% of customers using this channel could not check their account balance. But because AppDynamics gave real-time visibility into the cluster, it gave the support team the awareness to remove the node and repair it at a quieter time. This meant that banking services did not experience any issues and helped the support team to maintain their Mean Time Between Failures (MTBF) metrics. Additionally, Lenro talked about an Internet Banking outage in April 2016 when it took only five minutes to identify and the resolve the issue. Before AppDynamics this sort of outage would have resulted in teams being pulled into a war room which would have lasted for two to three hours a minimum before finding the root cause.
  1. Enable feature teams to do DevOps. One of the main drivers for DevOps adoption was to ensure that teams spent less time on support and more time on feature development and optimization. This is now starting to be achieved with Dev, Test, and Ops professionals all using AppDynamics to give them visibility into how code is performing and to identify areas that need improvement before the business and the customer is impacted. As AppDynamics is used across the software development lifecycle it means that better quality code now goes into production. Today Dev, Test, and Ops are all first-class citizens.

Standard Bank’s Tips for AppDynamics Adoption

Lenro finished the presentation with a number of key tips for successful enterprise adoption:

  • Automate as much as you can. At Standard Bank they use Chef to automate the deployment of AppDynamics agents and controllers. This means it’s quick and easy to deploy AppDynamics into new channels or parts of the business.
  • Put AppDynamics in the hands of the right people. Don’t centralize the deployment of agents but let dev and ops teams deploy them themselves. This speeds up adoption as dev and ops teams know their application best.
  • Prepare for the storm of adoption. As soon as teams see the value that AppDynamics can provide – they will want it! So make sure that you manage demand and can structure your rollout based on business importance.

In Conclusion

Standard Bank are continuing to rollout AppDynamics and to ensure adoption in the right way but one thing is certain: AppDynamics has helped Dev, Test and Operations teams to become first-class citizens within the enterprise.

Again, thank you Andrew and Lenro. This was a really good session, highlighting that enterprise adoption of AppDynamics has major benefits for every element of the software delivery lifecycle. I hope you have found this summary useful but please feel free to comment on your experiences below or to try AppDynamics in your enterprise.

Watch the AppSphere 2016 presentation

Okta is Driving Fast, but with Eyes Wide Open

The new “speed” conversation

When you’ve been writing marketing about tech companies since the dawn of the millennium, as I have, you hear the topic of speed come up a lot. And rightfully so. It is, of course, a vital metric and, in a marketing sense, a crucial differentiator when selling your product to the masses.

5X faster than the competition.

Fastest throughput.


It’s been interesting to watch those performance metrics explode. But a funny thing has happened to the speed discussion. More and more, it’s not really in terms of the measurable, like speeds and feeds. Now, it’s more conceptual. It’s about how fast you’re innovating or responding to changes in the industry landscape and with the demands of customers. This is the direct result of technology no longer servicing the business, but driving the business.

Yeah, your stuff is really fast. But everyone’s stuff is really fast, so it’s not as much of a focus of differentiation anymore. How many of us even seriously consider the performance specs of a laptop before buying it…with one click…on Amazon Prime…hoping you ordered it fast enough so it comes tomorrow and not — gasp — the day after tomorrow.

“Oh! I can get it same day? Meh. How many hours is that going to take?”

It’s not about being fast, because fast is a given. It’s being fast all the time, and being faster to innovate, and being fast enough to stay just ahead of the competition. Because, let’s face it, pretty much no one gets out way ahead and stays there.

Yeah, tech marketing is way different these days. And so much more exciting.

On this trip together

One company that’s very exciting these days is Okta, an identity management company and an AppDynamics customer. They use AppDynamics application performance management to help them follow through on what Hector Aguilar, Okta’s CTO and SVP of Engineering, describes as their “always on” culture. “Always On” even appears in the tab on your web browser when you go to okta.com.

We’ve recently posted a video that was featured at AppSphere 2016, as well as written a success story about our great relationship with Okta.

In the video, Mr. Aguilar payed us one of the most flattering and clever compliments we could have hoped for:

“If we don’t have AppDynamics, where we’re using it, it would be really like driving a car at 100 miles per hour with your eyes closed.”

And there it is again — speed. But not of their product (which is, of course, always on), but the speed in which Okta is moving within the industry. The speed at which they have to operate to meet the expectations of their customers — and their customers’ customers, employees and partners, who are all touch points of Okta’s solutions.

We’re incredibly proud to be helping Okta and other industry leaders concentrate on this incredibly fast-moving, twisty road that we’re all on together.

Please be sure to read the story.

Using DevOps to Deliver the Best User Experience

In a session at AppSphere 2016, Blackbaud presented “How Blackbaud Transformed From On-Prem to SaaS Through DevOps and AppDynamics”, where Blackbaud shared the challenges and benefits of moving from an on-premise model to an SaaS-delivered model. They also discussed how DevOps practices, along with the support of AppDynamics, allowed them to establish trust between teams and achieve their goal of servicing customers in a subscription model. If you’re not familiar with Blackbaud, they are the engine that powers the online presence for a large number of nonprofit organizations, supplying them with the tools to help them manage and drive their fundraising objectives. They are the largest cloud software company powering social good.

In Blackbaud’s journey to change their business model to SaaS, they encountered certain obstacles. These included communication challenges between the dev and ops groups that threatened to slow their ability to deliver positive experiences to their customers. As the move to SaaS continued, dev and ops had to work together more closely to address the realities of delivering the company’s offering on a distributed platform with multiple users. By partnering with AppDynamics, these groups were able to visualize that environment and start closing the divide between dev and ops to create a DevOps culture focused on improving their application.

Using AppDynamics’ tools as a basis for application performance monitoring, Blackbaud began to chip away at issues that had arisen from a previously complex environment. They were also able to easily discover the bottlenecks that could become bigger issues if the system becomes compromised due to load or other factors. As the relationship between the dev and ops teams further matured, the developers could quickly spot and evaluate potential areas that might cause bottlenecks, addressing them in their next sprints to continuously release more performant code. This direct collaboration between teams allowed Blackbaud to take a holistic approach to their application, with end-to-end monitoring and the ability to identify changes both pre and post release, comparing not only business transaction performance, but also database call performance and volume.

SaaS environments have multiple variables, and each has the potential to adversely impact the customer’s experience. With a DevOps approach, Blackbaud is able to proactively manage the experience of users and react quickly to mitigate any issues that arise. As Blackbaud’s client base grows and the complexity of supporting thousands of users increases, their internal teams are able to tackle challenges head-on due to their investment in DevOps, and Blackbaud’s partnership with AppDynamics helps to deliver the end-to-end monitoring needed to sustain it.

Want to learn more about DevOps? Find out Why DevOps is an Essential Part of Digital Transformation

United Airlines and Real-time Business Monitoring

Flying out in front

One of the world’s largest airlines, United Airlines, has an impressive history of using one of humankind’s greatest innovations — the airplane — to serve people. On April 6, 1926, pilot Leon “Lee” Cuddeback flew a Laird Swallow biplane flew from Boise, Idaho, to the railroad mail hub at Pasco, Washington, returning the following day with 200 pounds of mail. This historic round trip journey was the first contract airmail flight, making United Airlines (then called Varney Air Lines) the first ever commercial airline in the U.S. to fly fixed routes and schedules. Another customer-centric first was established in 1930, when United hired nurse and pilot Ellen Church to assist passengers and help calm them of their fear of flying, making her the first female flight attendant on a passenger airplane.

Millions of flights and billions of passengers later (140 million in 2015 alone), United is more dedicated than ever to using innovation to keep the skies, terminals, and maintenance hangars friendly for their passengers and employees.

“One of the goals given to us by leadership was, let’s win back the trust of employees and our customers,” says Bill Hineline, United’s Director of Application Performance Management.

No small task, seeing how Mr. Hineline’s team is responsible for managing over 125 applications, from united.com and ticketing to weight balance systems and baggage tracking.

United has enjoyed what has become a truly special partnership with AppDynamics, including Mr. Hineline taking the stage at the AppSphere 2016 keynote and leading a session where he revealed the depth of the successful partnership, which now includes Business iQ, our real-time business monitoring solution.

“We now understand business impact,” says Mr. Hineline. “Who is using it? How are they using it? Where are their problems being experienced?”

We couldn’t be prouder of the relationship and how we’ve been able to assist United in their digital transformation, allowing them to continue on the path of constantly innovating to create happy travelers and team members — and drive better business outcomes.

Bill Hineline will be joining us again to talk about how AppDynamics and United are working together to better serve customers, employees, and the business during an upcoming webinar that you won’t want to miss. Register now.

Want to hear more about the United-AppDynamics partnership? Watch the inspiring story here.


From Code to Customers, how Barclays uses AppDynamics

In the age of digital transformation, customer experience is the ultimate battleground. Every sector is striving to cater to their customers in effective and convenient ways. Therefore, the need to understand these customer journeys has never been more vital than it is right now — your company needs to provide the digital services yours users will get the most value out of.

To kick off our AppSphere Series, recaps and key lessons from the top presentations at AppDynamics AppSphere 2015, Peter Gott of Barclays talks about how he’s able to gain customer journey insights to better accommodate their users of Barclaycard.

“The right team with the right tools can make anything happen.”

Their challenges included:

  • Application issues impacting customers
  • Contact center efficiencies
  • Dev and Ops collaboration productivity
  • Developer productivity
  • Silo-ed software tools – expensive and inefficient

Their goals included:

  • Enable new paths to revenue with technology
  • Reduce customer impact due to technology incidents to zero by 2018

For Peter and his team, legacy IT systems were a burden to their innovation. Instead, they’re investing on next-generation systems that are able to power their digital objectives at scale. Peter goes on to talk about the benefits they see from AppDynamics including helping enable their move to DevOps, visibility into extremely complex systems, increased availability and productivity, and create a self-healing application.

Every vertical, not just financial services, is undergoing a radical digital transformation. Hear from Peter and his experience at Barclaycard in the video below.


Here are the slides from his presentation as well.

AppSphere 15 – From Code to Customers: The Digital User Journey from AppDynamics

We hope to see you at AppSphere 2016!

Jostle Workplaces Work Better Thanks to AppDynamics Application Intelligence

This blog post is based on the Jostle case study

I recently had the chance to catch up with Teg Ryan, director of technical operations at Jostle — an award-winning Vancouver-based software design firm. Jostle’s primary objective is to build “a radically better intranet” for every customer, where workplaces are incredibly vibrant and productive.

The cloud-based Software-as-a-Service (SaaS) business’ customers expect their intranets to deliver responsiveness on a par with on-premise software.  “Everything the company does has to be fast,” said Ryan. “All of our users expect to get their documents or have a discussion in our application without any kind of glitch or delay,” he added. “So, we use AppDynamics Application Intelligence to monitor every aspect of response time levels. If there’s slowdown, we can easily drill down and find exactly where in the stack is causing a problem. Our mean time to recovery has been cut in half given the visibility that we have now.”

Key Benefits

  • Provides the intelligence foundation for fast application performance

  • Instruments the complete software stack, replacing multiple tools

  • Provides visibility into software response time issues like never before

Why AppDynamics?

  • Minimal configuration requirements

  • Dashboard display provides window into entire environment

  • Provides comprehensive intelligence with negligible overhead

Jostle is heavily invested in Amazon Web Services as well as other cloud providers to serve their Java based web application. Today, the Jostle team is monitoring around 20 production servers and using AppDynamics Application Performance Monitoring to track both database response time and front-end web response times. According to Ryan, his team is constantly driving down both of those numbers because of how quickly they can see issues and fix them.

AppDynamics has helped Jostle go from highly limited visibility to a hugely well-instrumented environment. “Prior to implementing the AppDynamics Application Intelligence Platform, visibility was a big issue,” said Ryan. “We were relying on a hodgepodge of different monitoring tools, and when you do that, there are black holes that you just can’t see into—black holes that cause big trouble if there’s an issue in one of them,” he added.  Now those black holes are gone.

Slow databases are a thing of the past, too. “Prior to deploying the AppDynamics Platform, we had a bug in our code that was causing a query to fire millions of times over and over again,” said Ryan. AppDynamics Database Monitoring uncovered the bug and the Jostle team fixed it in a matter of minutes.

Application development timelines have also improved since the company began using the AppDynamics solution. In fact, Ryan mentioned that, because Jostle’s developers have access to the AppDynamics dashboard, they resolve most issues much earlier in the process than they used to, long before they ever reach production.  This has helped reduce the workload on the operations team and promote their culture of DevOps.

Near-zero performance overhead is also a big plus. “I’ve always been impressed by the low impact the AppDynamics software has on ours,” said Ryan. “How it instruments everything without any serious configuration is mind-blowing. I haven’t quite figured out how you do it yet.”

London School of Economics Gains Improved User Experience with AppDynamics

This blog post is based on the London School of Economics and Political Science press release published on September 15. 

This week, I had the chance to learn about an exciting new partnership we kicked off with the London School of Economics and Political Science (LSE).

LSE is a world-renowned institution that serves approximately 9,600 full-time students and more than 125,000 alumni. As they open their doors to more overseas students from over 140 countries, the demand for digital student resources to enable remote access before term starts has skyrocketed. Today’s software-defined business crosses into higher education more than ever as everything from connected devices and digital academia become a best practice as opposed to a supplement within academic institutions.

At the start and end of the academic year, LSE experiences a staggering rate of visits and usage of its digital properties from both students and faculty. Academic coursework at LSE functions on a first-come, first-served basis, and when IT operations are not scaled to respond to peak visitation rates, students risk losing the chance to enroll in courses required to graduate.

At LSE, an internal application development team works alongside the systems and databases teams, looking after student and staff portals and enterprise applications across the greater organization. These core applications are used by students to choose courses, check their fees and grades, and pay for services. Staff utilize them to keep tabs on their students, and the Student Union uses them to run their elections. With so many different players in their digital resources, LSE needed to define a set process to integrate their back-end development teams with their end users–the students and staff who continue to grow with the school. 

To break down the silos between IT operations and academic functions within the university, LSE recognized the need to implement an application performance management solution to troubleshoot the causes of their performance issues. This would further integrate IT capabilities into the university’s academic framework.

LSE needed to scale and improve its online performance before their students and faculty lost valuable time and course work. They recognized and selected AppDynamics as the provider to quickly improve performance of their digital resources to maintain student interest and competitive enrollment in the higher education market.

AppDynamics supported LSE to streamline management of online services and enable a better overall quality of education for their growing student body. “We want to move towards a DevOps model where we can deploy rapidly and more cost effectively, and respond quicker to end user demands,” says Ron Riley, enterprise technology manager at LSE.

AppDynamics enables end-to-end visibility for LSE across all digital operations to improve monitoring, quickly locate standout issues, and resolve potential system threats. AppDynamics’ deployment speed and quality customer support created the ideal relationship with LSE. “LSE’s complex architecture has many moving parts, which can make it challenging to identify and remediate bottlenecks that impact the performance of our services. AppDynamics has been paramount to improving this. It allows us to drill down into every line of code. Before it was more of a guessing game, but now we can spot issues quickly and resolve them before they affect user experience,” says Riley.

Today, LSE is already seeing success with the AppDynamics platform, and the university hopes to roll out operational and business dashboards across other areas to provide wider visibility and strengthen performance across the institution. LSE further intends to extend beyond Unified Monitoring to include comprehensive Application Analytics in the future, which enables organizations to capture and analyze every performance detail of every business transaction flowing through their applications. By integrating IT capabilities and academic functions, the London School of Economics and Political Science is not only scaled to serve a growing demand base, but they also create the potential for a more innovative higher education model in today’s software-defined economy.

Q&A: How The Container Store uses AppDynamics for Continuous Testing

Last week, I had the pleasure to host and co-present with August Azzarello from The Container Store in a webinar titled “How The Container Store uses AppDynamics for Continuous Testing”.

In this webinar, August explained how they have taken application performance management (APM) one step further by embracing APM in their development lifecycle, enabling their team to do continuous testing to catch and resolve issues before customers are impacted.

AppDynamics was being used by the eCommerce team at The Container Store to monitor the performance of their eCommerce applications. After learning the capabilities of AppDynamics in a meeting with the eCommerce team, August started to evaluate AppDynamics for addressing some of the challenges he was having in his testing environment:

  • Set performance expectations to business stakeholders prior to production deployment

  • Expand automated performance testing with efficient reporting and monitoring

  • Reporting on functional tests

  • Test new calls to third party remote services prior to going live

August explained how he addressed these challenges by deploying AppDynamics in his test environment to do continuous testing. He discussed some of the APM functionalities he is using in his testing environment, benefits to his organization and shared some of the best practices based on his learnings. After August, I discussed the process of BizDevOps that utilizes DevOps practices to further drive the overall business agenda and shared 5 keys to success with (Biz)DevOps.

How The Container Store uses AppDynamics in their development lifecycle from AppDynamics

We had a very interactive sessions after the prepared content was presented. August and I answered many questions asked during this Q&A, but, many of these questions remained unanswered. You could listen to the questions and answered that were covered during the webinar at the on-demand recording of the webinar.

August and I responded to all the unanswered questions and here are the top questions and answers from the webinar.

In the test environment, what diagnostics or root-cause analysis tools did you use before AppDynamics and how do they compare?

August: At The Container Store, prior to AppDynamics we just had our plain old logs and Splunk to help parse them. We were limited to what we can get out of the JVM/JMX layer for most metrics, and we usually had to depend on operations to get them. So AppDynamics really empowered us in the test environment.

In past, I have used Zabbix in a test environment. Mainly for low costs, but it was a nightmare to configure and maintain — the amount of time/money we spent trying to get it to be sufficient enough could have easily bought us AppDynamics (in hindsight). And, Zabbix never got us near the detail of information that AppDynamics does. No comparison, I’d fight that battle any day to get AppD in a test environment — the ROI is huge.

Suppose in one Business Transaction (BT) there are multiple requests…Does AppDynamics identify this as single BT or multiple BTs?

August: It depends on how they are configured, and where these requests are. If you could be a little more specific, I could help more precisely. For an example, if a servlet call to /shop makes many requests behind the scene. You see requests to the DB, CDN, and other service tiers. These all show on the one BT for /shop. At the service tiers, you would have separate BTs setup if desired – making those BTs specific to only what occurred at the service layer.

Do you guys use any automation tool for testing at the Container Store?

August: We utilize Selenium w/ Ruby bindings for web functional testing. For performance testing of all web and service layers, we utilize Locust.IO (Python framework). We also utilize a .NET based tool called Ranorex for our store systems point of sale functional automation. Links to the tools we use below.




Do you use Docker containers? If so how do you find the AppDynamics Docker Monitoring Extension integrates with your testing environment? If not, can the Docker extension identify performance bottlenecks in the contained code? 

August: We have not experimented with this as of yet – but recently started having these conversations, thus why I wanted to respond to this question. I will share when I know more details about how this works — and please let me know if you gain any ground on this subject – sharing our experiences could help us both! 🙂

Anand: The AppDynamics Docker Monitoring Extension monitors and reports on various metrics, such as: total number of containers, running containers, images, CPU usage, memory usage, network traffic, etc. The AppDynamics Docker monitoring extension gathers metrics from the Docker Remote API, either using Unix Socket or TCP giving you the choice for data collection protocol.

The Docker metrics can be correlated with the metrics from the applications running in the container. For example, the overall performance (calls per minute) of a web server deployed in Docker container can be correlated with Docker performance metrics (Network transmit/receive and CPU usage). You can also set-up health rules with these metrics as well.

It can be used for performance and functional testing in test environment as well like some of the metrics

Is there a way to physically limit/cap the amount of overhead used?

August: We’d need to confirm with a technical rep from AppDynamics, but I do not believe so. I think the only way to control the overhead is by selecting either Production or Development mode. If it helps, I’ve tested out performance with and without AppDynamics agents installed.

AppDynamics says that you can have up to ~2% overhead in production — I’ve found it to be more like 1%. I find it hard to detect any performance hit when in production mode. In development mode, there is more overhead – but we decided it was worth it for having EVERY transaction snapshot, all the data for a test environment. Since we baselines with the system in development mode, we include this in our testing knowing production will have less overhead.

Have you used information points for test? Or is the full BT more useful?

August: Currently, we’ve found full BTs to be more useful. We actually will attach a BT Snapshot export to bugs in our bug tracking system. We don’t really incorporate information points into our test strategy, but now you having me thinking we could explore that. I’ve found that the BTs were more efficient, because we had them set-up anyway – and are so important to the overall AppDynamics strategy being wrapped around business transactions. We do use information points for watching things like conversion, or email tracking — I’ll take another look at information points from a test standpoint and see if we’d add anything — thanks for bringing it up, great question. If you find a useful way to use information points in test – please share!

What would you wish APM could do that it currently doesn’t?

August: From a test only standpoint, I wish we had a better ability to archive reports/information in the most granular resolution of one minute. For snapshots you can manually archive them – otherwise they are only retained 14 days. We end up saving off reports, etc.. on disk for archive processes. I have not tried real hard, but there is no obvious way to do this all in AppDynamics to my knowledge. Graphs go to 10 minute resolution after 4 hours, and 1 hour after 3 days. You still have your information, it just is not as granular as sometimes needed.

How can we trace (compare) the performance latency before and after changes in the build? By profiling the transaction logs?

August: You can verify performance between builds very easily in a number of ways. I’ve found that response times per call are very indicative of performance changes. This can easily be verified on any of the flow maps, dashboards, metric browser or business transaction pages… and done at either the application as a whole, any specific tier – or any individual business transaction or call. We’ve never had to profile any transaction logs to detect performance changes between builds — it has all been done in the GUI.

What is the transaction max limit per tier for AppDynamics? How can we purge the transaction if it exceeds the threshold instead of manually tracking it?

August: For performance reasons, there is a default limit of 50 business transaction per APP AGENT, and 200 business transactions per Application. There is no limit at the tier level. We use catch_all transactions to catch anything that is not defined as a BT, and limit things from going into All Other Traffic. With Business Transaction Lockdown, you can enforce these rules – and not fill up your 200 BTs per application. I do weekly reviews of the catch alls and all other traffic buckets to determine if there is anything new that needs to be promoted to a real BT. I’ve included some links to some documentation below that may help a lot.



Does it have the ability to download to all the transaction snapshots into Excel or similar applications?

August: In the UI, you have the ability to export snapshots in a PDF format. Via the AppDynamics REST API, you can download metrics in XML or JSON format – giving you more flexibility. I’ve found the REST API very easy to use.

Here is a link to REST API information: https://docs.appdynamics.com/display/PRO40/Use+the+AppDynamics+REST+API

Did you evaluate any other tools before selecting AppDynamics?

August: When I started at The Container Store almost 2 yrs ago, AppDynamics was already in place – just not utilized in the facet we do now in a test environment. So I did not evaluate other tools here. At a previous position/company – we did quite a bit of work in Zabbix, mainly because it was a cheap solution. But, I feel the amount of time/effort to configure and maintain drastically outweighs the cost of AppDynamics — and you do not get the same level of information with Zabbix. Zabbix and AppDynamics are not a fair product comparison in my opinion.

Can AppDynamics capture SOAP web message envelopes and store them in snapshots?

Anand: Yes, you could do this by configuring data collectors. Data collectors enable you to supplement your performance monitoring or transaction analytics data with application data.

You can learn more about configuring data collectors at https://docs.appdynamics.com/display/PRO41/Collecting+Application+Data#CollectingApplicationData-ConfiguringaDataCollector

However, you should use judgement about storing application payload data in snapshots. Larger payloads may have performance impacts.

DIRECTV tunes in to AppDynamics Application Intelligence

Read more about DIRECTV’s work with AppDynamics in the full case study here.

The more I’m learning about application intelligence, the more I’m realizing the competitive advantage that our customers gain with AppDynamics. DIRECTV is one of the world’s leading providers of digital broadcast entertainment, providing a premium experience in television through cutting edge technology and campaigns. And just like every leading company, it focuses heavily on IT and software applications to deliver high valued customer satisfaction to over 39 million customers across the U.S. and Latin America.

Challenge: Upgrading from existing APM with poor performance & adoption

Like many software-defined businesses, DIRECTV faced the challenge of retrieving the right value from its existing APM solution. With over 6,000 physical and virtual servers running a combination of Windows, Linux, and HP-UX, applications built by Java, .NET, PHP, and Node.js, DIRECTV wasn’t achieving the results it needed with it’s existing APM tool. First, it saw a poor adoption rate outside its core operations team. With the multitude of tools and applications its operations team was already using to troubleshoot, there was no window to determine a positive performance impact on applications. Instead, it implemented what Senior Director of IT Operations, Glenn Trattner refers to as “after-the-fact investigation”. When Trattner and his team began to outline what they wanted out of an upgraded solution, they identified the following:

  • Proactive optimization

  • Ease of use for higher adoption across development teams for improved collaboration

  • Ease of deployment with low performance overhead for monitored applications

Solution: Immediate impact and visibility in critical times

AppDynamics came to the table for DIRECTV at a critical time. During its renowned pro football Sunday Ticket offering, which offers subscribers access to watch their favorite teams through second screens (mobile applications, desktop, in-home set top boxes), it started to experience issues with high traffic and load as its demand increased. “Sunday Ticket places a significant load on our website and infrastructure in a very short period of time—especially at the start of the season…We were impressed with the visibility and insights provided by AppDynamics and decided to move forward with our purchase and deployment” says Trattner.

Why DIRECTV Chose AppDynamics

After seeing success with implementing AppDynamics, DIRECTV decided to move forward and deploy a SaaS version of its APM to monitor its Java, .NET, PHP, and Node.js applications. Today, they’re able to produce upgraded alerts and reports with AppDynamics, delivering a much more proactive impact as opposed to its previous after-the-fact reactive performance. They’re also fans of the dashboard and flow maps feature of the tool, which allows them to visualize traffic behavior and transaction flows throughout their processes.

While the features and services made more than a significant impact for Trattner and the DIRECTV team, they also appreciated the efforts by the AppDynamics team. “During our pro football Sunday Ticket rollout, we were doing performance testing on our production infrastructure in the middle of the night. AppDynamics’ experts were on the phone helping us, and we hadn’t even bought the product yet…Not only do they listen, they actually follow up and get back to you with answers. I can’t say the same thing about many other vendors we work with.”

AppDynamics brought the right value and insights that DIRECTV needed to maintain its premium digital broadcast experience, without compromising performance or time. Not only did they achieve the goals listed beforehand, they came fully equipped with the best teams to get the job done.