How Top Investment Banks Accelerate Transaction Time and Avoid Performance Bottlenecks

A complex series of interactions must take place for an investment bank to process a single trade. From the moment it’s placed by a buyer, an order is received by front-office traders and passed through to middle- and back-office systems that conduct risk management checks, matchmaking, clearing and settlement. Then the buyer receives the securities and the seller the corresponding cash. Once complete, the trade is sent to regularity reporting, which insures the transaction was processed under the right regulatory requirements. One AppDynamics customer, a major financial firm, utilizes thousands of microservices to complete this highly complex task countless times throughout the day.

To expedite this process, banks have implemented straight-through processing (STP), an initiative that allows electronically entered information to move between parties in the settlement process without manual intervention. But one of the banks’ biggest concerns with STP is the difficulty of following trades in real-time. When trades get stuck, manual intervention is needed, often impacting service level agreements (SLAs) and even trade reconciliation processes. One investment firm, for instance, told AppDynamics that approximately 20% of its trades needed manual input to complete what should have been a fully automated process—a bottleneck that added significant overhead and resource requirements. And with trade volumes increasing 25% year over year, the company needed a fresh approach to help manage its rapid growth.

AppDynamics’ Business Transactions (BT) enabled the firm to track and follow trades in real time, end-to-end through its systems. The BT traces through of all the necessary systems and microservices—applications, databases, third-party APIs, web services, and so on—needed to process and respond to a request. In investment banking, a BT may include everything from placing an order, completing risk checks or calculations, booking and confirming different types of trades, and even post-trade actions such as clearing, settlement and regularity reporting.

The AppDynamics Business Journey takes this one step further by following a transaction across multiple BTs; for example, following an individual trade from order through capture and then to downstream reporting. The Business Journey provides true end-to-end, time-enabling tracking against SLAs, and traces the transaction across each step to monitor performance and ensure completion.

Once created, the Business Journey allows you to visualise key metrics with out-of-the-box dashboards.

Real-Time Tracking with Dashboards

Prior to AppDynamics, one investment bank struggled to track trades in real time. They were doing direct queries on the database to find out how many trades had made it downstream to the reporting database. This method was slow and inefficient, requiring employees to create and share small Excel dashboards, which lacked real-time trade information. AppDynamics APM dashboards, by comparison, enabled them to get a real-time, high-level overview of the health and performance of their system.

After installing AppDynamics, the investment bank instrumented a dashboard to show all the trades entering its post-trade system throughout the day. This capability proved hugely beneficial in helping the firm monitor trading spikes and ensure it was meeting its SLAs. And Business IQ performance monitoring made it possible to slice and dice massive volumes of incoming trades to gain real-time insights into where the transactions were coming from (i.e., which source system), their value, whether they met the SLAs, and which ones failed to process. Additionally, AppDynamics Experience Level Management provided the ability to report compliance against specific processing times.

Now the bank could automate complex processes and remove inefficient manual systems. Prior to AppDynamics, there was a team dedicated to overseeing more than 200 microservices. They had to determine why a particular trade failed, and then pass that information onto the relevant business teams for follow-up to avoid losing business. But too often a third-party source would send invalid data, or update its software and send a trade in an updated format unfamiliar to the bank’s backend system, creating a logistical mess too complex for one human to manage. With Business IQ, the bank was able to immediately spot and follow up on invalid trades.

Searching for Trades Across All Applications

Microservices offer many advantages but can bring added complexity as well. The investment bank had hundreds of microservices but lacked a fast and efficient way to search for an individual trade. In the event of a problem, they would take the trade ID and look into the log files of multiple microservices. On average, they had to open up some 40 different log files to locate a problem. And although the firm had an experienced support staff that knew the applications well, this manual process wasn’t sustainable as newer, inexperienced support people were brought onboard. Nor would this system scale as trade volume increased.

By using Business IQ to monitor every transaction across all microservices, the bank was able to easily monitor individual trade transactions throughout the lifecycle. And by capturing the trade ID, as well as supplementary data such as the source, client, value and currency, they could then go into AppDynamics Application Analytics and very quickly identify specific transactions. For example, they could enter the trade ID and see every transaction for the trade across the entire system.

This feature was particularly loved by the support staff, which now had immediate access to all of a trade’s interactions within a single screen, as well as the ability to easily drill down to find the find the root cause of a failed transaction.

Tracking Regulatory SLAs in Real Time

Prior to AppDynamics, our customer didn’t have an easy way to track the progress of a trade in real time. Rather, they were manually verifying that trades were successfully being sent to regulatory reporting systems, as well as ensuring that this was completed within the required timeframe. This was difficult to do in real time, meaning that when there was an issue, often it was not found until after the SLA had been breached. With AppDynamics they were able to set up a dashboard to visualise data in real time; the team then set up a health rule to indicate if trade reporting times were approaching the SLA. They also configured an alert that enabled them to proactively see and resolve any issues ahead of an SLA breach.

Proactively Tracking Performance after Code Releases

The bank periodically introduces new functionality to meet the latest business or regulatory requirements, in particular MiFID II, introduced to improve investor protection across Europe by harmonizing the rules for all firms with EU clients. Currently, new releases happen every week, but this rate will continue to increase. These new code releases introduce risk, as previous releases have either had a negative impact on system performance or have introduced new defects. In one two-month period, for instance, the time required to capture a trade increased by about 20%. If this continued, the bank would have had to scale out hugely—buying new hardware at significant cost—to avoid breaching its regulatory SLA.

The solution was to create a comparative dashboard in AppDynamics that showed critical Business Transactions and how they were being changed between releases (response times, errors, and so on). If any metric degraded from the previous version or deviated from a certain threshold, it would be highlighted on the dashboard in a different color, making it easier to decide whether to proceed with a rollout or determine which new feature or change had caused the deviation.

Preventing New Hardware Purchases

After refining its code based on AppDynamics’ insights, the bank saw a dramatic 6X performance improvement. This saved them from having to—in their words—“throw more hardware at the problem” by buying more CPU processing power to push through more trades.

By instrumenting their back office systems with AppDynamics, the bank gained deep insights that enabled them to refine their code. For instance, calls to third-party APIs were taking place unnecessarily and trades were being captured unintentionally within multiple different databases. Without AppDynamics, it’s unlikely this would have been discovered. The insight enabled the bank to make some very simple changes to fine-tune code, resulting in a significant performance improvement and enabling the bank to save money by scaling with their existing hardware profile.

Beneficial Business Outcomes

From the bank’s perspective, one of the greatest gains of going with AppDynamics was the ability to follow a trade through its many complex services, from the moment an order is placed, through to capture and down to regularity reporting. This enabled them to improve system performance, avoid expensive (and unnecessary) hardware upgrades, quickly search for trade IDs to locate and find the root cause of issues, and proactively manage SLAs.

See how AppDynamics can help your own business achieve positive outcomes.

Managing Software Reliability Metrics: How to Build SRE Dashboards That Drive Positive Business Outcomes

Customers expect your business application to perform consistently and reliably at all times—and for good reason. Many have built their own business systems based on the reliability of your application. This reliability target is your service level objective (SLO), the measurable characteristics of a service level agreement (SLA) between a service provider and its customer.

The SLO sets target values and expectations on how your service(s) will perform over time. It includes service level indicators (SLIs)—quantitative measures of key aspects of the level of service—which may include measurements of availability, frequency, response time, quality, throughput and so on.

If your application goes down for longer than the SLO dictates, fair warning: All hell may break loose, and you may experience frantic pages from customers trying to figure out what’s going on. Furthermore, a breach to your SLO error budget—the rate at which service level objectives can be missed—could have serious financial implications as defined in the SLA.

Why an Error Budget?

Developers are always eager to release new features and functionality. But these upgrades don’t always turn out as expected, and this can result in an SLO violation. With that being said, your SRE team should be able to do deployments and system upgrades as needed, but anytime you make changes to applications, you introduce the potential for instability.

An error budget states the numeric expectations of SLA availability. Without one, your customer may expect 100% reliability at all times. The benefit of an error budget is that it allows your product development and site reliability engineering (SRE) teams to strike a balance between innovation and reliability. If you frequently violate your SLO, the teams will need to decide whether its best to pull back on deployment and spend more time investigating the cause of the SLO breach.

For example, imagine that an SLO requires a service to successfully serve 99.999% of all queries per quarter. This means the service’s error budget has a failure rate of 0.001% for a given quarter. If a problem causes a 0.0002% failure rate, it will consume 20% of the service’s quarterly error budget.

Don’t Aim for Perfection

Developing a workable SLO isn’t easy. You need to set realistic goals, as aiming for perfection (e.g. 100% availability) can prove very expensive and nearly  impossible to achieve. Your SRE team, which is responsible for the daily operation of an application in production, must work with interested parties (e.g., product owners) to find the correct transactions to monitor for your SLO.

To begin, you must define your SLIs to determine healthy levels of service, and then use metrics that expose a negative user experience. Your engineering and application teams must decide which metric(s) to monitor, since they know the application best. A typical approach is to find a key metric that represents your SLO. For instance, Netflix uses its starts-per-second metric as an indicator of overall system health, because its baselining has led the company to expect X number of starts within any given timeframe.

Once you’ve found the right metrics, make them visible on a dashboard. Of course, not all metrics are useful. Some won’t need alerts or dashboard visibility, and you’ll want to avoid cluttering your dashboard with too many widgets. Treat this as an iterative process. Start with just a few metrics as you gain a better understanding of your system’s performance. You also can implement alerting—email, Slack, ticketing and so on—to encourage a quick response to outages and other problems.

People often ask, “What happens when SLOs aren’t met?”

Because an SLA establishes that service availability will meet certain thresholds over time, there may be serious consequences for your business—including the risk of harming your reputation and, of course, financial loss resulting from an SLO breach and a depletion of your error budget. Since the penalty for an SLA violation can be severe, your SRE team should be empowered to fix problems within the application stack. Depending on the team’s composition, it’s possible they’ll either release a fix to the feature code, make changes to the underlying platform architecture or, in a severe case, ask the feature team to halt all new development until your service returns to an acceptable level of stability as defined by the error budget.

How AppDynamics Helps You

AppDynamics enables you to track numerous metrics for your SLI.

But you may be wondering, “Which metrics should I use?”

AppD users are often excited—maybe even a bit overwhelmed—by all the data collected, and they assume everything is important. But your team shouldn’t constantly monitor every metric on a dashboard. While our core APM product provides many valuable metrics, AppDynamics includes many additional tools that deliver deep insights as well, including End User Monitoring (EUM), Business iQ and Browser Synthetic Monitoring.

 Let’s break down which AppDynamics components your SRE team should use to achieve faster MTTR:

  • APM: Say your application relies heavily on APIs and automation. Start with a few API you want to monitor and ask, “Which one of these APIs, if it fails, will impact my application or affect revenue?”  These calls usually have a very demanding SLO.

  • End User Monitoring: EUM is the best way to truly understand the customer experience because it automatically captures key metrics, including end-user response time, network requests, crashes, errors, page load details and so on.

  • Business iQ: Monitoring your application is not just about reviewing performance data.  Biz iQ helps expose application performance from a business perspective, whether your app is generating revenue as forecasted or experiencing a high abandon rate due to degraded performance.

  • Browser Synthetic Monitoring: While EUM shows the full user experience, sometimes it’s hard to know if an issue is caused by the application or the user. Generating synthetic traffic will allow you to differentiate between the two.

So how does AppDynamics help monitor your error budget?

After determining the SLI, SLO and error budget for your application, you can display your error budget on a dashboard. First, convert your SLA to minutes—for example, 99.99% SLO allows 0.01% error budget and only 8.77 hours (526 minutes) of downtime per year. You can create a custom metric to count the duration of SLO violation and display it in a graph. Of course, you’ll need to take maintenance and planned downtime into consideration as well.

With AppDynamics you can use key metrics such as response time, HTTP error count, and timeout errors. Try to avoid using system metrics like CPU and memory because they tell you very little about the user experience. In addition, you can configure Slow Transaction Percentile to show which transactions are healthy.

Availability is another great metric to measure, but keep in mind that even if your application availability is 100%, that doesn’t mean it’s healthy. It’s best to start building your dashboard in the pre-prod environment, as you’ll need to time tweak thresholds and determine which metric to use with each business transaction. The sooner AppDynamics is introduced to your application SDLC, the more time your developers and engineers will have to get acclimated to it.

 What does the ideal SRE dashboard look like? Make sure it has these KPIs:

  • SLO violation duration graph, response time (99th percentile) and load for your critical API calls

  • Error rate

  • Database response time

  • End-user response time (99th percentile)

  • Requests per minute

  • Availability

  • Session duration

Providing Value to Customers with Software Reliability Metric Monitoring

SLI, SLO, SLA and error budget aren’t just fancy terms. They’re critical to determining if your system is reliable, available or even useful to your users. You should be able to measure these metrics and tie them to your business objectives, as the ultimate goal of your application is to provide value to your customers.

Learn how AppDynamics can help measure your business success.

How to Ensure Your Applications Meet Business Goals

It makes sense that every application you use should provide tangible business value. But IT has long been stymied in proving this is true.

Traditional monitoring tools let you know how your systems were performing but offered little insight into whether a delayed response from a database—or any other performance issue—was having a cascading effect on business goals. Conversely, web analytics tools revealed how website interactions affected revenue but failed to connect adverse user behavior—like a spike in abandoned shopping carts—with a root cause.

For years, this lack of visibility has been accepted as the status quo. No longer. CIOs are increasingly taking action to capture business value. In this blog post, I’ll explain how you too can monitor your applications to ensure they are contributing to business goals as well as meeting more traditional IT performance benchmarks.

To begin, you’ll want to put together a cross-functional team that includes someone familiar with the application or applications in question from a technical perspective, someone from operations who is familiar with production support of the application, and someone who has a high-level understanding of the value the application provides to the business. This team will be responsible for defining the value of the application in measurable terms and identifying the relevant use case(s).

Defining Business Value

Applications that have a significant impact on business results usually fall into three buckets. The first bucket is obvious: Externally facing applications that bring in revenue like eCommerce or Bill Pay immediately affect the bottom line if they slow down or become unavailable. What needs to be determined by the team is how many dollars are at stake under different scenarios. In the second bucket are externally facing applications that mediate customer interactions and are generally not directly linked to revenue. However, these apps, which allow a customer to check an account balance, look up product information, or request help from a service rep, can be closely tied to other important metrics like new signups, adoption of new products and services, and customer churn. Performance failures affect those metrics, which in turn affects revenue. Over time, the poor performance of these apps will erode the value of a brand. The third bucket includes internally facing applications that deliver core business functionality like underwriting, policy quoting, and order fulfillment. The apps in this bucket drive revenue, but it is more common for them to affect metrics related to employee productivity. If these applications go down, employees are not able to get work done.

Other apps may not have a noticeable impact on the outward-facing business if they go down, but their effect on productivity and employee morale makes monitoring necessary. These include internally facing tertiary applications that handle human resource functions like onboarding, employee wellness, or expense management.

Establishing a Use Case

Once the business value of an application is determined, the next step is to understand your use case. Common use cases include business health monitoring, user journey monitoring, business journey monitoring, customer segment monitoring, and release validation. I’ll explain more about each use case below. While it is typically easier to brainstorm your own use case after learning what others have done, I expect many companies will end up creating their own unique use cases or blending common use cases together.

Business Health Monitoring: This applies to business owners who want to understand the impact of application performance on key business drivers. An example is an e-commerce site managing sales for different brands. The business owner is interested in conversion rates, the number of orders processed, total sales, and the percentage of customers moving into a loyalty program. He or she uses business health monitoring to determine the root cause behind a change in a KPI. For example, a decline in sales may be caused by an application error or a business problem.

User Journey Monitoring: User journeys are the pathways that users take through an application from start to finish. Applications that benefit from user journey monitoring are those in which the business has a vested interest in the user finishing a journey. These can be as simple as a conversion funnel for a user attempting to buy a book or a complex combination of searching, quoting, and purchasing an insurance policy. We want to understand where our users are falling out (or choosing not to continue) during their journeys so we can easily determine if their behavior is caused by application issues or something external to the application.

Business Journey Monitoring:  Rather than analyzing performance through the lens of a user, business journey monitoring is a way of evaluating the success of a holistic business process. The challenge is that a business process is likely to span applications, services, and events and involve multistep workflows. Looking at whether a complex process is meeting business objectives typically involves breaking it up into milestones and events that comprise those milestones. For a loan application, these would include submission, documents verification, credit approval, insurance underwriting, and final approval.

Customer Segment Monitoring: Identifying the most valuable users of an application and protecting them from performance problems is one way of ensuring better business outcomes. Take the case of an insurance company that manages a portal for financial professionals who want to purchase annuities on behalf of their clients. The professionals with the largest books of business are likely to generate the most revenue for the insurance company. By segmenting customers into tiers based on their number of clients, the insurance company can create health rules around their most valuable users and prioritize their issues.

Release Validation: Whether you are updating an existing application in your data center or migrating to the cloud, you need to be able to compare the before and after state of the application in reference to business KPIs. If you operate a hotel chain, your KPIs will likely include total revenue, average daily rate, and number of bookings. If KPIs fall after a new release or migration, you will want to determine the root cause and prioritize the resolution based on its impact on revenue. You may also want to measure how well a new release improved those KPIs to help validate the decision to invest the time and resources to improve that application.

Executing a Use Case

After you have determined the use case that is appropriate to your business, you will need to identify key metrics. Metrics can be as simple as the dollar value in a cart for an eCommerce application or the size or type of an internal transaction for an underwriting application. As you identify your metrics, you need to decide how you want to tell the story. It’s typically best to start with a whiteboard session where you outline what you want your dashboard(s) to look like.

This helps ensure that a layperson will be able to quickly and easily understand if the business is healthy or not.

The final step is to define your data collectors and build your dashboard. You will also want to create alerts if trends begin to deviate from established baselines. You are now ready to let the rubber of business value, use cases, and key metrics meet the road of real-time data. As you learn how application performance is affecting your business, you can now act on those results!

Learn more about how AppDynamics helps you drive business performance through application performance with Business iQ!

The Consumerization of IT and What It Means for Modern DevOps

As professionals in the IT space, we’re constantly introduced to new terms, concepts, technologies and practices. In many cases, we view these terms as IT-specific to help us be more proficient and cutting-edge. Or at least that’s what we strive to achieve. With many companies trying to disrupt the verticals they target, it’s important for us to understand how these new facets of technology impact the bottom line.

Business is under pressure to deliver more groundbreaking ideas than ever. Understanding the impact of new technology will empower you to engage business leaders to be supportive in both principal values and budgetary needs. When you look at companies that successfully disrupted a specific space, you’ll see one key element in the mix: the end-user experience. This used to mean how nice your app looks. Today, the user experience is more about speed and ease-of-access, while also maintaining a level of confidence that the app will do what it’s supposed to. If you fail to deliver this, your end user will simply move on. As a matter of fact, Bloomberg reports that approximately $6 trillion dollars are moving from digital laggards to businesses that provide the best user experience through digital transformation.

The first point we all must realize is that in the past 10 years, the consumerization of IT has taken the industry by storm. What this really means is that consumers of your IT services are not much different from consumers of your public-facing applications. We know that public-facing applications are the front door to your business—this is where customers are won and lost by the adoption of technology. As an IT professional, your internal business leaders are your customers, and it’s up to you to deliver and drive technology solutions that help drive the business metrics so near and dear to your “customers” hearts. So, making sure you articulate the changes to your principles reflects how IT will impact your business.

Secondly, a DevOps shift for internal process and procedure is no easy feat, especially when you’re dealing with years of hardened policies and practices. But it’s crucial for building a modern DevOps function. This means you’ll need to coordinate a holistic effort that includes development and operations teams, as well as the line of business. Otherwise, the moves you need to make will become exponentially more difficult as business demands become more severe, particularly with the rising number of disruptors in your market.

Lastly, the proof is in the pudding. When you begin your journey to the DevOps shift, it’s critically important to keep all the key players engaged, thereby enabling them to see the value you’re bringing to the table. This is particularly important when you’re demonstrating how new technology implementations are impacting the business in a positive—or negative—way. In this scenario, what you’ll show is either, “Yes, our technology is on the right path,” or “No, the implementation is giving us a negative response from our customers, so we need to quickly course-correct to minimize the damage and regain a positive direction.”

When all is said and done, we must understand the user experience is the new currency in the hyper-connected world we live in. But what is even more critical is that frequent change is required to stay ahead of the competition. This is where your business leaders come in. It’s in nobody’s best interest to stay stagnant, regardless of your industry. Disruption has hit retail, transportation, finance, healthcare and the list goes on. Making frequent changes to beat the disruptors requires you to build out a DevOps practice to ensure you have the ability and tools to respond to high business demands. Here’s how this impacts the business and helps push your DevOps plan forward:

  1. Business leaders are under tremendous pressure to drive continuous growth. A flat-line approach is a leading indicator that your company is falling behind competitors. Highlighting that you want to build out a practice that enables you to quickly develop, monitor, analyze and respond is exactly what your business wants to hear. But be prepared to knuckle down, as this is a never-ending loop to ensure you’re on the right path. When your leaders understand they’re now part of the process, they’ll become more tightly aligned with your strategy.

  1. Defining the critical metrics with your business leaders will allow you to understand how your technology provides the greatest impact. This can include any array of vital measurements that enable you to correlate your application performance to key business metrics. And not necessarily just monetary metrics either—they can be tied to conversions, promotion success, how frequently users are using (or not using) your application, and overall customer satisfaction.  Having these metrics in place will ensure your business leaders’ involvement moving forward, and gain their confidence that your strategy is on target.

  1. Embrace analytics to gain the ability to understand business transactions and the user journey.  The key part here is that you’re building out a DevOps strategy to be lean and nimble, but the end goal must be to understand how your end users are reacting to your applications. Leveraging an analytical platform like AppDynamics Business iQ is key to showing how your application ties directly to metrics defined by your business leaders. These leaders will gain immediate value from key data they’re not accustomed to seeing. This effort will also help you set priorities on which items you should develop first.

  1. As with Agile development and DevOps, this is an iterative process and a continuous cycle. Automating the process to remove the human element is key: Leveraging AI to help predict anomalies and stay ahead of the consumer will build the highest degree of confidence in your new DevOps implementation. However, this can’t be done in a silo. Once everyone is involved and engaged, showcasing your strategy to other parts of the business will be as celebratory a ticker-tape parade by a championship-winning team. Take your success and show how you’re an innovative technology leader—not one who sits in the server room, but rather one who’s engaged with the business. One who proudly bears the title, “Disruptor.”

Smart monitoring and automation help business leaders see issues that concern them most, and should be your first priority when rolling out organizational changes. In addition to pinpointing issues within applications, these tools help predict future issues by identifying trends as they arise. Consumerization of IT has taken our world by storm. Business leaders have lost faith in IT, which needs to reinvent itself as a leader driving business, rather than a team of technicians responding to the crisis of the day. By implementing a valuable, new technological shift—one with all the right tools in place, a keen understanding of the business, and impactful solutions—you’ll be seen as a key partner and leader with innovations that disrupt the competition and make your business a success.

Learn more about how AppDynamics can help you succeed with your business transformation.

Business iQ Enhancements in 4.5: Connecting the dots between applications and business

Digital transformation has brought applications into the limelight. The importance of application performance monitoring has grown many-folds and application and business correlation is more critical than ever. Applications are the touch-points between businesses and the end-users, and therefore, influence the business strategy.

Business iQ, the business performance monitoring solution from AppDynamics, helps our customers correlate application performance with critical business metrics in real-time. The focus on providing these real-time insights in an uncomplicated, easily operable manner is a theme persistent across our product releases. In 4.5 as well, released in July 2018, our team focused on delivering solutions that are easy to use and solve key business performance issues.

Improved Business Correlation

Business Journeys was released in 4.4, in Nov 2017, and provides end-to-end visibility into multi-step complex business workflows by stitching them together through unique identifiers. In 4.5, we provide out-of-the-box dashboards and metrics with no manual configuration required. These dashboards provide aggregate view of the key data points such as average total time, average wait time between milestones, conversion, number of events per milestone etc. for all the business journeys. An example of a loan application approval business journey is shown below.

Out-of-the-box metrics on business journeys allow users to create health rules and alerts to track deviations from the normal business workflow. These metrics can be compared with past data using machine-learning based dynamic baselines. The screenshot below compares the average total time for loan approval in the last hour with the last 15 days baselined data.

We also added support for custom data sources to the configuration and one-click access from aggregate view to the underlying event data – all these enhancements focused on providing more business insights with just few clicks.

Experience level management (XLM) allows users to monitor and report on key service levels and end-user experience levels critical to delivering high-performing applications. Since these experience levels and service levels differ based on geographies, time zone support for such reports is a must-have requirement. XLM configuration now supports different time zones. For e.g. United Airlines can now track the login response time separately for East Coast and West Coast, or for North America and Europe.

Business iQ’s analytics and reporting capabilities are powered by AppDynamics Query Language (ADQL) and UI widgets.

Additional ADQL Operators

New ADQL operators, HAVING, and SINCE..UNTIL, enable more sophisticated aggregation and filtering. The HAVING clause is used to filter groups created by aggregate functions such as SUM or AVG as the WHERE clause cannot be used with these aggregate functions. For example in a financial application, to list the business transactions with average response time greater than 5000 ms.

SINCE..UNTIL makes it easier for the user to select specific time window in the ADQL search query and not get tied to the UI based time picker. For example, the following search query can be used to return all events from Black Friday by using unix/epoch time for Nov 24, 2017

SELECT * FROM transactions SINCE 1511510400 UNTIL 1511596800

Or use the following query to simply search for events from the last one hour

SELECT * FROM transactions SINCE 1 hour

Widgets Enhancements

Enhancements to our widgets allows more precise widget customization and make it easier to interpret trends in your event data. Log axis for time series widgets, level of significance and trailing period comparison for numeric widgets along with other enhancements enable users to compare and highlight metrics that are most important.

Business Metrics

Business Metrics are used to monitor values of certain repetitive ADQL searches such as per minute data on the number of customers impacted by the slowness in a login application. The Metric Listing page has been updated to provide a more intuitive experience. Click on a metric name to open a pre-populated Metric Browser or select multiple metrics to view them together in the Metric Browser for comparison. The Metric page now displays the underlying ADQL query for metrics, making it easy to see what a particular metric represents and how it is calculated.

Greater Scale

On our underlying Platform, called the Event Service, we continue to make improvements to our existing architecture to ensure maximum uptime, real-time availability of data, and blazing fast query response time. This will allow our platform to scale to even greater heights, ingest more events, and respond to queries at the performance AppDynamics users can expect.

Agentless Analytics

We are also excited to launch a Beta program for using Transaction Analytics without the need to install an additional Analytics Agent and enable analytics data collection with the snap of a finger.

The focus of product teams at AppDynamics is to deliver easy to use solutions providing key application and business performance insights. This cannot be achieved without the valuable feedback from our customers. Feel free to reach out to your AppDynamics account team to share your thoughts and to learn more about what’s new in 4.5 or what’s coming in near future.

Louis Huard and Stefan Hermanek also contributed to this blog post. 

Improve the Productivity of Relationship Managers and Financial Advisors with Business iQ

Every job has its mundane administrative tasks, and we all hate them. In the world of wealth management, relationship managers are pressured to serve as many existing customers and prospects as they can with the ultimate goal of increasing the assets under management (AUM)—one of the key metrics used to measure their productivity. Similarly, in the insurance industry, financial advisors are driven to maximize their time with clients. Administrative tasks are not only irritating, they also reduce a salesperson’s paycheck by cutting into his or her time with customers.

But organizational forces in both the wealth management and insurance industries are conspiring against their top revenue generators. According to Seismic, a staggering 65% of a relationship manager’s time is spent on business processes like account opening, accessing collaterals, and creating customized portfolio review with customers.

In the last few years, financial institutions and insurance companies have sought to free up their salespeople by investing in productivity tools. Mobile apps, in particular, hold the promise of speeding up processes like filling out client forms for clients, creating proposals, and building portfolios. They are also, in theory, a great way to deliver real-time market insights.

But mobile apps are only effective when relationship managers and advisors use them.

AppDynamics Business iQ allows organizations to measure the effectiveness of their mobile apps by providing a window into user behavior. In the example below, I show how a financial institution can instantly see how many relationship managers have clicked on a market insight to access AI-driven financial advice—a killer feature for increasing AUM. The dashboard, which I created in the AppDynamics demo environment, also shows how many relationship managers proceeded to “Add to Cart” and re-balanced their clients’ portfolios. We see that as relationship managers moved through the funnel, they increasingly abandoned the app. The overall conversion rate is just 5.62%. Slightly over one in twenty relationship managers used the application to send a proposal to their clients.

Below, I show how to a create conversion funnel using a built-in widget. It is as simple as going to the Add Widget tab and selecting Analytics and Funnel Analysis.

 You then select the business transaction that you’d like to include in the conversion funnel.

You can also quickly design a custom widget to highlight information such as the relationship managers who are generating the most new business.

Figure:  RMs with the highest new AUM

Or see at a glance the relationship managers who are sending the highest number of proposals. Moreover, you can break down the proposals by customer type. So you can see which customer type (Silver, Gold, Platinum, Diamond) the relationship managers are creating the proposals for. In the example below, you can see that relationship manager “aleftik” is sending all 960 proposals to only “Silver” tier customers. Relating the previous graph where the highest AUM is “aleftik” and he focuses all his effort to selling to the “Silver” tier customers, it appears that this is a desirable behaviour and strategy that the business should educate and share among other relationship managers.

Figure: RMs with the highest “Send Proposal” Transactions

Moving beyond the performance metrics of individual relationship managers and financial advisors, you can combine technical and performance metrics in order to see if updates to an application are negatively affecting sales performance.

You can see from the above conversion graph that version 2 of the code has significantly reduced the slowness (yellow and orange color within the bar) for Portfolios Summary page, positively impacting the conversion ratio from 5.53% to 14.52%

The business may also want to identify relationship managers who are not using the new productivity tool enough. Below is a way to create such a list of managers with the least number of page hits.

Figure: Number of page visit on “Market Insights” by RMs in ascending order

You can even put all of this together to have a customizable dashboard combining both technical and business performance metrics. At a glance you’re able to see the new AUM achieved by the wealth management group using the iPad application, transaction health of each key business process, top performing relationship managers and the products sold, as well as relationship managers who have yet to adopt the new application as a productivity tool.

With AppDynamics Business iQ, institutions do not need to wait for a month or a week to see business insights in relation to application performance and user behaviour. All the information is available at a glance in real time.

The AppD Approach: Increasing Mobile Engagement with Business iQ

With 2.3 billion smartphone users worldwide, enterprises can’t afford to ignore the mobile channel. Last year, $2 billion of Black Friday’s total sales of $5.03 billion were spent on mobile devices. And mobile isn’t just about shopping. People are now spending equal amounts of time on mobile devices and desktop computers. Flurry Insights pegs the average time that people devote to mobile devices at 5 hours a day.

In an attempt to capture a fraction of their attention, nearly three-quarters of enterprises surveyed by Gartner last year deployed an average of 8 mobile applications. But their efforts were only modestly rewarded. That is because total mobile engagement is depressingly low. According to Localytics, 63% of users visit a mobile app less than 10 times, while 24% use an app only once.

Product managers are trying to address the engagement crisis by creating more compelling user experiences. But they find it hard to measure their success. They often have to guess: Is the new feature I rolled out last week pleasing users or annoying them? It turns out that producing great mobile experiences requires the same deep visibility as more traditional applications do. Product managers and their operations teams need to understand user journeys and to quickly diagnose performance issues. This is simple to do with AppDynamics Business iQ, of course, but I suspect that few product managers understand how easy it is to set up and use our solution.

I was reminded of this during a recent customer engagement at a large bank. The product manager was updating one of the bank’s mobile apps, and she asked me if AppDynamics could help her determine if users liked the new features she was adding or not.

Given that there will be around 200 billion mobile applications downloaded this year, I suspect her question is a common one. In the rest of this blog post, I’ll walk through the steps I showed the product manager so that others can get started just as quickly.

The first step is to go to the “Analytics” tab and “Add a new Search.”

Since feature registration is one of the Business Transactions automatically discovered by the APM agent, we can use AppDynamics Query Language (ADQL) to select all the features that have been registered by all users.

The screenshot below (a mockup) shows an ADQL statement that will count all the features that have been registered during a selected time period. You can see that the top features registered are “Quick Balance,” “Push Notification,” and “Quick Transactions.”

As developers continue to release new features for the application, the Analytics Dashboard will automatically display the registration occurrences of new features without any reconfiguration (i.e. ADQL statement).

Let’s say that “Quick Balance” is the new feature that the product manager has just released. The fact that there are 350 registrations indicates that the users are responding favorably to the pop-up screen promoting the feature.

However, as the promotion continues, the product manager notices a growing number of deregistrations from the Analytics Dashboard. By the end of the week, Quick Balance is showing the highest number of deregistrations.

Let’s assume the high number of deregistrations also coincides with negative feedback left on a handful of mobile app rating sites, providing quantitative confirmation that users do not like it.

By drilling into the “Quick Balance” transactions, the product manager can see at a glance that the performance of the feature is being affected by errors.

At this point, she can share the dashboard with IT Ops and ask them to follow up on the problems associated with Quick Balance. Once the performance issues are corrected, the Analytics Dashboard should register a decrease in deregistrations and the rating sites will also likely begin to show positive feedback.

Successful product management in the digital world requires rapid insight into user experience. Learn more about AppDynamics Business iQ and how you can gain real-time awareness of application performance, user experience, and their impact on your business performance.

How to Get a Seat at the Table with Business iQ Visualizations

In an era of software-driven business, there is no doubt that IT deserves a seat at the table with top business decision-makers. But how do you quickly convince the Business that IT is about more than servers and software? One way to demonstrate the critical role IT metrics play in business decisions is to simply click over to the Analytics tab of your AppDynamics screen. If you are new to Analytics you may be surprised by the powerful visualizations that are available out of the box.

For example, if you are an eCommerce firm, you can quickly learn the overall Conversion Rate as well as the percent of “Abandoners” for each step of your customers’ Check Out Journey, simply by utilizing the Funnel Widget:

If you are an insurance company, you could learn the “health” of every step of your on-line Customer Journeys by product:

If you are a travel agency, you will be able to visualize your revenue stream by customer segment and see the impact of defective transactions by location:

Some have gotten the impression, based in part on their experience with other business intelligence tools, that getting real, actionable business information from their IT systems will require a major investment of time. They worry about the steps it takes to extract business data from within their technical systems. What they don’t realize is that a lot of this work is done by AppDynamics out of the box. This is especially true when you leverage Business iQ on top of our core APM and EUM offerings.

I’ll show you how easy this is. Let’s walk through examples of some correlated business and performance information you can learn just by playing around with the visualization widgets.In these examples we’ll focus on e-commerce, but you can easily imagine relevant business metrics for your own use case.

If you place APM metrics, like Average Response Time and/or Number of Errors, next to the sequential order of customer events, you can better understand what is the business impact of your performance problems.

Or you can also try placing a funnel with Conversion Rates with the Abandoners numbers for all major steps in your Customer Journeys just below the “overall health” of every step of their respective business journeys (represented by the rows of circles):

A little more advanced “play” can involve ADQL (AppDynamics Query Language) queries, creating metrics for the ADQL queries’ search results and Health Rules for those metrics.

Let us start with the ADQL queries. In the Controller UI, go to Analytics > Searches and create the required ADQL queries, for example:

-What is the number of Unique Visitors?

SELECT distinctcount(segments.userData.SessionID) FROM transactions WHERE application = "eCommerce"

-What is the number successful Orders Placed?

SELECT count(*) FROM transactions WHERE application = "eCommerce" AND transactionName = "CheckOut" AND userExperience != "ERROR"

-What is the total amount of Revenue?

SELECT sum(segments.userData.ProductPrice) FROM transactions WHERE application = "eCommerce" AND transactionName = "CheckOut"

-What is the total amount of Revenues at risk?

SELECT sum(segments.userData.ProductPrice) FROM transactions WHERE application = "eCommerce" AND transactionName = "CheckOut" AND userExperience = "ERROR"

Save the most interesting ADQL queries search results so that you can create metrics that are updated in near real time. In the Controller UI, click over to Analytics > Metrics and then simply create from your saved queries. E.g.:

-> # of Orders Placed Metric
-> # of Unique Visitors Metric
-> Check Out Total Processed Metric
-> Check Out Total Revenues at Risk Metric

Use a time series graph chart widget and “Check Out Total Processed Metric” data to visualize the total amount of Revenue trend over time. Notice that you can include a dynamic baseline so that your thresholds are based on average behaviour, calculated hour by hour for a given time period

(i.g. Daily Trend, Weekly Trend, Monthly Trend etc.). Using the baseline helps you to find an answer to the following question: Is the total amount of Revenue normal at the given hour?

Now you may wish to create Health Rules for the remaining business metrics so you can visualize them and receive alerts in the same way you would for your technical performance metrics. In the Controller UI, go to Analytics > Alert & Respond > Health Rules and create your Health Rules for the new business metrics. We’ll use static thresholds here for simplicity, but again dynamic baselines are likely the better way to go for your use-case.

-eComm – Orders lower than expected:

Critical condition: # of Orders Placed < 700
Warning condition: # of Orders Placed < 800

-eComm – Unique Visitors lower than expected

Critical condition: # of Unique Visitors < 10
Warning condition: # of Unique Visitors < 60

-eComm – Check Out Processed lower than expected

Critical condition: Check Out Total Processed < 200
Warning condition: Check Out Total Processed < 100

-eComm – Revenues at risk higher than expected

Critical condition: ({RevenuesAtRisk}/{CheckOutTotalProcessed}*100) > 10 (%)
Warning condition: ({RevenuesAtRisk}/{CheckOutTotalProcessed}*100) > 2 (%)

Now you can create and link some Health Status widgets with the relative Health Rules:

And finally you can put all of your interesting components together in Dashboards, for example:

Refer to the AppDynamics documentation for more ideas on ways to explore your data.

It doesn’t take long to create these kinds of visualizations, but the insight you can offer to your business leaders is incredibly powerful. As you get comfortable using these basic building blocks you will see opportunities to assemble them into more complex analyses such as business health, the steps of the business journey, performance by segment, and more.

Learn more about Business iQ and how it can give you a seat at the table.

Krzysztof Gawronski is part of AppDynamics Global Services team, which is dedicated to helping enterprises realize the value of business and application performance monitoring. AppDynamics’ Global Services’ consultants, architects, and project managers are experts in unlocking the cross-stack intelligence needed to improve business outcomes and increase organizational efficiency.

AI’s Arrival in the Enterprise Will Have Profound Implications for IT

At the end of Twentieth Century in the wake of Deep Blue’s triumph over Garry Kasparov, it was popular for computer scientists to speculate about when human begins would begin to interact with artificial intelligence. It was generally believed that machines with true reasoning capabilities were decades away. In an interview with author Michio Kaku, published in Visions: How Science Will Revolutionize the 21st Century, AI expert and Carnegie Mellon professor Hans Moravec predicted that robots would be able to model the world and anticipate the consequences of different actions sometime between 2020 and 2030.

Twenty years later, we are no longer wondering about how artificial intelligence (AI) will first appear in our lives. It has arrived in the form of virtual assistants like Alexa and self-driving cars. But this can give a misleading impression of what we can expect from AI in the next few years. AI software is not going to evolve human-like reasoning capabilities anytime soon.

Indeed, most of what is described as AI is really machine-learning algorithms that act largely as detectors. These algorithms analyze massive amounts of data and learn to discriminate between normal and anomalous behavior. AI, where it exists, is similar to a decision-support system for reacting to behavior as it changes. But even in these early stages, machine learning and AI are changing the game for IT operations. In the next few years, the impact of machine learning and AI will be profound.

The problem enterprises are facing is that computing environments have simply grown too large and too complex for human beings to monitor alone. To effectively monitor enterprise systems, IT must track millions of metrics per second. This is not a challenge that can be met by putting another screen on the wall of the network operations center. There are already too many screens, and just contemplating the number of screens that would be required is overwhelming. Even more daunting is figuring out the five or ten metrics that matter the most out of five or ten million as every new millisecond brings the system to a new dynamic state.

The company I founded, Perspica, which was was acquired last year by AppDynamics/Cisco, solved this problem for our customers by applying machine learning and AI to massive amounts of streaming telemetry data generated by applications and IT infrastructure. What Perspica did was surface all the relevant metrics and then use those metrics to accelerate root cause analysis and reduce the mean time to repair. But Perspica’s ability to grow beyond that was limited by the data that we had access to. In fact, everyone involved in machine learning and AI at that time faced the same limitation. We lacked a source of truth on which to train our algorithms to go beyond what they had already achieved.

But this limitation is rapidly being overcome. Increasingly, data scientists are gaining access to new sets of what we call labelled data—sets of numbers or strings of text that a computer can understand as a true representation of something else. Data scientists who work with IT data, in particular, are finding that enough labelled data exists that we can realistically begin talking about automating large parts of IT in the next two or three years. And that is only the beginning.

In the future, every enterprise is going to have some combination of machine learning and AI to monitor its computing environments, and, equally important to understand how changes to those environments affect business goals. As these systems are deployed, they will become smarter and more sophisticated. Every application, every server, and every port on that server will have its own unique AI model, which means if you have 50 applications running on 10,000 servers you will need to train 500,000 models. This is not something that is going to be created overnight. But once these models are put in place, self-healing systems will become standard.

We’ll see AI playing a role in everyday IT and business events. For example, imagine a large airline that is planning on holding a worldwide promotion. The airline’s IT department rolls out new application code as a canary deployment. But the monitoring system soon reveals the new release is performing worse than the old code. While the business owner and IT staff are realizing that the code push has failed, the airline’s AI system is determining the root cause is a disk space issue and taking steps to address the problem.

For many years, Perspica and others were doing detection. Today, as we broaden the libraries, increase the sets of problems that have solutions and bundle those solutions together, we’ll be able to start doing remediation. Moravec, it seemed, had the timeline correct.

What will happen in the next twenty years? The media sometimes promotes “fear of AI.” But I see AI making business more profitable and people more productive. It will improve service quality, reliability, and availability not only in IT but across all industries. And in this way, AI will not only have profound implications for IT. It is also bound to improve the human condition.

Customer Spotlight: Carhartt Wins 2018 CIO 100 Award

Today, our customer Carhartt was honored as a 2018 CIO 100 Award Winner by CIO.com and to say that we are thrilled is simply an understatement. It’s an incredible accolade and a nod to the tireless work that CIO, John Hill, and his team have done to transform Carhartt into a premiere fashion and workwear brand.

The CIO 100 Award program recognizes organizations around the world that exemplify the highest level of operational and strategic excellence in information technology – and Carhartt’s digital transformation was not overlooked. Founded in 1889 with just two sewing machines and five employees, Carhartt has since grown into a global brand, overcoming the Great Depression, and embracing the digital age to build a vibrant online sales platform and loyal customer base.

Progressive CIOs like John are leading their companies through exciting change in their industries and we’re thrilled Carhartt chose AppDynamics Business iQ to aid in their digital transformation.

“Retailers live and die by the consumer experience, whether that experience comes in person, through mobile or on the web,” John explains in a press release announcing the win. “It was imperative to understand how to position our business for the evolving consumer needs of today and in the future. With Business iQ, we now have a direct lens into the health of business transactions across our entire digital shopping journey in real time, so we can continue to deliver the highest quality user experience to our consumers.”

I got a chance to steal a few minutes of John’s time to congratulate him on this win and to find out what else Carhartt has been up to. Check out our brief Q&A:

Prathap Dendi: The CIO 100 award is a recognition of the innovation that you and your team have been able to implement. What does this acknowledgement mean for you personally and for Carhartt as a whole?

John Hill: The CIO 100 award is a great honor for Carhartt and represents our commitment as an IT organization to further drive Carhartt into the digital era through innovation, enablement, thought leadership, and collaboration.

Prathap Dendi: What has the past year looked like in terms of collaboration with the IT and Business teams at Carhartt? What do you attribute that to?

John Hill: Increasing collaboration between our business product owners and our IT teams has been instrumental in driving our ability to quickly deliver better solutions that are more stable, efficient, and effective.  The increased transparency in real time access to application/business metrics and data has made these collaboration efforts more effective than ever before.

Prathap Dendi: If you could describe the growth you’ve witnessed over the past year in one word, what would it be?

John Hill: Awesome.

Prathap Dendi: How do your teams feel about having access to real-time data and being able to collaborate more with the business teams?

John Hill: The ability to see in real-time what is happening in the application is key to supporting a 24/7 platform.  The ability to see issues as they happen and trace them back to their root causes has enabled our teams to become more proactive and confident in the solutions they are delivering. The business teams have also embraced the ability to see real time data regarding new features and releases. It has increased the trust and confidence that both teams have in each other.

Prathap Dendi: How have the other executives at Carhartt responded?

John Hill: We have received great feedback from senior leaders as we continue to expose key real-time business data to key stakeholders throughout the organization. This data is empowering leaders to make better, faster decisions that directly impact the business as it happens.

Prathap Dendi: What’s next for Carhartt? What are you hoping to delve into in the next six months, year?

John Hill: Deeper user experience metrics and analysis, complete business process monitoring across additional systems of record.

Prathap Dendi: What has been your experience working with AppD?

John Hill: AppD has been a true partner. They have engaged at every level to deliver a complete solution. AppD brought their extensive APM experience to our implementation and enabled us to hit the ground running. They immediately identified a number of key processes that could be monitored and analyzed through the tool.

I’m so excited for everything that John and the Carhartt team have accomplished, and I’m looking forward to seeing what else is in store for them. Here’s to many more wins like this one!

Take a guided tour today for an in-depth look at how Business iQ puts application teams in a position to drive the business.