Understanding Performance of PayPal as a Service (PPAAS)

In a previous post – Agile Performance Testing – Proactively Managing Performance – I discussed some of the challenges faced in managing a successful performance engineering practices in an Agile development model.  Let’s continue this with a real world example, highlighting how AppDynamics simplifies the collection and comparison of Key Performance Indicators (KPIs) to give visibility into an Agile development team’s integration with PayPal as a Service (PPaaS).

Our dev team is tasked with building a new shopping cart and checkout capability for an online merchant. They have designed a simple Java Enterprise architecture with a web front-end, built on Apache TomEE, a set of mid-tier services, on JBoss AS 7, and have chosen to integrate with PayPal as the backend payment processor. With PayPal’s Mobile, REST and Classic SDKs, integrating secure payments into their app is a snap and our team knows this is a good choice.

However, the merchant has tight Service Level Agreements (SLAs) and it’s critical the team proactively analyze, and resolve, performance issues in pre-production as part of their Agile process. In order to prepare for meeting these SLAs, they plan to use AppDynamics as part of development and performance testing for end-to-end visibility, and to collect and compare KPIs across sprints.

The dev team is agile and are continuously integrating into their QA test and performance environment. During one of the first sprints they created a basic checkout flow, which is shown below:

Screen Shot 2014-04-29 at 9.28.21 AM

For this sprint they stubbed several of the service calls to PayPal, but coded the first step in authenticating — getting an OAuth Access Token, used to validate payments.

Enabling AppDynamics on their application was trivial, and the dev team got immediate end-to-end visibility into their application flow, performance timings across all tiers, as well as the initial call to PayPal. Based on some initial performance testing everything looks great!


NOTE: in our example AppDynamics is configured to identify backend HTTP Requests (REST Service Invocations) using the first 3 segments of the target URL. This is an easy change and the updated configuration is automatically pushed to the AppDynamics agent without any need to change config files, or restart the application.


In a later sprint, our dev team finished integrating the full payments process flow. They’re using PayPal’s SDK and while it’s a seamless integration, they’re unclear exactly what calls to PayPal are happening under the covers.

Because AppDynamics automatically discovers, maps, and scores all incoming transactions end-to-end, our dev team was able to get immediate and full visibility into two new REST invocations, authorization and payment.


The dynamic discovery of AppDynamics is extremely important in an Agile, continuous integration, or continuous release models where code is constantly changing. Having to manually configure what methods to monitor is a burden that degrades a team’s efficiency.

Needing to understand performance across the two sprints, the team leverages AppDynamics’ Compare Releases functionality to quickly understand the difference between performance runs across the sprints.


AppDynamics flow map visualize the difference in transaction flow between the sprints, highlighting the additional REST calls required to fully process the payment. Also, the KPI comparison gives the dev team an easy way to quickly measure the differences in performance.


Performance has changed, as expected, when implementing the full payment processing flow. During a performance test AppDynamics automatically identifies and takes diagnostics on the abnormal transactions.


Transaction Snapshots capture source line of code call graphs, end-to-end across the Web and Service tiers. Drilling down across the call graphs, the dev team clearly identifies the payment service as the long running call.


AppDynamics provides full context on the REST invocation, and highlights the SDK was configured to talk to PayPal’s sandbox environment, explaining the occasional high-response times.

To recap, our Agile dev team leveraged AppDynamics to get deep end-to-end visibility across their pre-production application environment. AppDynamics release comparison provided the means to understand differences in the checkout flows across sprints, and the dynamic discovery, application mapping, and automatic detection allowed the team to quickly understand and quantify their interactions with PayPal. When transactions deviated away from normal, AppDynamics automatically identified and captured the slowness to provide end-to-end source line of code root-cause analysis.

Take five minutes to get complete visibility into the performance of your production applications with AppDynamics today.

Accessing your performance data with the AppDynamics REST API

App Man previously covered how to go mobile with the AppDynamics REST API and Dojo highlighting the AppDynamics Gauges project. In this post, we will cover how to get the most out of AppDynamics by leveraging the REST API.

Custom Dashboards

AppDynamics comes with very powerful and sexy custom dashboards that anyone can build. With our custom dashboards you can combine and graph any metric and easily come up with an ops- or business- focused dashboard for your executive team.

DevOps Dashboard:

DevOps Dashboard

Management Dashboard:

Management Dashboard

Get started building a custom dashboard.

Sometimes a custom dashboard is not enough and you really need the data underneath. If our amazing HTML5 custom dashboards are not enough, at the end of the day your data is yours. The goal of this post is to provide an introduction on how to access your performance data.

AppDynamics is extensible

AppDynamics Pro allows you to track any custom metric through the machine agent. This means you can easily track any application metrics via AppDynamics and use the REST api to mashup the data (think of it as a replacement for the popular StatsD and Graphite combination). The REST API allows you to access metrics and mashup your data with your internal tools.

How to access performance data via REST API

From within the AppDynamics metrics browser you can easily access the url for the API for a particular metric by right clicking as shown here:


Access data via the command line

Once you have the API endpoint needed for the metrics you want you can easily access it via curl on the command line. Our REST API uses HTTP basic authentication. If you have installed the Controller on a single-tenant platform, your default account is:

Username: username@customer1
Password: xxxx

If you are using the SaaS Controller, your username is your AppDynamics user name in your AppDynamics account:


The credentials are the same you use to log in to the controller. Below are common examples:

Retrieve a list of applications available

curl --user username@example:password 'http://example.saas.appdynamics.com:8090/controller/rest/applications?output=JSON'

    "description": "",
    "id": 7,
    "name": "AppDynamics"
    "description": "",
    "id": 5,
    "name": "Knp Ipsum"
Retrieve the average response time of a specific application

curl --user username@example:password 'http://example.saas.appdynamics.com:8090/controller/rest/applications/AppDynamics/metric-data?metric-path=Overall%20Application%20Performance%7CAverage%20Response%20Time%20(ms)&time-range-type=BEFORE_NOW&duration-in-mins=60&output=JSON'

  "frequency": "ONE_MIN",
  "metricPath": "Overall Application Performance|Average Response Time (ms)",
  "metricValues": [  {
    "current": 41,
    "max": 131,
    "min": 9,
    "startTimeInMillis": 1375298040000,
    "value": 50
Retrieve number of requests per minute for a specific application

curl --user username@example:password 'http://example.saas.appdynamics.com:8090/controller/rest/applications/AppDynamics/metric-data?metric-path=Overall%20Application%20Performance%7CCalls%20per%20Minute&time-range-type=BEFORE_NOW&duration-in-mins=15&output=JSON'

  "frequency": "ONE_MIN",
  "metricPath": "Overall Application Performance|Calls per Minute",
  "metricValues": [  {
    "current": 2,
    "max": 0,
    "min": 0,
    "startTimeInMillis": 1375300860000,
    "value": 2
Getting started with the REST API via SDKs

The easiest way to get started working with the REST API is to use the SDK available for Java and Python. The SDKs provide a lightweight wrapper around our REST API and are available via AppDynamics X. The AppDynamics REST API offers XML and JSON for interoperability.

Accessing the REST API via the Python SDK

from appd.cmdline import parse_argv
from appd.request import AppDynamicsClient

args = parse_argv()
c = AppDynamicsClient(args.url, args.username, args.password, args.account, args.verbose)
for app in c.get_applications():
print app.name, app.id

The best example is for fetching metrics for business transactions. Checkout the Python SDK for more examples.

How to track a custom metric with machine agent

The machine agent can act as a REST gateway to capture custom metrics into the AppDynamics metrics browser. In order to enable the machine agent API, start the machine agent with the following options:

sudo java -Dmetric.http.listener=true -Dmetric.http.listener.port=8293 -jar /opt/appdynamics/machine-agent/machineagent.jar

Once the machine agent is listening, tracking custom metrics is as easy as making a post request to:


Going even farther

Two weeks ago we announced AppDynamics X where we released many integrations that leverage the machine agent to track custom metrics from Nginx, Apache, Cassandra, and more. The real power of AppDynamics is not the user interface, but the stats and business intelligence platform under the hood.

As always, please feel free to comment if you think I have missed something or if you have a request for content for an upcoming post.

Going Mobile with AppDynamics REST API

Its always great when customers want to build their own applications on top of your data and platform. A few weeks back one of our customers in Europe decided to build their own mobile application, so they could monitor the performance of their mission-critical business transactions from any smart phone or mobile device. Here is the unedited story we received of how this customer went mobile with AppDynamics:

I looked into AppDynamics’ REST API and was very keen to use that data but was unsure of how I could visualize the it. Since all data per monitoring point was available in either XML or JSON format, it seemed the ideal choice to go with a Javascript user interface. After searching around I found the Dojo Gauges and started to code up a simple webapp using AppDynamics’ REST API and data we collect everyday.

Technology Stack

Here is a quick overview of the architecture and technologies used in the mobile application. 

The Mobile Application

Our application is hosted within Django so I can use django’s powerful dynamic admin interface for a backend. Though this is not implemented yet so for now I am just using it to serve some Static content and proxy AJAX calls to AppDynamics.

The core application consists of:

  • index.html
  • dojo + jquery
  • views.js + settings.js
  • widget-glossygauge.js
  • widget-jira.js
  • images

Functionally, the webapp serves the basic static page layout to the browser on the client device and then instantiates the Dojo GlossyGauges which I have extended to include their own embedded self-update timers. The update timers update the gauge values by making REST calls to AppDynamics through the proxy module. The proxy module is necessary as the browser will block cross domain ajax requests. You probably don’t need it if you host the application on the same server as AppDynamics and use apache + mod_proxy.

Here is a screenshot of our mobile application:

The gauges update themselves in real-time so there is no need to refresh the page.

Adding new gauges is as simple as exporting the Information Point REST-URL from AppDynamics and adding it to my settings.js file and then creating a view for it in views.js. This manual process will be replaced by simply adding it to the Django admin interface at a later stage and then dynamically generating views.js and settings.js via a django view.

It was also simple enough to extend the interface to get current open cases from JIRA and retrieve current events from our CMS systems.

It is also possible to show our performance data by location on a map layout as shown below:

If you would like to share any applications, plugins or custom reports that utilize AppDynamics data then drop me an email at: appman @ appdynamics (.) com.