AppDynamics Enhances Pivotal Cloud Foundry Performance Monitoring with New Infrastructure View

AppDynamics’ strong commitment to Pivotal Cloud Foundry’s opinionated platform philosophy makes it easy to add our industry-leading APM to your PCF applications. Our new Service Broker Tile, available on PivNet, is the latest example of our commitment to the most advanced Pivotal architecture.

The tile was completely rewritten to support all the latest Pivotal Service Broker APIs, and now adds support for Cloud Foundry’s Loggregator System for detailed metrics on the health of the underlying infrastructure.

Full PCF Support

Many AppDynamics customers are moving to PCF 2.x to take advantage of the new platform.

We’ve refreshed our implementation of the tile—not only to support PCF versions 2.0 and 2.1, but also to make the tile lightweight, faster to install, and easier to use.

Using our open extensibility framework, we’ve developed a custom nozzle to pull all the important KPI metrics from the Logregator System—and we’re packaging this up as a BOSH Release for tight integration with PCF.

Our Dashboard and PCF Metrics

Administrators need an easy way to visualize—and receive alerts on—the state of their PCF environments. Unfortunately, it’s often difficult to get a clear view of these key metrics. The PCF KPI Dashboard provides that clear view.

Pivotal has very good documentation on the metrics they provide. These metrics are split between performance indicators and capacity scaling indicators. Our new dashboard encapsulates all of these metrics, providing a comprehensive view in a single pane of glass.

At the lowest level, there are health- and resource-related metrics for each of the services running. At a higher level, the KPIs focus on monitoring capacity.

The KPIs we show in our dashboard are from Pivotal’s recommended list. We ingest and publish these KPIs to the AppDynamics Controller.

PCF KPI Dashboard: An Overview

Our dashboard works with Pivotal’s comprehensive set of recommended metrics. Here are some examples:

  • Diego cell metrics: remaining memory available (cell memory chunk and overall remaining memory), remaining disk available, cell rep time to sync, unhealthy cells.

  • GoRouter: router file descriptors, router exhausted connections, router throughput.

  • System (BOSH): VM health, VM memory used, VM disk used.

Here’s a sample PCF KPI Dashboard:

  • The top-left section shows Key Capacity Scaling Indicators, including Diego Cell memory capacity, disk capacity, container capacity, and so on.

  • Top-center to right: Key Capacity Scaling Indicators are plugged into widgets, giving administrators an easy-to-visualize, graphical representation of the state of their PCF environment.

  • The middle section shows a series of KPIs, plus a widget for the current total number of routes registered with the Gorouter.

  • Near the bottom, we see the services that make up the PCF environment, including core metrics for CPU, memory, disk, etc.

For more details on PCF KPI Dashboard installation and configuration, go to https://docs.pivotal.io/partners/appdynamics/index.html  

The Loggregator System

The Loggregator gathers and streams logs and metrics from user apps in a Cloud Foundry (CF) deployment, as well as metrics from CF components.

AppDynamics was one of the first vendors to come out with a Service Broker tile to allow easy configuration of APM agents with Cloud Foundry deployments. Today, our tile is one of the most downloaded extensions from the Pivotal Network. Now we’ve completely refreshed the implementation and replaced the older JMX Bridge, a more limited way of extracting metrics from a PCF system.

Our new tile uses the Loggregator, showing our commitment to the latest PCF platform enhancements.

The Loggregator offers many advantages. Administrators, for example, can access the Loggregator Firehose, the combined stream of logs from all apps, as well as the metrics data from CF components.

If you’re not familiar with the Firehose, it’s a WebSocket endpoint that streams all event data coming from a CF deployment. This data stream includes logs, HTTP events and container metrics from all applications, and metrics from CF system components.

Custom Nozzle

Operators can deploy nozzles to the Firehose. A nozzle is a component that monitors the Firehose for specified events and metrics, and streams this data to external services.

We’ve built our own nozzle—a standard AppDynamics extension packaged as part of the Service Broker Tile—enabling us to pull metrics from the Firehose. This nozzle gives us access to all of the main metrics that administrators need to monitor a PCF system.

Our new Service Broker Tile, with its powerful dashboard, faster performance, and Loggregator support, is the latest example of AppDynamics’ strong commitment to the Pivotal Cloud Foundry platform. Download it today!

Mark Prichard, co-author, is Senior Director of Product Management at AppDynamics.

The AppD Approach: Pivotal Cloud Foundry Performance Monitoring

One of the core strengths of Pivotal Cloud Foundry is its well-established, consistent and practical method of deploying applications. PCF provides the same user experience when deployed over different IaaS layers, and the same developer experience regardless of application language. AppDynamics adheres to this opinionated approach, making it simpler for our customers to monitor their apps on the PCF platform.

(For a brief history of PCF and AppDynamics’ commitment to Pivotal Cloud Foundry performance monitoring, read my earlier blog.)

Enabling AppDynamics’ best-in-class APM on PCF is a two-step process. You must:

  1. Deploy your application with a PCF buildpack, which provides the framework and runtime support for your application.

  2. Use the AppDynamics Service Broker, which enables third parties like AppDynamics to configure plug-in services used by the buildpack.

Pivotal’s opinionated approach offers many advantages. Say you’re a Java developer who’s building an app and deploying it in staging environments. But things aren’t running smoothly: You’re encountering performance issues and getting negative feedback from beta users. Unsurprisingly, you decide to try APM to locate the source of the problem. But rather than make significant changes to the app to accommodate an APM agent, you simply install the AppDynamics Service Broker Tile (more on this below), enter the config parameters, and redeploy the app. It’s really that simple.

Using the Buildpack

The PCF buildpack examines your apps to determine which dependencies to download, and how to configure the apps to interact with bound services. When you push up an app, you don’t have to write fabric support code to access Pivotal services and those of its partners. Rather, the framework provided by the buildpack does this work for you. (See Pivotal’s documentation for more details.)

Let’s use another Java example. (We’ve chosen the Java buildpack because it’s the most widely used with the longest history. Many AppDynamics customers use it, and our support for it is first class.) Your Java application—for example, a web app or JAR file—goes into the Java buildpack. The Cloud Foundry infrastructure handles how the app interacts with the BOSH system layer, as well as services such as logging and deployment. This article shows how to use AppDynamics with Java applications on PCF.

Service Broker Tile

The PCF tile for AppDynamics installs the AppDynamics Service Broker as an application. As the image below shows, the app does appear as a “tile” in the PCF Ops Manager, hence the name:

Essentially, the Service Broker provides a consistent way for third parties like AppDynamics to configure plug-in services that get consumed in the buildpack. These services become part of the “service fabric” that applications can use. In addition to APM, other examples include queuing services, MySQL, and many others.

The AppDynamics Service Broker plugs into this fabric to become a integral component of the Pivotal platform. The Service Broker Tile automates the BOSH deployment and configuration of the AppDynamics agents and controller. It’s a central point for entering your configuration information.

The Service Broker Tile registers the AppDynamics app as a service broker on Cloud Foundry, displays its service plans on the PCF Marketplace, and installs the Machine Agent extension, a separate application that ingests management metrics and shows them on the AppDynamics dashboard. PCF also attaches AppD’s Cloud Foundry performance monitoring agent to your applications.

New, Lightweight Service Broker Tile

We recently added a new version of the Service Broker Tile, which you can find on PivNet.  Many of our customers are moving to PCF 2.x to take advantage of the new platform, and we took the opportunity to refresh our implementation of the tile, not just to support PCF versions 2.0 and 2.1, but also to make the tile lightweight, faster to install, and easier to use. The new version of the tile was rewritten using Python, and supports the very latest Pivotal Service Broker APIs. The Pivotal platform evolves rapidly, and so we removed some older APIs that aren’t relevant any more: the result is a leaner, faster version of the tile that lets you get your applications configured and deployed more easily.

Future Major Enhancements

By adhering to Pivotal’s opinionated platform philosophy, AppDynamics makes it easy to add APM to your PCF applications. In the future, we will greatly expand our Pivotal Cloud Foundry performance monitoring capabilities, adding new metrics and dashboards that provide more insights than ever before. The next blog in this series will take a closer look at these enhancements.

Do you need more robust APM to effectively manage your Pivotal applications? To learn more about AppDynamics Service Broker for PCF, explore our product overview page on the Pivotal Network site.

Future features and functionality are subject to change at the sole discretion of AppDynamics, and AppDynamics will have no liability for delay in the delivery or failure to deliver any of the features and functionality set forth in this document.

Introducing Pivotal Cloud Foundry

You may be wondering why we’ve chosen to write an introductory blog on Pivotal Cloud Foundry (PCF). The platform, after all, has been around for a good while. The answer is simple: many of our largest customers have chosen to use Cloud Foundry, particularly PCF, as their strategic platform for multi-cloud deployment. With so many organizations already using AppDynamics to monitor their critical business applications running on Cloud Foundry, we thought it was high time we took a deeper look at the integration between AppDynamics and CF, and at some of the new initiatives we’re collaborating on. We’ll get into details in the next blogs in this series. But for now, if you’re new to Cloud Foundry, read on.

It’s hard to overstate the importance of cloud computing in the evolution of IT. Cloud platforms enable an organization to quickly deploy network apps and services that scale easily. Migration and build-out tasks that once took weeks, or even months, now take just a few minutes. And rather than focusing on the underlying infrastructure, PaaS frees you to focus on your applications and data.

Cloud Foundry (CF) is an open source, ready-to-use platform that allows your company to get up and running quickly in the cloud. Originally developed in-house at VMware, CF made its public debut in April 2011. A year later came BOSH—an open source tool for deploying Cloud Foundry PaaS—that unifies release engineering, deployment and life-cycle management of large-scale distributed services.

Why Cloud Foundry?

Enterprises today want the flexibility, ease of deployment, consistent management philosophy and framework of PaaS, but often need these capabilities in an on-premises environment. With Cloud Foundry, your organization can run applications on its own computing infrastructure, or deploy them on an infrastructure-as-a-service platform (IaaS) such as Amazon Web Services, Microsoft Azure, vSphere or OpenStack. Alternatively, you could use a PaaS deployed by a commercial CF provider.

There are myriad reasons why companies need to run specific apps within their data centers. A recent IDG/Sapho study shows that the vast majority of enterprise respondents (78%) plan to upgrade their applications in 2018, an effort driven largely by an inability to move cloud-based apps for reasons of security, compliance, and other factors (e.g., apps that are too ingrained in daily processes). A large number of AppDynamics’ customers, for instance, have committed to PaaS, which often involves moving to an internal cloud.

Today, CF is available from either the Cloud Foundry Foundation as open source software, or from multiple commercial providers as a product or service. It’s a good choice for enterprises that want to reduce the cost and complexity of configuring infrastructure for their apps. Developers can use existing tools— with zero modification to their code—to deploy apps to CF.

Highly Opinionated

Cloud Foundry is a prime example of an opinionated platform, one designed to operate consistently across environments, with every feature working as intended “out of the box,” notes O’Reilly. For example, CF provides the same user experience when deployed over different IaaS layers, and the same developer experience regardless of application language.

Many multinational corporations, including Ford and Gap Inc., have used Cloud Foundry to launch cloud projects and modernize their internal software development processes, reports GeekWire. AppDynamics has numerous Fortune 500 customers in major verticals—including banking, financial services, telecom and technology—that have been using Cloud Foundry successfully for years.

Enter Pivotal

Pivotal, a tech company backed by notable investors such as EMC, General Electric and VMware, launched a commercial version of Cloud Foundry in 2013. In fact, several large AppDynamics customers were the driving force behind the Pivotal project. Pivotal Cloud Foundry (PCF) provides extra functionality not included in the open software version, such as additional tools for installation and administration, enterprise services, support, documentation and certificates.

PCF is deployable on-premises and on many cloud providers, offering enterprises a hybrid and multi-cloud platform. It has many high-profile corporate users, including Allstate’s CompoZed Labs unit, which uses PCF to speed development of its new software services, and to build applications that run both on its internal servers and on third-party public cloud providers.

AppDynamics has long been an active supporter of the Cloud Foundry platform—both Pivotal and open source—and will continue to do so. Many of our largest customers have been successfully running large-scale CF deployments for years. In our next PCF blog, we’ll talk about challenges facing PCF users, and take a closer look at our expanding efforts in the Cloud Foundry ecosystem.

Getting Started with Pivotal Cloud Foundry and AppDynamics

This post originally appeared on Pivotal’s blog

The primary objective of a platform should be to provide a high-level of automation. This provides easy management of applications and services, while delivering consistent and error-free deployment of applications. While this high level automation provides a critical foundation, additional specialized services can be added to increase manageability of the applications deployed on the platform. To assist operators in this pursuit the Pivotal Cloud Foundry platform provides a number of integrated services out-of-the-box, including AppDynamics, New Relic, and CloudBees Jenkins. This blog will focus on the “out-of-the-box” integration between Pivotal Cloud Foundry and AppDynamics.

Pivotal Cloud Foundry Buildpacks and Agent Integration

A buildpack in Pivotal Cloud Foundry provides the framework and runtime support for your application. As applications are deployed or “pushed”, the platform detects the language and includes the appropriate versions of the runtime, as well as any additional application infrastructure. The buildpack also contains the AppDynamics agent, with numerous versions to support different deployments. To successfully deploy the agent we need to provide the proper configuration based on the environment during the deployment.

To create the proper configuration a User Provided Service (UPS) is created. A UPS is an external service created for an organization and space and is bound to an application in Pivotal Cloud Foundry. Once the service is bound to the application and the application is restaged, a process to prepare an application for deployment, the AppDynamics agent has the proper configuration to register with the AppDynamics controller and to transmit metrics to the designated endpoint.

A common practice in AppDynamics is to create a logical model representing different layers of an application. AppDynamics uses the application name, tier name and node name to align the deployed applications with the AppDynamics logical model. The Pivotal Cloud Foundry platform, specifically the buildpacks and UPS, provide the mechanism to align deployed applications to the logical model.

The diagram below depicts the Spring Trader application deployed on Pivotal Cloud Foundry. Spring Trader includes three Spring-based applications that are designed to represent a simple trading platform. The deployment artifacts include a web UI component written in Spring MVC, a RESTful API component that provides portfolio and trade data via a persistent store and RabbitMQ, and an asynchronous service for quote generation. Each of these applications are deployed into their own container, and each have their own UPS to define a tier within AppDynamics. This deployment aligns the applications to the AppDynamics logical model using the application and tier name.

image04

To establish a deeper understanding let’s walk through the process and commands required to create the user-provided service and applications on Pivotal Cloud Foundry.

Configure and Push Applications onto Pivotal Cloud Foundry

Before we can “cf push” the application, we need to create a user-provided service in the organization and space where we deploy our applications. In the example below, we use the Cloud Foundry CLI to create the user-provided service providing the required information in name/value format. Notice the application and tier name are configured, and we have created three different user-provided services to represent different tiers in our application.

cf cups app-dynamics-ST-web -p '{"account-name":"myacct","account-access-key":"key","host-name":"myacct.saas.appdynamics.com ","port":"80","tier-name":"web","application-name":"SpringTrader"}'

cf cups app-dynamics-ST-rest -p '{"account-name":"myacct","account-access-key":"key","host-name":"myacct.saas.appdynamics.com ","port":"80","tier-name":"rest","application-name":"SpringTrader"}'

cf cups app-dynamics-ST-asynch -p '{"account-name":"myacct","account-access-key":"key","host-name":"myacct.saas.appdynamics.com ","port":"80","tier-name":"asynch","application-name":"SpringTrader"}'

With our services defined, let’s deploy the Spring Trader application. Using the command line, issue the following commands:

## Create required managed services for deployment
cf create-service p-rabbitmq standard tradermessaging
cf create-service p-mysql 100mb-dev tradersql

## Deploying Rest component, binding services and restaging application
### Note the -b option to use the latest Java buildpack from github.com/cloudfoundry/java-buildpack
cf push -p dist/spring-nanotrader-services-1.0.1.BUILD-SNAPSHOT.war -b https://github.com/cloudfoundry/java-buildpack.git --no-start stfront
### Binds Services
cf bind-service stfront tradersql
cf bind-service stfront tradermessaging
cf bind-service stfront app-dynamics-ST-rest
### Specify which JVM version to use in the Java Buildpack
cf set-env stfront JBP_CONFIG_OPEN_JDK_JRE '[version: 1.7.0_+]'
### Specify which AppDynamics agent to use in the Java Buildpack
cf set-env stfront JBP_CONFIG_APP_DYNAMICS_AGENT '[version: 4.0.1_+]'
cf restage stfront.

## Deploying Web UI component, binding services and restaging application.
cf push -p dist/spring-nanotrader-web-1.0.1.BUILD-SNAPSHOT.war -b https://github.com/cloudfoundry/java-buildpack.git --no-start stweb
### Bind services
cf bind-service stweb app-dynamics-ST-web
cf set-env stweb JBP_CONFIG_OPEN_JDK_JRE '[version: 1.7.0_+]'
cf restage stweb

## Deploying the asynch service component, binding services and restaging application.
cf push -p dist/spring-nanotrader-asynch-services-1.0.1.BUILD-SNAPSHOT.war -b https://github.com/cloudfoundry/java-buildpack.git --no-start stback
### Bind services
cf bind-service stback stsql
cf bind-service stback tradermessaging
cf bind-service stback app-dynamics-ST-asynch
cf set-env stback JBP_CONFIG_OPEN_JDK_JRE '[version: 1.7.0_+]'
cf restage stback

As the “cf restage” command is executed the output will show which pieces are required to create the application and prepare it for the container. In the example below, you’ll see the AppDynamics agent is added to the container along with OpenJDK and the Tomcat server. If the AppDynamics agent is not included, verify the user-provider service is properly configured and its bound to the application.

image05

Once the commands are complete you should see the applications deployed in Pivotal Cloud Foundry Apps Manager. You should also see the user-provided services, with an application bound to each service. A number of applications can be bound to the user-provided service, but in our example its one per service.

image01

image08

We’ve successfully deployed our applications and bound AppDynamics services to each application instance. Let’s review the AppDynamics dashboard to understand how our application maps to the AppDynamics logical model.

The AppDynamics Model and Dashboards

In the home dashboard below you’ll notice the Spring Trader application. This application is made up of three independent applications in Pivotal Cloud Foundry.

image00

If we click into the Spring Trader application, we’ll see the app servers and you’ll notice how each application instance uses the tier name to represent different layers of the Spring Trader application. In our example we have the web, rest and asynch tiers. If you expand each tier you’ll see the node names, in this case node name is “0” for the first instance of the application. Node names are configurable, but I prefer to use the default node names as they align with the Pivotal Cloud Foundry application instance sequence number. As we scale different tiers to support new workloads, the new application instances will register with the AppDynamics controller, each with a sequential node name.

image02

By supporting the logical model in Pivotal Cloud Foundry, we can use the application dashboard to review transactions across the different applications.

image06

Troubleshooting AppDynamics Agents in Pivotal Cloud Foundry

During my initial installation, I had misconfigured the user-provided service which prevented the agent from registering with the AppDynamics controller. As a result, I needed to troubleshoot the agent configuration and review its logs. The good news is Pivotal Cloud Foundry makes this extremely easy. To access the agent logs, or any other buildpack information, just issue a “cf files stfront” command similar to the image below replacing “stfront” with your application name.

image07

The logs are located on app/.java-buildpack/app_dynamics_agent/ver4.0.1.0/logs and each instance of your application will have its own log. At the top of the agent log you can find AppDynamic JVM arguments used to configure the agent. If you configure application security groups in Pivotal Cloud Foundry you’ll need to modify the policy to allow egress traffic for the controller port. In the example below, you can see all logs available for review and each file can be output to the screen by issuing the “cf files” command with a full path to the file.

“cf files stfront app/.java-buildpack/app_dynamics_agent/ver4.0.1.0/logs/agent.2015_03_31__14_26_55_0.log”

image03

As we’ve shown,  Pivotal Cloud Foundry makes the delivery and management of applications extremely easy and error free. Working with our partners to create an ecosystem of integrated services for customers delivers tremendous value, and provides an elegant transition to the delivery of cloud native applications while preserving and integrating with existing solutions.

Monitoring Apps on the Cloud Foundry PaaS

At AppDynamics, we pride ourselves on making it easier to monitor complex applications. This is why we are excited to announce our partnership with Pivotal to make it easier to deploy built-in application performance monitoring to the cloud.

 

Getting started with Pivotal’s Cloud Foundry Web Service

Cloud Foundry is the open platform as a service, developed and operated by Pivotal. You can deploy applications to the hosted Pivotal Web Services (much like you host apps on Heroku) or you can run your own Cloud Foundry PaaS on premise using Pivotal CF. Naturally, Cloud Foundry is an open platform that is used and operated by many companies and service providers.

1) Sign up for a Pivotal CF account and AppDynamics Pro SaaS account

In the future, Pivotal Web Services will include the AppDynamics SaaS APM services, so you’ll only need to sign up for Pivotal Web Services and it will automatically create an AppDynamics account.

2) Download the Cloud Foundry CLI (Command Line Interface)

Pivotal Web Services has both a web based GUI as well as a full featured linux style command line interface (CLI). Once you have a PWS account, you can download a Mac, Windows or Unix CLI from the “Tools” tab in the PWS dashboard or directly for OSX, Linux, and Windows.

Pivotal Web Services CLI

3) Sign in with your Pivotal credentials

Using the CLI, log in to your Pivotal Web Services account. Remember to preface all commands given to Cloud Foundry with “cf”.  Individual Cloud Foundry PaaS clouds are identified by their domain API endpoint. For PWS, the endpoint is api.run.pivotal.io. The system will automatically target your default org (you can change this later) and ask you to select a space (a space is similar to a project or folder where you can keep a collection of app(s).

$ cf login

Cloud Foundry CLI 

Monitoring Cloud Foundry apps on Pivotal Web Services

Cloud Foundry uses a flexible approach called buildpacks to dynamically assemble and configure a complete runtime environment for executing a particular class of applications. Rather than specifying how to run applications, your developers can rely on buildpacks to detect, download and configure the appropriate runtimes, containers and libraries. The AppDynamics agent is built-in to the Java buildpack for easy instrumentation so if you have AppDynamics monitoring running, the Cloud Foundry DEA will auto-detect the service and enable the agent in the buildpack. If you start the AppDynamics monitoring for an app already running, just restart the app and the DEA will autodetect the new service.

1) Clone the Spring Trader demo application

The sample Spring Trader app is provided by Pivotal as a demonstration. We’ll use it to show how monitoring works. First git clone the app from the Github repository.

$ git clone https://github.com/cloudfoundry-samples/rabbitmq-cloudfoundry-samples

2) Create a user provided service to auto-discover the AppDynamics agent

$ cf create-user-provided-service demo-app-dynamics-agent -p “host-name,port,ssl-enabled,account-name,account-access-key”

Cloud Foundry CLI

Find out more about deploying on PWS in the Java buildpack docs.

3) Use the Pivotal Web Services add-on marketplace to add a cloud based AMQP + PostgreSQL database instance

$ cf create-service elephantsql turtle demo-db

$ cf create-service cloudamqp lemur demo-amqp

Cloud Foundry CLI

4) Bind PostgreSQL, AMQP, and AppDynamics services to app

$ git clone https://github.com/cloudfoundry-samples/rabbitmq-cloudfoundry-samples

$ cd rabbitmq-cloudfoundry-samples/spring

$ mvn package

$ cf bind-service demo-app demo-app-dynamics-agent

$ cf bind-service demo-app demo-amqp

$ cf bind-service demo-app demo-db

Cloud Foundry CLI

5) Push the app to production using the Cloud Foundry CLI (Command Line Interface)

$ cf push demo-app -i 1 -m 512M -n demo-app -p target/rabbitmq-spring-1.0-SNAPSHOT.war

Cloud Foundry CLI

Spring AMQP Stocks Demo App

Spring Trader

Pivotal Web Services Console

Pivotal PaaS CloudFoundry

 

 

Production monitoring with AppDynamics Pro

Monitor your critical cloud-based applications with AppDynamics Pro for code level visibility into application performance problems.

AppD Dashboard

Pivotal is the proud sponsor of Spring and the related open-source JVM technologies Groovy and Grails. Spring helps development teams build simple, portable, fast, and flexible JVM-based systems and applications. Spring is the most popular application development framework for enterprise Java. AppDynamics Java agent supports the latest Spring framework and Groovy natively. Monitor the entire Pivotal stack including TC server and Web Server, GreenPlum, RabbitMQ, and the popular Spring framework:

AppD

 

Take five minutes to get complete visibility into the performance of your production applications with AppDynamics today.