Arikah Map

Performance engineering

Performance Engineering is the set of roles, skills, activities, practices, tools, and deliverables applied at every phase of the Systems Development Lifecycle which ensures that a solution will be designed and implemented to meet the non-functional requirements defined for the solution. It may be alternatively referred to as software performance engineering; however since performance engineering encompasses more than just the software, the term performance engineering is preferable.

Performance engineering has become a separate discipline at a number of large corporations, and may be affiliated with the Enterprise Architecture group. It is pervasive, involving people from multiple organizational units; but predominantly within the Information Technology organization. It can be considered a sub discipline of Systems Engineering.


Contents

Performance Engineering Objectives

Performance Engineering Approach

Because this discipline is applied within multiple methodologies, the following activities will occur within differently specified phases. However if the phases of the Rational Unified Process (RUP) are used as a framework, then the activities will occur as follows:

Inception

During this first conceptual phase of a program or project, critical business processes are identified. Typically they are classified as critical based upon revenue value, cost savings, or other assigned business value. This classification is done by the business unit, not the IT organization.

High level risks that may impact system performance are identified and described at this time. An example might be known performance risks for a particular vendor system.

Finally performance activities, roles, and deliverables are identified for the Elaboration phase. Activities and resource loading are incorporated into the Elaboration phase project plans.

Elaboration

During this defining phase, the critical business processes are decomposed to critical use cases. Such use cases will be decomposed further, as needed, to single page (screen) transitions. These are the use cases that will be subjected to script driven performance testing.

The type of requirements that relate to Performance Engineering are the non-functional requirements, or NFR. While a functional requirement relates to what business operations are to be performed, a performance related non-functional requirement will relate to how fast that business operation performs under defined circumstances.

The concept of "defined circumstances" is vital. This will be illustrated by example:

Note the critical differences between the two specifications. The first example provides no conditions. The second clearly identifies the conditions under which the system is to perform. The second example may have a service level agreement, the first should not. The capacity planners and architects can actually design and build a system to meet the criteria for the valid nonfunctional requirement – but not for the invalid one. Testers may build a reliable performance test for the second example, but not for the invalid example.

Each critical use case must have an associated NFR. If, for a given use case, no existing NFR is applicable, a new NFR specific to that use case must be created.

At this point it is suggested that performance modeling be performed using the use case information as input. This may be done using a performance lab, and using prototypes and mockups of the “to be” system; or a vendor provided modeling tool may be used; or even merely a spreadsheet workbook, where each use case is modeled in a single sheet, and a summary sheet is used to provide high level information for all of the use cases.

It is recommended that Unified Modeling Language sequence diagrams be generated at the physical tier level for each use case. The physical tiers are represented by the vertical object columns, and the message communication between the tiers by the horizontal arrows. Timing information should be associated with each horizontal arrow; this should correlate with the performance model.

There are also some performance engineering activities related to performance testing that should be executed in this phase. They include creating a performance test strategy, developing a performance test plan, determining the sizing of test data sets, and identifying performance test scenarios.

The risk document generated in the previous phase is revisited here. A risk mitigation plan is determined for each identified performance risk, and time and cost is determined and documented.

Finally performance activities, roles, and deliverables are identified for the Construction phase. Activities and resource loading are incorporated into the Construction phase project plans. These will be elaborated for each iteration.

Construction

Early in this phase a number of performance tool related activities are required. These include:

Member of the performance engineering practice and the development technical team leads should work together to identify performance-oriented best practices for the development team. Ideally the development organization should already have a body of best practices, but often these do not include or emphasize those best practices that impact system performance.

The concept of application instrumentation should be introduced here with the participation of the IT Monitoring organization. While vendor monitor systems do have performance capabilities, these normally operate at the operating system, network, and server levels; e.g. CPU utilization, memory utilization, disk I/O, and for J2EE servers the JVM performance including garbage collection.

But this type of monitoring does not permit the tracking of use case level performance. To reach this level of monitoring capability, normally requires that the application itself be instrumented. The monitoring group should have specified the requirements in a previous phase, and should work with the development team to ensure that use case level monitoring is built in.

The group responsible for infrastructural performance tuning should have an established "base model" checklist to tune the operating systems, network, servers (application, web, database, load balancer, etc.), and any message queueing software. Then as the performance test team starts to gather data, they should commence tuning the environment more specifically for the system to be deployed. This requires the active support of subject matter experts, for example, database tuning normally requires a DBA who has special skills in that area.

The performance test team normally does not execute performance tests in the development environment, but rather in a specialized pre-deployment environment that is configured to be as close as possible to the planned production environment. This team will execute performance testing against test cases, validating that the critical use cases conform to the specified non-functional requirements. The team will execute load testing against a normally expected (median) load as well as a peak load. They will often run stress tests that will identify the system bottlenecks. The data gathered, and the analyses, will be fed back to the group that does performance tuning. Where necessary, the system will be tuned to bring nonconforming tests into conformance with the non-functional requirements.

If performance engineering has been properly applied at each iteration and phase of the project to this point, hopefully this will be sufficient to enable the system to receive performance certification. However, if for some reason (perhaps proper performance engineering working practices were not applied) there are tests that cannot be tuned into compliance, then it will be necessary to return portions of the system to development for refactoring. In some cases the problem can be resolved with additional hardware, but adding more hardware leads quickly to diminishing returns.

For example: suppose a portion of a program can be sped up by parallelizing it. Suppose we can improve 70% of a module by parallelizing it, and go from 1 to 4 CPUs. If F is the fraction of a calculation that is sequential, and (1-F) is the fraction that can be parallelized, then the maximum speedup that can be achieved by using P processors is: 1/(.3+(1-.3)/4)=2.105. If we go to 8 processors we get 1/(.3+(1-.3)/8)=2.581.

Throwing more CPU’s at the problem helps a little, but since adding more CPU’s causes more memory utilization and more I/O traffic we may not even get that increase. Instead we might get a decrease by causing a bottleneck traffic jam. So, more money for diminishing returns. Not optimal.

Transition

During this final phase the system is deployed to the production environment. A number of preparatory steps are required. These include:

Once the new system is deployed, ongoing operations pick up performance activities, including:

Articles and Papers

Testing Tools


See also

Organizations

Categories


Information technology management | Software development process

Find

Find

Find