EssayBark -Excellent Writing Services

    4.4.2 Shared Understanding of Definitions/Key Concepts

While there are numerous key concepts involved with Agile development methods, the two we

heard most about during our interviews were type and amount of documentation and technical

reviews.

One of the key concepts related to Agile is the type and amount of documentation that is

produced. Typically, Agile teams produce just enough documentation to allow the team to move

forward. Regulatory documentation is still completed and may be done in a variety of ways. In

our interviews, we saw several different approaches to producing regulatory documentation that

was not considered to contribute directly to the development activities, but will be needed in the

future (user manuals, maintenance data) or to meet programmatic requirements:

• A SETA contractor was hired by the program office to review the repository of development

information (embedded in a tool that supported Agile methods) and produce required

documentation from it.

CMU/SEI-2011-TN-002 | 41

• Technical writers embedded with the Agile team produced documentation in parallel with

the development activities.

• Contractor personnel doing program controls activities produced required documentation

toward the end of each release. The contractor program control personnel took the outputs

from the Agile process and formatted them to meet the 5000 required documents.

Note that if the documentation is not of any value to the overall product or program, then waivers

should be sought to eliminate the need.

Another key concept is that the whole point of using Agile is to reduce risk by defining the

highest value functions to be demonstrated as early as possible. This means that technical reviews

like PDR and CDR will be used to validate early versions of working software and to review

whatever documentation had been agreed to in the acquisition strategy. Thus, additional

documentation would not be created just to meet the traditional set of documents since that set

would be tailored a priori.

4.4.3 Addressing Expected Document Content Mismatch

As stated above in Section 4.2, “A structured technical review process should demonstrate and

confirm completion of required accomplishments and exit criteria as defined in program and

system planning.” Thus, the documents produced for any of these reviews should support the

required accomplishments and exit criteria that were defined in the program and system planning,

and reflected in the acquisition strategy chosen by the program.

As stated in Section 2.5, Agile planning is done at multiple levels, similar to a rolling-wave style

of planning, though usually in smaller increments than is typical of rolling waves. The top-level

overview of the system and architecture is defined and all releases and associated iterations (or

sprints) are generally identified up front. The releases and associated iterations to be undertaken

soon have more detail than those to be undertaken later—the further in the future, the less detail.

Given this approach to planning, how does document production fit it? Typically using traditional

methods, the same level of detail is provided for all sections of a document and then refined as

time progresses. This approach does not work for programs employing Agile methods. Thus, the

way documents are created will need to evolve just as the methods for development are evolving.

Future methods could employ tools that examine the code to extract information to answer

predefined and ad hoc questions.

For today’s Agile-developed work, because the work is being accomplished using an incremental

approach, one should expect the documentation will also be accomplished in that manner.

However, as discussed in Section 4.4.2, several of the programs interviewed have used varying

approaches to creating the documentation. None of the approaches are better than the others; they

are just tailored for the specific situation found within the program.

The intent and content of each artifact should be considered and agreed upon by both the PMO

and the developer. The current Data Item Descriptions (DIDs) applied to the contract can be used

as a starting point for these discussions. Once the intent and content are determined, appropriate

entry and exit criteria can be created for each document. Keep in mind the timing of the releases

and iterations when setting the entry and exit criteria so that the content aligns with the work

CMU/SEI-2011-TN-002 | 42

being performed. Each program we interviewed took slightly different approaches to creating

documentation. In all cases, the content and acceptance criteria were determined with the PMO.

4.4.4 Addressing Regulatory Language

While we do not know of any regulations that expressly preclude or limit the use of Agile, many

in the acquisition community seem to fear the use of Agile because of their prior interpretations of

the regulations [Lapham 2010]. Thus, care should be taken to ensure that all regulations are met

and corresponding documentation is produced. At the very least, there should be traceability from

the Agile-produced documentation to the required documentation. Traditional documentation as

defined by CDRLs follow well established DIDs. Agile methods do produce documentation;

however, the form is not the same as that prescribed in the DIDs, although the content may be the

same. This difference tends to spark issues about documentation unless the contractor and the

government PMO agree on some type of mapping. If needed, waivers can be requested to reduce

or eliminate some of the regulatory implications.

Many of the current regulatory practices are at the discretion of the Milestone Decision Authority

(MDA). In order for Agile to succeed, the MDA will need to be supportive of Agile processes. As

one of our interviewees pointed out, ideally more responsibility would be delegated to PMs as

they are more closely associated with Agile teams. In fact, this issue was one that a particular

program continually ran into.

There are several endeavors underway to help define a more iterative or incremental acquisition

approach for the DoD. These include the IT Acquisition Reform Task Force, which is responding

to Section 804.25 In addition, a smaller white paper effort is underway in response to Secretary

Gates’s solicitation for transformative ideas to improve DoD processes and performance, titled

Innovation for New Value, Efficiency & Savings Tomorrow ( INVEST).26 The white paper

submitted to INVEST originated within the Patriot Excalibur program at Elgin Air Force Base

and suggests specific language changes to some regulations to make it easier for those choosing

an acquisition strategy to understand how they could effectively use Agile methods.

4.5 Potential Solutions for Technical Reviews

Many DoD programs require technical reviews; however, there is very little written about

accomplishing these types of reviews when employing Agile methods. Written guidance on

performing such Agile-related reviews is beginning to emerge at a high level (e.g., the 804

response paper), and several programs have done or are doing these reviews. The information

presented here is based on many interviews performed during the research for this technical note

and on the ideas of the author team, which arose during spirited debates.

First, the underpinning for any solution has to be developed. This is the agreement the PMO and

developer have made about how to perform their reviews. The reviews are planned based on the

program and system planning. Given the incremental nature of Agile work, the most productive

 

25 http://www.afei.org/WorkingGroups/section804tf/Pages/default.aspx

26 http://www.defense.gov/home/features/2010/0710_invest/. At least one of the programs we interviewed

submitted a paper to this contest sponsored by Secretary of Defense Gates.

CMU/SEI-2011-TN-002 | 43

approach to technical reviews will be incremental in nature also. We recognize that an

incremental approach to reviews may not always be possible.

Our interviews showed that there tend to be two ways to accomplish the technical reviews:

1. Traditional technical reviews, like those that have existed on “waterfall” programs, have

been added to an Agile process.

2. Technical reviews are an integral part of the Agile process and as such are a series of

progressive technical reviews.

4.5.1 Traditional Technical Reviews

Where Agile methods are being used in a program to develop software, we saw two common

approaches to dealing with technical milestone reviews:

• The program proceeded as though the software was being developed via waterfall as

traditionally practiced, and the developer was responsible for ensuring that they produced the

expected documentation and satisfied the entry and exit criteria.

• The program office or developer performed necessary reviews based on the Agile methods in

use and then translated the Agile documentation into the expected artifacts for the review.

In either case, a translation step exists. In some instances, the developers are performing the

translation internal to their program and presenting the standard artifacts one normally sees at the

technical reviews. This approach has been called covert Agile. The developer and presumably the

acquirer are still getting some of the risk reduction benefits from employing Agile but may be

losing some of the potential cost savings due to creating the translation layer.

The other form of traditional technical reviews is one where the PMO is fully aware of the Agile

method being employed by the development contractor. In this instance, the PMO does not want

its developers dealing with the translation but rather hires a Systems Engineering and Technical

Assistance (SETA) contractor to take the “raw” data produced from the Agile method and uses it

to produce the required artifacts.

4.5.2 Progressive Technical Reviews

Progressive technical reviews use each successive wave of reviews to build on its predecessors.

What does this mean? The program has only one PDR, CDR, etc., of record. However, when

Agile is employed, the complete set of information for all requirements that is normally required

for a PDR will not be available at that time. Thus, the available information will be used for the

PDR with the understanding that multiple “mini” or progressive PDRs will occur as more

information becomes available. One program we worked with is using this approach. Another

program uses mini-PDRs and CDRs where they obtain signoff by the customer representatives.

This is used to prove that the customer signed off, saying “they liked it.” This type of sign-off can

be used to document performance throughout the program. A third program calls these

Incremental Design Reviews (IDRs). Currently, SDR is the only milestone besides IDRs that they

use. At SDR, they establish the vision, roadmap, and release plan for the remainder of the project

(which spans multiple contracts). At the end of each release, they have an IDR. Each IDR will

deliver threads of functionality defined at SDR.

CMU/SEI-2011-TN-002 | 44

Some of the information available for the PDR will be far more detailed, as it will represent

completed functionality from the iterations that are already finished. For example, in Figure 8 the

relationship of a PDR to a release and its associated sprints is shown. This notional diagram

shows that the PDR is programmatically scheduled for June and the CDR is scheduled for

December (milestone timeline). The iteration timeline shows all iterations. Iterations 1-5 are

completed by June. Thus, there are five iterations completed by the PDR date. The PDR would

include data from iterations 1-5.

The PDR27 also needs to demo a working, high-level architecture for the entire program

implementing a critical set of functionality from the iterations completed by the PDR. This

provides a demonstrable risk-reduction activity. In addition, PDR should also show how the

requirements have been tentatively allocated to each release. For releases nearer in time (i.e.,

those closer to execution) the requirements will have been allocated to individual sprints. The

PDR should also address at a high level the external interfaces and the infrastructure for the

program. This information is used to provide context and enable the overall review.

The PMO should consider holding CDRs for each release, which would ensure the next level of

detail is available and that the highest value capabilities are done or broken down to a level that is

ready for implementation.

Figure 8: Relationship of Sprint to Release to PDR of Record

4.5.3 An Alternate Perspective on Milestones

Another way to look at milestones such as PDR is to see how they align to releases and the

associated iterations. This discussion provides an alternate look at how the milestones could be

achieved.

Each iteration within a release follows a standard process. However, before the iteration starts the

development team and stakeholders determine which of the capabilities (functional or nonfunctional)

will be accomplished within the upcoming iteration. These capabilities have already

 

27 It should be noted that this discussion is redefining the terms PDR and CDR for use with Agile. This could lead

to confusion and contribute to people “acting Agile” as opposed to “being Agile.” Perhaps the milestones should

be renamed. This renaming is left to future work.

CMU/SEI-2011-TN-002 | 45

been defined earlier in the program and listed in a backlog by priority and/or risk-reduction value.

Once a capability is assigned to a release, it is decomposed into features as shown in Figure 9.

Figure 9: Capabilities Decomposed [Schenker 2007]

In this illustration, each feature of the capability is assigned to a different team for development

during the iteration. The feature would be described using stories. The teams would be considered

feature teams with one team singled out to be the capability guardian. Each feature team would

then develop its portion of the overall capability using the process shown in Figure 10.

Figure 10: Development Process Within Each Team

Figure 10 depicts the development process each team uses on individual features during their

iteration. For simplicity of comparison, the diagram looks very much like a waterfall process.

However, all disciplines are represented on the team and they all work on the process together,

with each person contributing in their discipline (analysis, coding, testing, etc.). Typically, the

PMO is invited to participate at each Quality Review (QR) gate. This insight provides more touch

LEGEND

Analyze

Capability

Requirements

Perform

Logical

Analysis

Perform

System

Design

Implement

& Integrate

Components

Verify

System

Activity

Quality

Review

Gate

[Local Approval]

[Local Approval]

[Local Approval]

[Local Approval]

[Success Criteria]

A

B

C

E

F

Prepare

Test

Case [Local Approval]

D

1

Feature

[Planned]

1

Feature

[Defined]

1

Feature

[Analyzed]

1

Feature

[Implemented]

1

Feature

[Ready for Test]

1

Feature

[Verified]

Artifact

[state]

CMU/SEI-2011-TN-002 | 46

points for the acquisition agent, typically the government, to be involved in the progress of the

program. The information gained by attending these QR gates can be used to augment the formal

PDRs, CDRs, etc. The diamond shapes lettered A-F are QR gates which must be passed before

the artifacts from the iteration can be considered done. Each QR can be considered “mini” PDRs,

CDRs, etc. As you can see, analysis, design, implementation and integration, and test are all

accomplished by the team within the time box allowed for the iteration. This type of PDR

iteration would be expected to go beyond two weeks. Two weeks seemed to be a norm that is

common in the interviewed programs that are not using formal technical reviews.

Now that we understand how the team functions, we can return to creating the capability shown in

Figure 9. Capability 117 is decomposed into clear features that can be completed by a team within

one iteration. Each feature is assigned to a team (1-4). This work is described in the work plan or

scoping statement for the individual iteration. The teams get their individual features and proceed

to develop their portion of the capability. Once all features are done, the capability is also

completed.

This discussion of capabilities and features showing quality gates is another way to think about

doing progressive milestones. It shows one possible detailed approach to solving the issue of

technical milestones while using Agile methods.

It should be noted that this discussion does not address the difference between systems

engineering/design and iterative development of the capabilities. These could be two very

different items depending on the type of system. The systems engineering and design effort

essentially feeds the product backlog. Iterative development addresses the creation of the actual

products. The rhythm of each does not need to be the same. Further analysis and discussion of this

area of concern are left to future work.

CMU/SEI-2011-TN-002 | 47

5 Estimating in Agile Acquisition

5.1 Introduction

Estimation activities occur throughout the DoD acquisition life cycle. Estimates are used by all

DoD programs in a variety of ways, and they are generated and processed in a variety of ways.

Books have been written, inside and outside the DoD, on cost estimation for large, complex,

software-intensive programs [Stutzke 2005]. We cannot deal with all the many connections

between DoD estimation practices and Agile estimation in this report. We address the issues that

we have most frequently seen and discussed, in interviews and through reviewers. Also, in all of

the generalizations we make below, it should be understood that the needs and constraints of a

particular program could result in estimates being treated in similar or quite different ways than

what we describe in this section.

Some general estimation activities that government program offices support on many (though

certainly not all) programs include:28

• Producing a Program Life-Cycle Cost Estimate. Prior to Milestones A and B, the

Acquisition Program Office (the government) must develop a program life cycle cost estimate

(PLCCE). The PLCCE is presented to the program’s Milestone Decision Authority (MDA) at

each milestone. The PLCCE must look forward from the current program state to the end of

the system’s life, and assess the cost of the product or system over its entire life.

• Program Monitoring. During source selection, the Acquisition Program Office may want to

gain insight into the manner by which the bidding contractors have prepared their estimates.

During contract execution, the Acquisition Program Office is constantly reviewing the

performance of the contractor with respect to the contractor’s estimate. This is typically done

by reviewing the contractor’s earned value management (EVM) data, although there will be

further opportunities to review the contractor’s estimation process each time an engineering

change is processed.29

• Transition to Sustainment. After the system is fielded and in sustainment, there is generally

a two-year cycle of maintenance, technology refresh, and upgrade.30 These system

enhancements are estimated and budgeted by the relevant program office, and by the

contractor, who may or may not be the same organization that originally developed the

system.

To understand the varied uses of estimates by the program office staff throughout the acquisition

life cycle, and how these uses may relate to the use of an Agile development process by a

contractor, the reader must understand estimation practices in general, and Agile estimation

practices in particular. The next section of this report (5.2) will begin to provide that insight and

 

28 The specifics for what is and is not required are defined in the FAR and DFAR for EV programs.

29 How this occurs and the various ways programs can implement this estimation process is covered in the earned

value management system description (EVMSD) and its work instruction.

30 The timing and nature of sustainment activities is, of course, ultimately dependent on the particular program

contracting and technical characteristics. However, we have observed this pattern in many programs.

CMU/SEI-2011-TN-002 | 48

we will show how a government program office (including the program manager, the staff, the

contracting officer’s technical representative (COTR), and the procurement staff) could take

actions with their cost estimating practices that would enable an Agile acquisition of a new system

or sustainment of an existing system in the DoD.

5.2 Estimating to Support Request for Proposal (RFP) Preparation

The following discussion assumes that the government program office is acquiring software

products (i.e., buying a system through a cost or incentive contract as covered by DFAR 234.201).

We are not discussing acquiring software development capacity (i.e., software development

expertise of a certain capability over a period of time), which is an alternate way we have seen

government software needs being met. This model is more common in sustainment and operations

and maintenance (O&M) programs. It generally consists of determining how many resources you

can afford and how much capability those resources will allow you to build. This model is what

some successful Agile programs have used, but it is not available to all programs.

During the RFP preparation phase of a new system acquisition, the government program office

will make the decisions that are pivotal to enabling or disabling an Agile development contractor

to bid and meet the program’s needs. It is during this phase that the government program office

will prepare its PLCCE, which will be based on the government’s work breakdown structure

(WBS). The prohibition during this timeframe against engaging with the development team when

this is a competitive contract is a significant barrier to establishing the trust that is key to Agile

project success; however, the considerations below could help to mitigate this issue.

If the program office wants to allow a developer using Agile methods to effectively compete,

there are considerations that relate to both the acquisition strategy and its follow-on activities, as

well as considerations related to execution of the Agile methods within the boundaries of the

Program Management Baseline (PMB). From the acquisition perspective, the government

program office must address how typical Agile methods artifacts fit into the traditionally specified

artifacts of an acquisition, for example:

• The acquisition strategy should describe how the program office would interact with its

contractor in order to provide the subject matter expertise needed on a continuous basis

throughout the iterations of an Agile development.

• To ensure that the Agile acquisition strategy is enacted, the statement of work (SOW) or

program work statement (PWS) must include language that allows the program office to

provide subject matter experts with the ability to participate in the development of the

software. This may be complicated by the hierarchical structure of contracts in a large system

acquisition. The program life cycle cost estimate and budget must include funding for these

subject matter experts throughout the development of the system. Because the SMEs usually

come from government operational units, agreements must be crafted (e.g., memorandum of

agreement [MOA], memorandum of understanding [MOU]) that make clear the expectations

of participation of different stakeholders.

• The government program office must have a notional plan for what to do with the interim

product releases that come from an Agile development process. Specifying these in the

SOW is one way to emphasize the importance of working software being available in short

iterations. There should be an evaluation environment established along with a feedback

CMU/SEI-2011-TN-002 | 49

mechanism in place that permits the end-user community to try out these interim releases in a

safe, secure environment, while waiting for required acceptance and certification testing

activities to take place.31

Generally speaking, the most visible element of a software product estimate in DoD programs is

the estimate of product size.32 Even though modern software development tools and techniques

reduce dependence on handcrafted source code, size is still frequently expressed in source lines of

code (SLOC) or in function points. In Agile development environments, the development team

may use “story points” as an alternative to either of these. Story points can be problematic in

acquisition settings accustomed to SLOC or function points because they are explicitly a relative

measure of size, not an absolute measure. Therefore, when story points are used outside of the

team that generated them, it is necessary, though not trivial, to make some translation between

story points and, typically, function points. Some of the programs we interviewed acknowledged

they made the translation from story points to product size to provide cost estimates to those

outside the development team. We saw proprietary tools that address this translation, and the

commercial vendors for estimating tools are starting to address this new market need. In

acquisition settings where trust has already been established between the contractor and the

acquisition program office, this dependence on an absolute, versus relative, measure may be

reduced.

Most parametric cost-estimation models base their outputs on software size, so errors in the size

estimate will propagate into the estimate of effort and schedule. According to the GAO Best

Practices Guide for Estimation, the keys to producing a defensible software cost estimate are (1)

to have a reliable method for estimating the size of the product and (2) to employ a method for

transforming the size estimate into an estimate of cost and schedule demonstrated to be accurate

on similar projects [GAO 2009].

One popular parametric cost-estimation tool is the constructive cost model (COCOMO).

According to Boehm, “COCOMO is an algorithmic-based parametric software cost-estimation

model for estimating a software project as an ‘effort equation,’ which applies a value to tasks

based on the scope of the project (ranging from a small, familiar system to a complex system that

is new to the organization). COCOMO II is the successor of COCOMO 81, incorporating more

contemporary software development processes such as code reuse, use of off-the-shelf software

components, and updated project databases” [Boehm 1981].

At the heart of the COCOMO II model are the cost parameters themselves. These parameters

include five scale factors and seventeen effort multipliers . Scale factors represent areas where

economies of scale may apply. Effort multipliers represent the established cost drivers for

software system development. They are used to adjust the nominal software development effort to

reflect the reality of the current product being developed.

 

31 Note that certification and accreditation (C&A) issues within the DoD acquisition life cycle are currently being

addressed on multiple policy and implementation fronts, all with the goal of reducing the time, usually spent at

the end of a program, to get the software system certified and then accredited by the appropriate governance

body. We are not dealing with the specific requirements of the DIACAP process in this report.

32 Software size may not be the most reliable predictor of software effort and cost (see Capers Jones, for

example, who cites programmer skill as a better predictor of software outcome than size, among other

attributes).

CMU/SEI-2011-TN-002 | 50

It would be reasonable to assert that an Agile development process would have an impact on some

of these parameters. For example, the COCOMO II model includes an effort multiplier for

domain knowledge, or applications experience. The cost estimating multiplier based on the

domain knowledge and capability of the software developer staff is called “application

experience” (APEX). The rating for this cost driver is dependent on the level of applications

experience of the project team developing the software system or subsystem. The ratings are

defined in terms of the project team’s equivalent level of experience with this type of application.

In an Agile development environment, there would be subject matter experts (users) participating

with the system developers. The participation of users in the development process should improve

the domain knowledge of the development team. The magnitude of the improvement can be

assessed by changing the assignment of this effort multiplier, and observing the impact on the

estimate.

A selected list of COCOMO II scale factors and effort multipliers is provided in Appendix D.

Factors listed there that we would expect to be impacted by the use of an Agile development

process include

• the development flexibility factor
• the architecture/risk resolution factor
• the team cohesion factor
• the analyst capability effort multiplier
• the programmer capability effort multiplier
• the application experience effort multiplier

Appendix D also contains information about Agile COCOMO, a 2004 prototype product that

reflects some Agile estimation principles while relating back to concepts familiar to COCOMO

users [Agile COCOMO 2011].

Among the many software estimation tools generally available (including Price-S, Software

Lifecycle Management-Estimate [SLIM], and others) is the Software Evaluation and Estimation

of Resources (SEER) model. It is one of those that actively updates its products to accommodate

Agile estimation.

SEER for Software (SEER-Software Estimation Model [SEM]) is an algorithmic project

management software application designed specifically to estimate, plan, and monitor the

effort and resources required for any type of software development and/or maintenance

project. SEER, which comes from the noun referring to one having the ability to foresee the

future, relies on parametric algorithms, knowledge bases, simulation-based probability, and

historical precedents to allow project managers, engineers, and cost analysts to accurately

estimate a project’s cost schedule, risk and effort before the project is started [SEER-SEM

2011].

For Agile projects, SEER uses three kinds of estimates. These are planning, forecast, and

working. The planning estimate is still used to determine how big the project will be and is

usually based on analogies of previous projects of similar size. The forecast estimate is

accomplished after you have built your backlog. Several things can be defined at this time, such

as incremental delivery, release cycle, the length of the iteration, exit criteria for a deliverable, and

negotiation for scope change requests. (Baseline change requests accomplish this in the DoD

CMU/SEI-2011-TN-002 | 51

acquisition cycle.) Finally, working estimates are done for all iterations after the first iteration is

complete. This allows assessment of the team and customer as well as an understanding of the

individual team velocities.

SEER, like COCOMO, uses a variety of parameters for their model. These parameters include

• requirements formality
• requirements volatility
• personnel capabilities – analyst and programmers
• familiarity with the process
• process maturity
• staffing complexity
• development system volatility
• automated tools usage
• testing level
• quality assurance participation
• infrastructure and tooling costs

Before the build, your estimate considers these parameters in relationship to your team. We

recommend that you revisit your foreast estimate as your team changes.33