3.3 Contracting Issues and Solutions for Agile
One of our government reviewers said, “This is the single biggest barrier to the government
operating like a commercial entity. We take 5 to 10 times longer to execute routine contract
actions.”
A particular management concern for Agile methods in the DoD, especially from the acquisition
side, but also from the execution side, is the selection and implementation of appropriate
contracting vehicles to support the types of practices that successful Agile projects exhibit. Due to
the iterative nature of Agile and its propensity to accept (even welcome) change, many
contracting vehicles present unique challenges for employing Agile methods. A particular issue is
the reporting and milestone requirements often levied against DoD contracts. These may be
especially difficult to meet using Agile methods. For example, one of the interviewed programs
was entering PDR. The biggest issue was to ensure the government review team understood what
was included in PDR and what was not. Requirements assigned to iterations in the future would
not have any design detail available, as opposed to a traditional PDR where everything would
have some type of design available at PDR. Section 4 discusses technical milestones in detail.
Another challenge is to construct any contract type so that it rewards working software and
meaningful milestone review compliance rather than just traditional artifact production. Several of
the interviewees stated that achieving rewards for working software that is balanced with
sufficient documentation was a challenge.
This section is not meant to be an all-inclusive and exhaustive discussion of the contracting issues
associated with employing Agile methods. Rather, it is only a brief introduction to the types of
contracts and some associated issues. Four of the most common contracting vehicles are discussed
in this section, along with comments about what we found in our interviews in relation to the use
of these contracting vehicles. One interviewee specifically stated that any contract type could be
used; some were just easier to work with than others.
3.3.1 Cost Plus Incentive Fee (CPIF)20
CPIF contracts are used for a wide variety of purposes in the DoD. In terms of Agile
development, a CPIF vehicle is a reasonable contracting vehicle to use when (as is often the case
with Agile projects) requirements will be quickly evolving and redirection of the tactical details of
the software is expected. The incentive fees can be constructed around measures and motivators
that support Agile methods, if the acquisition manager is cognizant of the kinds of management
and measurement aspects, as we have described above.
The management team of one of our interviewees credits its CPIF contract structure as a key
success factor. The program delivers multiple releases per year of ever-improving functional
requirements despite having zero software requirements specified in the enabling contracts.
20 For information on contracting vehicles discussed here, as well as other contract types typically used in the
DoD, please refer to http://www.dtc.dla.mil/dsbusiness/Info/contracts1.htm or
http://guidebook.dcma.mil/18/ContRecRevconttypes.htm.
CMU/SEI-2011-TN-002 | 34
This is possible because the contract is a services type contract for software engineering support.
This allows the development team to avoid situations where a contractor is obligated to satisfy a
requirement that has become irrelevant or obsolete. The team can respond to changing needs and
priorities without onerous contracting actions.
3.3.2 Fixed Price (FP)
There are a variety of fixed price contracts: firm fixed price, fixed price incentive, fixed price with
economic price adjustment, fixed price redeterminable, and fixed price level of effort.21 The
government prefers this type of contract because it encourages the contractor to contain costs.
Fixed price contracts are used with Agile projects in industry, but usually after the product
backlog and critical elements for each release are defined. In industry, the contracts are generally
short in time span in comparison to what is typical in the DoD.
3.3.3 Indefinite Delivery Indefinite Quantity(IDIQ))/Delivery Orders
IDIQ/Delivery orders, or task orders, comprise the largest percentage of contract type in terms of
the programs that we interviewed. Some of the benefits were that IDIQ vehicles provided the
most flexibility for adjusting to changing operational needs and provided more opportunities for
close working relationships between the developers and the end-user community they are serving.
Although none of our interviewed programs expressed it exactly this way, IDIQ types of vehicles
can be managed more like a service contract, where the service being provided is on-demand
software evolution, rather than like a product contract. Some of the author team sees this framing
of software development as a productive service for Agile projects in the DoD, one that is
supported by the prevalent use of the IDIQ contract type.
However, just because IDIQ provides flexibility, the government still needs to require an estimate
in cases where a release is mapped to (or the subject of) a task order. The releases closer to being
performed should have better estimates than those further out in the schedule. That is, the
immediate releases will have deliberate estimates and those further out will have planning
estimates. If the estimates change, the contractor needs to provide a reason. Otherwise, the
contractor may not look ahead further than a single increment.
One potential frailty of the IDIQ contracting model, particularly for a large overall program, is the
management of multiple delivery order contracts running in parallel. Even if the art of the
iteration has been mastered by the team(s) involved, there is a potential for the focus of program
management to be limited to each of the delivery order contracts in isolation, particularly if metric
reporting is limited to the scope of individual delivery orders. Make sure that oversight is across
the entire program rather than focused on individual tasks, and keep a continuing focus on quality
and integrity within product management rather than just looking at the project management
detail. While this may seem obvious to the seasoned program manager, remember that your team
is learning how to work with Agile and may need to be reminded of the basics.
21 http://guidebook.dcma.mil/18/ContRecRevconttypes.htm
CMU/SEI-2011-TN-002 | 35
3.3.4 Time and Material (T&M)
T&M contracts are used when it is not possible to estimate accurately the extent or duration of the
work or to anticipate costs with any reasonable degree of confidence.22 From a government
perspective, this type of contract requires significant oversight to assure that the contractor is
performing efficiently and using effective cost control measures. From a contractor perspective, it
fits rather well with the typical Agile planning and estimation process. T&M contracts may be the
most successful of the contracting types discussed here for use with Agile. In fact, many of the
interviewees, both contractor and government, commented that T&M contracts were the easiest to
use.
3.3.5 Contracting Vehicles Summary
Different contracting vehicles can work for Agile methods. Acquirers must be savvy about the
interpretation limits of different contracting regulations that affect the type of contract they are
contemplating. In cases where we saw awareness of differences in the cultural norms of Agile and
traditional acquisition teams, the contract vehicle chosen was not an impediment to getting the
desired results using Agile. In other words, the contracting personnel were aware of how Agile
methods worked: change is part of the norm, not an exception, they knew what would be
delivered when using those methods, and they adjusted their expectations accordingly within the
limits of the contracting rules and regulations.
One of the key management and contracting aspects in the DoD is the structure and requirements
for technical milestones. The next section addresses technical milestones explicitly, within the
context of DoD management and contracting.
22 http://guidebook.dcma.mil/18/ContRecRevconttypes.htm#Time and Material
CMU/SEI-2011-TN-002 | 36
CMU/SEI-2011-TN-002 | 37
4 Technical Milestone Reviews
The DoD 5000 series provides the framework for acquiring systems. This framework includes a
series of technical reviews including but not limited to System Requirements Review (SRR),
System Design Review (SDR), Software Specification Review (SSR), Preliminary Design
Review (PDR), and Critical Design Review (CDR) among others. Historically, documents such as
MIL-STD-1521, Technical Reviews and Audits for Systems, Equipment, and Computer Software,
or the United States Air Force Weapon Systems Software Management Guidebook are used as
guidance for conducting these reviews.23 Some smaller programs do not require this level of
review; however, programs that do require them expect a certain level of documentation and rigor
regardless of the development methodology employed.
These milestone reviews are system reviews. From a systems point of view, some would say these
reviews are not part of software development methodology and that this is an incorrect use of the
terms. For purposes of this report and in the view of the authors, software intensive systems
typically are subjected to PDRs, CDRs, and other reviews. While the overall system may be a
plane, tank, ship, or satellite, the software still must pass the PDR, CDR, and other milestones. If
the system in question is an IT system, then the PDR, CDR, and other reviews apply directly, as
software is the main component of the system (along with hardware to run it on). This section is
aimed at readers who are contemplating using Agile methods and who are subject to the typical
review activities prevalent in DoD acquisitions. The authors hope that this section will provide
some useful guidance on how to approach Agile methods when following traditional technical
milestones.
4.1 Milestone Review Issue
In the SEI report Considerations for Using Agile in DoD Acquisition, the authors stated: “A very
specific acquisition issue and sticking point is that Agile methodology does not accommodate
large capstone events such as Critical Design Review (CDR), which is usually a major, multi-day
event with many smaller technical meetings leading up to it. This approach requires a great deal
of documentation and many technical reviews by the contractor” [Lapham 2010]. The types of
documentation expected at these milestone events are considered high ritual and are not typically
produced when using Agile.24 If the PMO intends to embrace Agile methods then it will need to
determine how to meet the standard milestone criteria for the major milestones reviews,
particularly SRR, SDR, PDR, and CDR. However, Agile can be adapted within many different
types of systems. One of our reviewers said that “the Atlas V heavy launch guidance system is
produced using eXtreme Programming and has all the DoD procurement program events.” The
23 Note that this report does not address the major milestone decision points, i.e., Milestone A, B, or C. Only
technical milestones are addressed here.
24 “High ritual” is a term used within the Agile community often interchangeably with “high ceremony.” Alistair
Cockburn defined ceremony as the amount of precision and the tightness of tolerance in the methodology
[Cockburn 2007]. For instance, plan-driven methods such as waterfall are considered high ritual or ceremony as
they require an extensive amount of documentation and control. Agile methods are generally considered low
ritual as the amount of documentation and control should be “just enough” for the situation.
CMU/SEI-2011-TN-002 | 38
key here is knowing how long or how much effort is required to review the program in question
(an hour or a week).
In addition, according to one of our reviewers who has monitored Agile-based projects from a
government perspective:
[I] found that in projects using an Agile methodology, technical reviews are around delivery
of major capabilities and/or leadership is invited to certain sprint reviews. [I] also found
that technical reviews are not as needed as much because stakeholders buy-in to
stories/development/schedule at the end of every sprint, which we found is more productive
than large reviews every so often.
In order to determine the type of criteria needed for any review, the intent of the review or
purpose must be known. This raises a question: What is the purpose of any technical milestone
review?
4.2 Intent of Technical Milestone Reviews
The Defense Acquisition Guidebook (DAG) says,
Technical Assessment activities measure technical progress and assess both program plans
and requirements. Activities within Technical Assessment include …the conduct of Technical
Reviews (including Preliminary Design Review/Critical Design Review Reports)…A
structured technical review process should demonstrate and confirm completion of required
accomplishments and exit criteria as defined in program and system planning…. Technical
reviews are an important oversight tool that the program manager can use to review and
evaluate the state of the system and the program, redirecting activity if necessary. [Defense
Acquisition University 2011a]
From a software perspective, each of these reviews is used to evaluate progress on and/or review
specific aspects of the proposed technical software solution. Thus, expectations and criteria need
to be created that reflect the level and type of documentation that would be acceptable for those
milestones and yet work within an Agile environment.
4.3 Challenges
The intent of any technical milestone review is for evaluation of progress and/or technical
solution. For PMOs trained and experienced in the traditional acquisition methods, evaluating
program progress and technical solutions follows well established guidelines and regulations.
Very specific documentation is produced to provide the data required to meet the intent of the
technical review as called out in the program specific Contract Data Requirements List (CDRL).
The content of these documents and the entry and exit criteria for each review is well
documented. However, even in traditional acquisitions (using traditional methods), these
documents, exit and entry criteria can be and usually are tailored for the specific program. Since
the documentation output from Agile methods appears to be “light” in comparison to traditional
programs, the tailoring aspects take on additional aspects. Some of the specific challenges for
Agile adoption that we observed during our interviews that must be addressed are
- incentives to collaborate (see Section 4.4.1)
- shared understanding of definitions/key concepts (see Section 4.4.2)
CMU/SEI-2011-TN-002 | 39
- document content—the look and feel may be different but the intent is the same (see Section
4.4.3)
- regulatory language (see Section 4.4.4)
Many of these challenges are not specific to adopting Agile in DoD acquisitions but are also
common to other incremental development approaches such as Rational Unified Process (RUP).
In fact, RUP does address this level of milestone review. The reviews are called Lifecycle
Architecture (LCA) Review and Lifecycle Objectives (LCO) Review. Some potential solutions to
these challenges for Agile adoption are discussed in the following sections.
4.4 Agile Success Depends on Tackling Challenges
The PMO that is adopting or thinking of adopting Agile for an acquisition must set the stage to
allow for the necessary collaboration required between the PMO and other government
stakeholders and between the PMO and the contractor(s). The introduction of Agile into DoD
acquisition can be considered yet another type of acquisition reform. The Defense Science Board
has provided some cautionary words on any novel approaches for acquiring IT:
With so many prior acquisition reform efforts to leverage, any novel approach for acquiring
IT is unlikely to have meaningful impact unless it addresses the barriers that prevented prior
reform efforts from taking root. Perhaps the two most important barriers to address are
experienced proven leadership and incentives (or lack thereof) to alter the behavior of
individuals and organizations. According to the Defense Acquisition Performance
Assessment Panel, ‘… current governance structure does not promote program success—
actually, programs advance in spite of the oversight process rather than because of it.’ This
sentiment was echoed by a defense agency director in characterizing IT acquisition as
hampered by the oversight organizations with little “skin in the Game.” [Defense Science
Board 2009]
4.4.1 Incentives for Acquirers and Contractors to Collaborate
While this topic is relevant to all contracting issues, it is particularly important if the “world of
traditional milestone review” collides with the “world of agile development.” The authors have
observed programs where collaboration was not yet optimal, which resulted in major avoidable
issues when milestone review occurred. Thus, we emphasize them here.
We take the comment “skin in the game” to mean that the parties involved must collaborate.
Thus, some form of incentive to do so must be created. (As the Defense Science Board noted, “the
two most important barriers are leadership and incentives or lack thereof.”). It is key that
leadership convey clear goals, objectives, and vision for people to create internal group
collaboration and move toward success. One of our interviewees likened an Agile program to a
group of technical climbers going up a mountain: you are all roped together and if one of you
falls, you all fall. If the Agile project is thought of in these terms, it is quite easy to have an
incentive to collaborate. The key for DoD programs is to determine some incentive for both
contractor and government personnel that is as imperative as the life and death one is for technical
climbers.
Incentives need to be team based, not individual based. Agile incentives need to cross government
and contractor boundaries as much as is practical, given regulations that must be followed. Each
individual program has a specific environment and corresponding constraints. Even with the
CMU/SEI-2011-TN-002 | 40
typical constraints, the successful programs we interviewed had a program identity, which crossed
the contractor–government boundary. The incentives to collaborate were ingrained in the team
culture with rewards being simple things like lunches or group gatherings for peer recognition.
It is incumbent upon the PMO and the contractor to negotiate the appropriate incentives to
encourage their teams to collaborate. This negotiation needs to consider all the stakeholders that
must participate during the product’s lifecycle, including external stakeholders. The external
stakeholders (i.e., special review panels, other programs that interface with the program, senior
leadership in the Command and OSD, etc.) need to be trained so they understand the methods
being used, including, but not limited to, types of artifacts that are produced and how they relate
to typical program artifacts. Without specific training, serious roadblocks to program progress can
occur. Buy-in occurs with understanding and knowledge of Agile and the value it can bring.
Defense Acquisition University (DAU) is beginning to provide some training on Agile methods
and the associated acquisition processes.
Incentives should not be limited to just the contractor team but also encouraged for the
government team as well. Government teams are often incented to grow their programs and
follow the regulations. However, these actions should not be incented in an Agile environment
unless the users’ needs are met first.
The type of contract could also play into the incentive issue. One of the interviewees using a
T&M contract stated that they had a limited structure for rewards mainly due to the structure of
the contract, which provided limited motivation for the contracting company to provide rewards
as the contract personnel were relegated to the role of body shop. With this said, we observed a
very collaborative environment on this program.
In order to collaborate, all the parties must have a common understanding of terms and key
concepts. According to Alan Shalloway, Agile “has to be cooperative [in] nature and [people] are
not going to be convinced if they don’t want to [be]” [Shalloway 2009]. We have addressed a set
of Agile and DoD terms for which common understanding is required in Section 2.3.1. Some
additional key concepts specific to supporting technical reviews in an Agile setting are discussed
in the following sections.
