Policy for Conducting Operational Analysis

Table of Contents

  1. Purpose
  2. Background
  3. Scope
  4. Policy
    1. Required Content for OA Results Report
    2. OA Results – Determining Next Steps for the IT Investment
    3. Special Considerations for sufficiently complex IT Investments
  5. Roles and Responsibilities
    1. HHS CIO Roles and Responsibilities.
    2. OpDiv Roles and Responsibilities
    3. Investment Level Roles and Responsibilities
  6. Applicable Laws and Guidance
  7. Information and Assistance
  8. Effective Date/Implementation
  9. Approved.
  10. Glossary

Table 1 – Definitions of Terms

Table 2 – List of Acronyms


Purpose

This Policy provides a standard methodology and establishes roles and responsibilities for conducting an Operational Analysis (OA). An OA is required for all Department of Health and Human Services (HHS) information technology (IT) Investments that are currently in the Operations and Maintenance (O&M) or Steady State life cycle phase. This Policy provides detailed instructions regarding the content of OA documentation as well as the roles and responsibilities for conducting and approving an Investment’s OA. Proper completion of an OA for every applicable IT Investment will help ensure that HHS IT Investments are viable, relevant, and remain pertinent to achieving the agency’s overarching mission.

This Policy is first issuance.

Back to Top

Background

The Clinger-Cohen Act (CCA) of 1996 (Section E of Public Law 104-106, formerly known as the IT Management Reform Act (ITMRA) of 1996), requires federal agencies to use a disciplined CPIC process to acquire, use, maintain, and dispose of IT assets. This CPIC process consists of three phases: Select, Control, and Evaluate. The Evaluate phase of the CPIC process must include methods to manage and report the performance of the Agency’s IT assets using quantifiable measurements of benefits, risks, and progress in terms of cost, capability to meet specified requirements, timeliness, and quality. The OA is the mandated method for evaluating Steady State IT Investments.

The required components of an OA are based on the aforementioned legal mandates of the CCA and on OMB’s Capital Programming Guide, V3.0, the Supplement to OMB Circular A–11, Part 7: Planning, Budgeting, and Acquisition of Capital Assets. This suite of documents provide guidelines for conducting and using the results of the IT OA as a key part of successfully managing the Department’s IT portfolio, ensuring that Steady State IT Investments are continuing to meet their stated goals and intended objectives and yield expected benefits. OMB’s Capital Programming Guide defines operational analysis as “a method of examining the ongoing performance of an operating asset Investment and measuring that performance against an established set of cost, schedule, and performance goals.”

Back to Top

Scope

The requirements within this policy will be applied to all Steady State IT Investments and their associated Steady State IT projects, systems, products, and services. Per OMB’s Capital Programming Guide, “regardless of performance of operational indicators, a formal operational analysis is warranted for every Steady State project.” An annual OA is required for all IT Investments that are currently in the O&M or Steady State life cycle phase. These requirements are in effect and are to be implemented regardless of size, scope, complexity, or funding source, whether owned and operated by HHS or operated on behalf of HHS.

This Policy applies to all Department Operating Divisions (OpDivs) and Staff Divisions (StaffDivs), including the Office of the Secretary (OS) and Office of the Inspector General (OIG), and to all organizations conducting business for and on behalf of the Department. Assigned officials within HHS shall apply this policy to contractor personnel, interns, and other non-government employees conducting business for or on behalf of the Department through contractual relationships or memoranda of agreement. The content of and compliance with this Policy shall be incorporated into applicable contract language, as appropriate.

OpDivs and StaffDivs shall use this Policy or may create a more restrictive OpDiv/StaffDiv policy tailored to their specific environment; however, that policy may not be less restrictive, less comprehensive, or less compliant than this Department Policy.

This Policy shall be applied and implemented, as appropriate, in conjunction with the HHS Policy for IT CPIC; the HHS OCIO Policy for IT Enterprise Performance Life Cycle (EPLC); the HHS-OCIO IT Policy for Enterprise Architecture (EA); and any Federal Acquisition Regulation (FAR) and HHS Acquisition Regulation (HHSAR) rules that are promulgated on the subject.

This Policy does not supersede any other applicable law, higher level agency directive, or existing labor management agreement.

Back to Top

Policy

Conducting an OA is an essential requirement for any operational Investment because the OA process is used to determine the disposition of an Investment in the Evaluate phase of the CPIC cycle. An OA is required of all IT Investments on an annual basis as a part of CPIC best practices. Additionally, the OA Results Report is a critical EPLC artifact for all IT Investments in the O&M phase of the framework.

Business owners are required to conduct an OA on Major and Non-Major IT Investments on an annual basis as the OA is a critical standard that is used Department-wide to evaluate the performance and continued viability of HHS IT Investments that are wholly or partially in Steady State. As such, the timeliness of an Investment’s OA plays a significant role in the calculation of any Major IT Investment’s Chief Information Officer (CIO) Evaluation, which is publicly available on OMB’s IT Dashboard. Major IT Investments that have not provided an OA Results Report in the preceding 18 month period are at risk of receiving a lower overall CIO Evaluation.

Major IT Investment Managers are also required to provide OMB with the Investment’s latest OA as part of the annual budget cycle. Any new or updated OA for a Major IT Investment must be uploaded to the Datapoint Artifact Sharing site within OMB’s MAX Collect tool.

Required Content for OA Results Report

OA Results Reports must at a minimum cover the areas defined below. The required content implements the recommendations of the OMB Capital Programming Guide, Supplement to OMB Circular A-11, Part 7, which states that beyond the typical developmental performance measures of cost and schedule performance, an OA should seek to answer more subjective questions in the specific areas of: Customer Satisfaction, Strategic and Business Results, Financial Performance, and Innovation. The Government Accountability Office (GAO) Report: Agencies Need to Strengthen Oversight of Multibillion Dollar Investments in Operations and Maintenance derives the 17 key factors listed below from the Capital Programming Guide that should be addressed in the OA Results Report.

For enhanced guidance regarding OA Results Report content, refer to the guidance and templates referenced in section 7 of this policy, “Information and Assistance”.

OA Subsection

Description

OA Results Report Key Factor

Customer and Stakeholder Satisfaction Assesses feedback from customers and stakeholders to determine the extent to which the Investment supports customer needs as designed. Identifies whether the Investment supports customer processes as designed and is delivering the goods and services it was designed to deliver
Strategic and Business Results Measures and evaluates the Investment’s impact on the performance of the OpDiv and HHS to determine how well the Investment is meeting business needs in its strategic alignment and business impact.

Includes a structured assessment of performance goals (i.e., measures the performance of the Investment against established goals)

Measures the effect the Investment has on the performing organization itself

Measure how well the Investment contributes to achieving the organization’s business needs and strategic goals

Identifies recommendations to redesign or modify an asset before it becomes a problem

Identifies a need to redesign, modify, or terminate the Investment after a problem has been identified

Identifies any lessons learned

Financial Performance Measures and assesses the Investment’s current and expected performance compared to the pre-established cost baseline through performance baseline metrics, expenditures, and comparative measures.

Assesses current costs against life-cycle cost

Includes a structured schedule assessment (i.e., measures the performance of the Investment against its established schedule)

Compares current performance with a pre-established cost baseline and estimates

Includes an ongoing review of the status of the risks identified in the Investment’s planning and acquisition phases

Identifies if the Investment had a cost or schedule variance

Innovation An opportunity to conduct a qualitative analysis of the Investment’s performance within the previously mentioned areas – demonstrating that alternative methods for meeting the same needs have been revisited.

Identifies any areas for innovation in the areas of customer satisfaction, strategic and business results, and financial performance

Identifies if the agency revisited alternative methods for achieving the same mission needs and strategic goals

Addresses issues such as greater utilization of technology or consolidation of Investments to better meet organizational goals

Includes an analysis on the need for improved methodology (i.e., better ways for the Investment to meet cost and performance goals)

Includes information on the overlap of the Investment with other systems

Investment Recommendation

 

Provides a recommendation to the OpDiv governance board regarding next steps for the IT Investment. The recommendation might include one or more of the following conclusions:

  • Continue the IT Investment without adjustment
  • Modify the associated business processes
  • Increase operational management controls
  • Reduce or increase funding to sustain current operating levels
  • Enhance one or more of the IT Investment’s products or services, or acquire a new product or service
  • Retire or replace one or more of the IT Investment’s IT products or services
  • Restructure the IT Investment by incorporating another IT Investment’s IT products or services, or transfer the IT Investment’s products or services to another IT Investment

OA Results – Determining Next Steps for the IT Investments

The applicable OpDiv IT governance board will review the OA Results Report taking into account the Investment recommendation in the OA Results Report and will determine the final disposition of the IT Investment. Appeals to the OpDiv IT governance board decision may be made to the HHS CIO.

Special Considerations for Sufficiently Complex IT Investments

If an IT Investment consists of multiple operational IT products, services, or systems; a separate OA may be conducted for each or for each related set. In the event that separate OAs are conducted for the individual operational IT products, services, or systems of the IT Investment, the IT Investment’s overarching OA shall comprise summaries of the processes used and results of the annual OAs that were conducted. When an IT Investment includes only one operational IT product, service, or system, the annual OA conducted shall serve as the OA for the IT Investment.

Back to Top

Roles and Responsibilities

HHS CIO Roles and Responsibilities

The HHS CIO shall:

The HHS IT CPIC Officer shall:

OpDiv Roles and Responsibilities

The OpDiv CIO shall:

The OpDiv IT CPIC Officer shall:

The OpDiv IT Governance Board shall:

Investment Level Roles and Responsibilities

The Business Owner shall:

The IT Investment Manager shall:

Critical Partners shall:

Back to Top

Applicable Laws and Guidance

Back to Top

Information and Assistance

All Department OCIO policies, standards, and procedures will be posted on the HHS Internet (http://www.hhs.gov/ocio/policy/index.html) and the HHS Intranet (http://intranet.hhs.gov/infosec/policies_memos.html).

Further guidance regarding the EPLC process, to include templates, checklists, and practice guides for all artifacts including OAs can be located at: (http://www.hhs.gov/ocio/eplc).

All questions, comments, suggestions, or requests for further information should be directed to the HHS EPLC Program at HHSEPLC@hhs.gov.

Back to Top

Effective Date/Implementation

The effective date of this Policy is the date the policy is approved.

The undersigned acknowledge they have reviewed the HHS OCIO Policy for Information Technology Capital Planning and Investment Control. Changes to this policy will be coordinated with and approved by the undersigned or their designated representatives.

Back to Top

Approved

Beth Killoran

Director, Office of IT Strategy, Policy & Governance

Office of the HHS Chief Information Officer (CIO)

Back to Top

Glossary

Table 1 – Definitions of Terms

Term

Definition

Business Owner The organizational executive who is the primary IT Investment customer, advocates for the IT Investment, and serves as the primary point of contact to the CIO and the IT Governance Board (or designated governance body).
Critical Partners Subject matter experts in enterprise architecture, security and privacy, acquisition management, finance, budget, performance, human resources, and other areas.
Information Technology (IT) Any equipment or interconnected system or subsystem of equipment used in the automatic acquisition, storage, manipulation, management, movement, control, display, switching, interchange, transmission, or reception of data or information. This includes equipment used by the executive agency directly or used by a contractor under a contract with the executive agency that (i) requires the use of such equipment, or (ii) requires the use, to a significant extent, of such equipment in the performance of a service or the furnishing of a product. The term includes computer, ancillary equipment, software, firmware, and similar procedures, services (including support services), websites, subscriptions to electronic services and products, and related resources.
Investment Manager The Investment Manager is responsible for planning and executing the Investment to achieve approved baselines. The Investment Manager may or may not be a subject matter expert in the business area supported by the Investment.
IT Investment

The expenditure of resources on selected information technology or IT-related initiatives with the expectation that the benefits from the expenditure will exceed the value of the resources expended (GAO-04-394G, IT Investment Management, A Framework for Assessing and Improving Process Maturity, March 2004, v.1.1).

The acquisition of an IT asset and the management of that asset through its life cycle after the initial acquisition.

IT Operations Manager The IT Operations Manager reports to the IT Investment Manager and manages the cost, schedule, and performance of one or more of the IT Investment’s operational IT products, services, or systems. The role of the IT Project Manager and Operations Manager might be held by the same or multiple people throughout the life cycle of the Investment.
IT Project A temporary planned endeavor funded by an approved information technology Investment, thus achieving a specific goal and creating a unique product, service, or result. A project has a defined start and end point with specific objectives that, when attained, signify completion.
IT System A discrete set of information resources organized for the collection, processing, maintenance, transmission, and dissemination of information, in accordance with defined procedures, whether automated or manual to support HHS’s or OpDiv’s (including OS’s) mission. An interconnected set of information resources under the same direct management control, which shares common functionality. A system normally includes hardware, software, information, data, applications, communications, and people. Refers to a set of information resources under the same management control that share common functionality and require the same level of security controls. Includes automated information system (AIS) applications, enclaves, outsourced IT-based processes, and platform IT interconnections.
Life Cycle The duration of all activities associated with the Investment from its initiation through disposal of its assets.
Life Cycle Costs All initial costs, plus the periodic or continuing costs of operation and maintenance (including staffing costs), and any costs of decommissioning or disposal.
Major IT Investment

An IT Investment that meets at least one of the criteria listed below:

  • Is designated by the HHS CIO as critical to the HHS mission or to the administration of programs, finances, property, or other resources.
  • Is for financial management and obligates more than $500K annually.
  • Requires special management attention because of its importance to the mission of function of HHS or an OpDiv.
  • Has a significant program or policy implication.
  • Has high executive visibility.
  • Has high development, operating, or maintenance costs, deemed by HHS as:
    • Annual costs equal to or greater than $10M
    • Estimated life cycle costs equal to or greater than $75M.
  • Is defined as Major by the HHS CIO.
Operational Analysis An Operational Analysis (OA) evaluates IT Investment performance, user satisfaction with the IT Investment, the IT Investment’s adaptability to changing business needs, and new technologies that might improve the IT Investment. The OA review is diagnostic in nature and can lead to the initiation of development or maintenance activities. Any major IT Investment modifications needed after the IT Investment has been implemented follow the EPLC framework life cycle process from planning through implementation. The OA ultimately determines whether the IT Investment should continue, be modified or be terminated.
Operations and Maintenance (O&M)

The activities related to the performance of routine, preventive, predictive, scheduled, and unscheduled actions aimed at preventing equipment failure or decline with the goal of increasing efficiency, reliability, and safety. The phase within the Capital Planning and Investment Control process which maintains the IT system(s) for ongoing activities within the production environment.

 

Performance Measurement Baseline

A primary tool to measure IT Investment, IT project, or IT contract performance and identifying risk. The baseline identifies the work that will be accomplished, and defines the cost and schedule to accomplish that work. The Performance Measurement Baseline, which consists of the cost, schedule, and scope baseline, is derived from the scope of work described in a hierarchical Work Breakdown Structure (WBS) – which, in turn, decomposes the entire project into a logical structure of tasks and activities tied to deliverables and to assigned responsibilities – and the associated WBS dictionary. The Performance Measurement Baseline comprises:

  • The cost baseline, which defines the approved, projected, time-phased, life-cycle costs for acquiring, operating, and disposing of the physical and/or logical system represented by the scope baseline.
  • The schedule baseline, which is the approved timeline for acquiring, operating, and disposing of the physical and/or logical IT asset/system.
  • The scope baseline, which represents the configuration of the product of the project as developed and described in the project’s technical documentation.

The Performance Measurement Baseline is integrated where the time-phased cost baseline is consistent with the schedule baseline, and the costs are related to acquiring, operating, and disposing of the physical and/or logical IT asset represented by the scope baseline.

Back to Top

Table 2 – List of Acronyms

Acronym

Term

AIMD GAO’s Accounting and Information Management Division
AIS Automated Information System
AMD Administration and Management Domain
ANSI/EIA American National Standards Institute/Electronic Industries Alliance
CCA Clinger-Cohen Act of 1996
CIO Chief Information Officer
CPIC Capital Planning and Investment Control
EA Enterprise Architecture
E-GOV Electronic Government
EPLC Enterprise Performance Life Cycle
FAR Federal Acquisition Regulation
FEA Federal Enterprise Architecture
FISMA Federal Information Security Management Act of 2002
GAO Government Accountability Office
GPRA Government Performance and Results Act of 1993
HHS Health and Human Services, Department of
HHSAR HHS Acquisition Regulation
HHSD Health and Human Services Domain
IRM Information Resource Management or Information Resources Management
IT Information Technology
ITIRB Information Technology Investment Review Board
ITMRA IT Management Reform Act
ITSC Information Technology Steering Committee
O&M Operations and Maintenance
OA Operational Analysis
OAMP Office of Acquisition Management and Policy
OAP Operational Analysis Policy
OCIO Office of the Chief Information Officer
OMB Office of Management and Budget
OpDiv Operating Division
OS Office of the Secretary
PBM Performance Baseline Management
PIR Post-Implementation Review
PMA President’s Management Agenda
PMT Portfolio Management Tool
SRD Scientific Research Domain
StaffDiv Staff Division
WBS Work Breakdown Structure

Back to Top