NASA IG: SLS/Orion Ground Systems Software Development Needs Independent Review

NASA’s Spaceport Command and Control System (SCCS) — the software component  of the Ground Systems Development and Operations (GSDO) program for the Space Launch System (SLS) and Orion spacecraft at Kennedy Space Center (KSC) — is over budget, behind schedule, and may not work according to a new report from NASA’s Office of Inspector General (OIG). NASA’s approach to developing this software was chosen 10 years ago and may no longer be valid, but the agency refuses to change course, reflecting a cultural legacy of “over-optimism and over-promising.”  The OIG recommended that NASA commission an independent assessment of the SCCS effort and NASA agreed, but will wait until all the software for the first SLS/Orion launch (Exploration Mission-1 or EM-1) is successfully delivered.  The OIG concurred with that decision.

The SCCS software will “control pumps, motors,  valves, power supplies and other ground equipment; record and retrieve data from systems before and during launch; and monitor the health and status of spacecraft as they prepare for and launch,” according to the report.   All of that requires a lot of computer code and NASA decided to use multiple existing commercial software products and “glue” them together with 2.5 million lines of “glue-ware” that NASA itself is developing.  The OIG notes that reengineering the Hubble Space Telescope command and control system required just one-fifth of that amount of glue-ware code.

The effort has turned out to be more daunting than NASA expected, with cost growth of 77 percent (to $207.4 million) and a schedule slip of 14 months (to September 2017).  The OIG notes that both Orbital ATK and SpaceX use commercial software for their missions to the International Space Station (ISS) and thinks NASA should revisit its decision, made 10 years ago, to “glue” together a variety of products from multiple vendors.  The report cites two prior efforts by NASA to develop software on this scale — the Core Electronics System for space shuttle operations and its successor, the Checkout and Launch Control System — that “failed to meet their objectives and were substantially scaled back or cancelled prior to completion” despite the expenditure of more than $500 million.

The OIG’s overall concern is that ultimately the SCCS will not work as expected.  GSDO managers have had to reduce or eliminate capabilities in order to “balance technical capabilities against schedule and cost,” creating concerns that too much has been lost.  Despite efforts to reinstate some of those capabilities, the OIG found that the software that will be used for EM-1 will not have all its planned capabilities, including the ability to “automatically detect the root cause of specific failures.”  Furthermore, as of the end of FY2015, version 4.0 was “3,320 hours ‘out of the budget box’ — meaning there is more estimated work than time and staff available to perform it,” raising concerns that further reductions to content and functionality may result.

The report concluded that much has changed over the past 10 years in the commercial software market and NASA’s decision to “glue” together code from multiple vendors with software developed by NASA itself no long may be the best approach.   GSDO managers reportedly expressed concern about schedule delays that might result from changing the approach now, but the OIG concluded that the “reluctance to change course reflects a cultural legacy at NASA of over-optimism and over-promising what the Agency can achieve in a specific timeframe.”   OIG concluded that while “altering course at this point would be ambitious,” continuing challenges in developing SCCS warrants a reassessment.

NASA noted in response to the OIG that a 2013 review by the Aerospace Corporation found the SCCS Standard Based Architecture to be “generally sound,” and the OIG agreed, but added that Aerospace also recommended an annual independent assessment of the cost and schedule and none has taken place since then.

Therefore, the OIG recommended that NASA commission an independent assessment to take place in parallel with the ongoing development effort, but NASA responded that it would wait until after all the software for EM-1 was successfully delivered.   The OIG said that is “responsive” to its findings and its recommendation is “resolved and will be closed upon completion and verification of the proposed corrective action.”