Boeing Software Errors Could Have Doomed Starliner’s Uncrewed Test Flight

A NASA/Boeing Independent Review Team (IRT) looking into problems encountered during Boeing’s December 2019 uncrewed Orbital Flight Test (OFT) has tentatively concluded that two software errors could have led to loss of the spacecraft.  Only intervention by ground control teams during the flight saved the mission. News about the interim IRT findings comes one day after the second of the two software errors, which the agency and Boeing had not publicly disclosed previously, was revealed by a NASA safety panel.

At a quickly organized media teleconference this afternoon, three NASA and two Boeing officials provided an update on the joint investigation into the OFT anomalies.  The overall messages were that the IRT is not done yet and that is why NASA has not spoken about these issues in public, and while media attention is focused on a few specific problems right now, NASA has broader worries.

Doug Loverro, the head of human spaceflight at NASA, said these two software problems are just “indicators.”

“They are likely only symptoms, they are not the real problem.  The real problem is that we had numerous process escapes in the design, development and test cycle for software, and as we go forward that is what we will be concentrating on, how do we assure ourselves that all of the software” is fixed.  “What this test revealed is … we have a more fundamental problem, which is what we briefed the ASAP team” about.

ASAP is NASA’s Aerospace Safety Advisory Panel (ASAP).  In its public meeting yesterday, panel members surprised many by reporting on the second, previously unknown, software anomaly and characterizing it as potentially “catastrophic.”

Boeing is developing Starliner through a Public-Private Partnership with NASA to ferry crews to and from the International Space Station (ISS).  The OFT was a required step toward certification of the vehicle.  No one was aboard the spacecraft for this test.

On December 20, 2019, Starliner was launched into a suborbital trajectory by a United Launch Alliance (ULA) Atlas V rocket. Starliner is designed to use its own engines to achieve orbit following a pre-set timing sequence after separating from the rocket .

The Atlas V worked perfectly, but a Boeing software error had set Starliner’s Mission Elapsed Timer to the incorrect time.  It was off by 11 hours.  Consequently, when the spacecraft separated from the rocket, it thought it should already be in orbit.  Detecting it had the wrong parameters, it fired its thrusters to correct the error.  Communications problems between ground controllers and the spacecraft — which are also being investigated — delayed their ability to gain control of the situation.

Eventually Starliner did reach orbit, but its fuel was depleted to the extent that it could not continue on to rendezvous and dock with the ISS.  It did complete other objectives and two days later landed in White Sands, New Mexico as planned.

What was not known publicly until ASAP reported it yesterday is that during those two days ground controllers detected a second software error. ASAP member Paul Hill said that if the anomaly had not been detected and corrected, it “would have led to erroneous thruster firings and uncontrolled motion” when Starliner’s Service Module separated from the Crew Module during reentry “with potential for catastrophic spacecraft failure.”

In a statement posted on NASA’s commercial crew website this morning and expanded upon by NASA’s commercial crew program manager Kathy Lueders during the telecon, the second software error could have caused the Service Module to bump into the Crew Module after they separated. The Service Module is commanded into a disposal sequence so it burns up in the atmosphere while the Crew Module descends under parachute to the ground.  On Saturday night, with a Sunday morning landing coming up, Boeing discovered the error, rewrote the software, uploaded it to Starliner and the two segments separated correctly and landing went off without a hitch. But the happy ending was only because Boeing engineers were scouring the software for coding errors because of the first anomaly.

Boeing’s Jim Chilton, Senior Vice President for Space and Launch, said at today’s telecon that Boeing learned a lot from the test flight and wishes “we’d done better on software.” Still, he was proud of his team for finding and fixing the second software bug in time.  “We learned more about our software processes than we wanted to.” John Mulholland, Boeing’s Vice President and Program Manager of the Starliner program, added that no one was more disappointed than the Starliner team, but they are committed to resolving the issues.

Boeing is pushing back on ASAPs characterization that the outcome of the second coding error might have been catastrophic, insisting there is no way to know because it would depend on how hard the two modules hit and whether the Crew Module’s heat shield was damaged.  Nevertheless, NASA’s statement said that both software errors “could have caused loss of spacecraft.”

NASA is responsible for oversight and insight of the Boeing and SpaceX commercial crew programs.  Loverro conceded that NASA made mistakes, too.  “Our NASA oversight was insufficient, that’s obvious.”

The IRT is expected to complete its work by the end of February.  NASA has not made a decision on whether it will require the uncrewed flight test to be repeated. Boeing says that it will take all the input from NASA, the IRT and ASAP and come up with a work plan for the way forward and await NASA’s approval.  None of the NASA or Boeing participants would speculate on when the next launch — either an OFT reflight or a crewed test flight — will take place.

NASA Administrator Jim Bridenstine stressed that NASA is “100 percent committed to transparency” and he has told his senior officials to “never, ever, ever be afraid of the truth.”