The thousand-ring circus

On the 50th anniversary of Apollo 11, I thought readers might get a kick out of seeing this funny 1968 memo regarding a problem that needed to be fixed in the Lunar Module (it was), and learning about its extraordinary author, NASA engineer Howard W. “Bill” Tindall, Jr. I wrote about this memo five years ago with just a little information on Tindall, but I wanted to expand on that a fair amount this week because without his efforts, I’m pretty certain we would not have reached the moon before that decade was out.

Click for a larger version

I first learned of Tindall in 1989 when I read Apollo: The Race to the Moon by Murray and Cox, which I think will ever remain the definitive Apollo history from the perspective of technical people on the ground, and have since gathered the information that’s included here from 1,700 pages of his memos that the Kennedy Space Center History Office sent to me in 1999, individual memos kindly provided by the University of Houston-Clear Lake from their Johnson Space Center History Collection, some JSC oral histories, and several other books and online resources.

After his earlier work on Mercury trajectories and Gemini rendezvous techniques, Bill Tindall’s parchment-dry title was “Chief, Apollo Data Priority Coordination,” a position created by Apollo Program chief George Low that quite unusually cut across several branches of the Manned Spacecraft Center in Houston. Tindall worked with design engineers, contractors, mathematicians, programmers, mission controllers, and astronauts – everyone, really – to develop and hone the dozens of mission techniques that were used in each one of the twelve distinct phases of lunar missions. Guidance flight controller Steve Bales said of Tindall, “He had a thousand-ring circus going all the time.”

Flight Director Gene Kranz: “Tindall was pretty much the architect for all of the techniques that we used to go down to the surface of the moon. Tindall was the guy who put all the pieces together, and all we did is execute them. If there should have been a plaque left on the moon for somebody in Mission Control or Flight Control, it should have been for Bill Tindall. I respected Bill so much that when the time came for the [Apollo 11] lunar landing, the day of the lunar landing, I saw him up in the viewing room, and I told him to come on down and sit in the console with me for the landing. He didn’t want to come down, but I cleared everybody away and we had Bill Tindall there for landing, and I think that was probably the happiest day of his life. A spectacular guy.”

Late last month, the Johnson Space Center re-opened the painstakingly and beautifully restored Apollo-era Mission Operations Control Room, MOCR 2: https://arstechnica.com/science/2019/06/behind-the-scenes-at-nasas-newly-restored-historic-apollo-mission-control/. How that restoration came about is discussed in detail by JSC Historic Preservation Officer Sandra Tetley and contractor lead Adam Graves in this hour-long episode of “Houston We Have a Podcast”: https://www.nasa.gov/johnson/HWHAP/restoring-the-apollo-mission-control-center

Tindall’s frequent memos – usually two to four a week – were all dictated because Patsy Saur, his secretary, said he’d better learn how because she was not going to lose her shorthand proficiency. They were called Tindallgrams by those who eagerly awaited their common sense, humor, and perfect condensations of discussions and decisions made during the meetings he conducted – and conducted is precisely the right word. Some of those meetings went on for two or three twelve-hour days, with anywhere from half a dozen to a hundred people in the conference room discussing – or, sometimes, shouting and arguing vehemently – and coming to a consensus on every item on the agenda – or, sometimes, accepting Tindall’s final decisions via Tindallgram. Tindall, Buzz Aldrin’s equal in orbital mechanics (Aldrin’s MIT doctoral thesis was “Line-of-Sight Guidance Techniques for Manned Orbital Rendezvous”), once estimated that he spent just 10 to 20% of his time on standard mission techniques and the rest developing finely-detailed “what if” contingency plans, many of which were never needed but some of which came in very handy indeed. The increased peace of mind I’m sure he had as a result was no doubt shared by many because they all knew that there was a precise plan for just about any problem imaginable.

They were after what was right, and everybody was passionate about it. Everybody was young so they were kind of brash and there wasn’t a lot of patience anywhere. So some of those meetings were very, very colorful. Some of the characters were colorful. At the end of this, you were just inundated with all of this stuff you’ve heard. And now what?

And the next day you would get this two-, maybe three-page memorandum from Bill Tindall written in a folksy style, saying, ‘You know, we had this meeting yesterday. We were trying to ask this. If I heard you right, here’s what I think you said and here’s what I think we should do.’ And he could summarize these complex technical and human issues and put it down in a readable style that – I mean, people waited for the next Tindallgram. That was like waiting for the newspaper in the morning. They looked forward to it. I just remember that I’ve always talked to people about this amazing skill.

– Ken Mattingly, Command Module Pilot, Apollo 16

Just how complicated could Tindall’s mission techniques get? Consider that Apollo 11 Command Module Pilot Mike Collins put this CMP Solo Book on a string around his neck a few hours before Armstrong and Aldrin departed for the lunar surface (onboard audio: “Neil, I hate to bother you; could you get my solo book out of R-1 there? Big frapping book, with a bunch of updates on the cover.”). Starting on page 60 are summarized procedures – cheat sheets, if you will – for eighteen different Lunar Module rescue scenarios that Collins might have to execute if his crewmates “never made it to the lunar surface, or if they got there early or late, or departed crooked or straight” (Collins in Carrying the Fire). Some involved Collins diving the 32-ton Command-Service Module from its 60-nautical-mile lunar orbit to as low as they dared – possibly down to 35,000 feet, but I think they would have been a tad more conservative – in order to catch up to the LM if its orbit was higher and slower than the CSM’s, an example of how counter-intuitive orbital mechanics can be.

Here’s a YouTube link to an MIT “Engineering Apollo” class with the sharp and funny Collins in 2015. The interviewer/presenter is Professor David Mindell, the author of Digital Apollo.

Tindall also kept up with the latest scuttlebutt, which at times required that he step in to protect things that needed protecting. For example, when he heard that a NASA high mucky-muck said they should get rid of the Lunar Module’s rendezvous radar to save weight, and that people were beginning to take the idea seriously, Tindall took action to nip that in the bud immediately by writing this memo to George Low, the boss of all Apollo bosses. He didn’t name the official in the memo, but it was Associate Administrator for Manned Space George Mueller who made the flippant suggestion after a visit to Grumman on Long Island, where LM weight reduction was a constant focus for years. After Low read Tindall’s high-energy memo, some memos went between higher mucky-mucks and a few weeks later Mueller’s boss told him, in summary, “Yeah…no.

Sometimes fairly unlikely scenarios gnawed at him a bit – such as whether their re-entry targeting was so good that a Command Module might, by mistake and with a catastrophic result, hit the aircraft carrier that was waiting for its splashdown. His method of dealing with small worries was the same as the large ones: address all eventualities completely through thorough planning. In this case, his memo titled Let’s move the recovery forces a little. (“PAO requirements for good commercial TV” refers to the NASA Public Affairs Office.)

Another of the 1,000+ Apollo memos Tindall wrote from 1966 to 1970 was on the topic of why Apollo 11’s Eagle overshot its intended landing site by four miles. It described how incomplete venting (that is, depressurization) of the docking tunnel prior to undocking caused the Lunar Module to pop like a cork off the Command Module with just a little extra velocity, which in turn caused significant changes in its descent profile. A new rule for subsequent missions required that Mission Control confirm complete depressurization of the tunnel. A related Tindallgram on other venting sources adversely affecting the descent trajectory was titled Vent bent descent, lament!, and he wasn’t shy about making his strong feelings on those vexing vents known to all the top brass at NASA, including chief spacecraft designer – also a culprit – Max Faget, in an unusually all-caps-titled VENTS (“This will either amuse you, waste your time, or just possibly accomplish something great.”)

After a three-day-long “Mission Techniques free-for-all” not even two weeks after Apollo 11, he wrote How to land next to a Surveyor – a short novel for do-it-yourselfers. That and a follow-up memo, in which he revised his previously pessimistic targeting prognosis, detailed new mission techniques that were key to Apollo 12 Commander Pete Conrad being able to set Intrepid down just 535 feet from the Surveyor 3 spacecraft that had, two-and-a-half years earlier, soft-landed on the Ocean of Storms after bouncing twice due to a slightly-too-early engine shutdown.

Click for a larger version

Such pinpoint accuracy was life-critical for later landings, in particular Apollo 17, which landed in the Taurus-Littrow Valley, a box canyon surrounded by mountains on three sides.

Click for a larger version

Here’s an excellent 2015 Apollo 17 documentary in two parts: https://www.youtube.com/watch?v=vIGbOoZzlYI https://www.youtube.com/watch?v=SQOEC9gHpmA

Oh, yeah…for a period of about a year in 1966-67, Tindall, who grew up in Scituate, Massachusetts, flew up to Cambridge from Houston for two or three days every week to help organize, focus, and speed up – effectively manage, sometimes in a blunt manner – the MIT Instrumentation Lab’s previously somewhat free-form development of the COLOSSUS and LUMINARY software for the Apollo Guidance Computers (AGC) in the Command and Lunar Modules, respectively. (He visited often enough that he sent out a TripAdvisor-style memo every now and then.)

Early on, Lab engineers reported, to Tindall’s great alarm, that the Command Module code was about 30,000 bytes in excess of the 72,000 available in the AGC and the Lunar Module software was around 10,000 over its 72,000. 13 October 1966, the day Tindall directed them, in person, to eliminate much duplicated code that he had found, and to cut several elegant but non-essential and hence memory-wasting routines, became known to those in the Instrumentation Lab as “Black Friday.” Two weeks after Black Friday, he discussed his strategy in this memo, which began with the important point that “There are a number of us who feel that the computer programs for the Apollo spacecraft will soon become the most pacing item for the Apollo flights.” Despite the initial hard feelings at the Lab, they did what he asked, and over time came to realize just how beneficial his involvement was to their work – and best of all, that work was ready when it needed to be.

Here’s a profile of Margaret Hamilton, who, two years after the Lab’s early difficulties, became leader of the Apollo spacecraft software development effort: https://www.smithsonianmag.com/smithsonian-institution/margaret-hamilton-led-nasa-software-team-landed-astronauts-moon-180971575/

In late 1965 just before his work on Apollo began, the New York Times profiled Tindall in a brief Gemini 6/7 sidebar titled “Rendezvous Planner Howard W. Tindall, Jr.” (reprinted in the January 1966 Brown Alumni Monthly here), but Charles Fishman, who contacted me while researching his new book, One Giant Leap: The Impossible Mission That Flew Us to the Moon, says that when Tindall died in 1995, not one newspaper in the US ran an obituary. It’s even difficult to find any photographs of him bigger than a postage stamp, but here are a couple: below, one in his office (a screenshot from episode 3 of the also excellent “Moon Machines” series, playlist here: https://www.youtube.com/playlist?list=PLTu8nanTJo7GvulBxz9JT9JcXeXimM1Vr) and he’s in the center of this photo taken during Apollo 13, chin in hand, looking at papers – some probably written by him.

I’ve always thought that more people ought to know about this remarkable man. To paraphrase him, if you are still with me, hardy reader, now you do.

Bill Tindall; click for a larger version

I think it’s safe to say he thoroughly disliked inaccuracy and inexactitude, which may be reflected in the “H. Timdell” [sic] name I noticed taped to the wall behind him in that photo, the misspelling perhaps from some conference he attended. I’ve no evidence for it, but I like to think he kept it up there to point out to visitors at appropriate moments, maybe with a raised eyebrow and a little flourish of sarcasm.

We’d all get in there and defend our [computer] requirements, and then Tindall would cut them. And then we’d cuss him. And Tindall would grin, and cuss back, and laugh his loud, infectious laugh, and keep right on going.

– Apollo Flight Director Cliff Charlesworth

We weren’t working overtime, we were playing!

– Bill Tindall

Edited 9 August 2019 to add: My theory above about that misspelling on his wall is now inoperative…defunct…shot down. The Johnson Space Center History Office has kindly found and sent me the original of that official photo along with nine others of Tindall from 1965-1979, which I’ve just posted here: https://finleyquality.net/The-ringmaster. Some deductive reasoning on the uncropped version of that one that they sent reveals the much more likely source of “H. Timdell” [sic].

Edited 21 August 2019 to add: I just happened upon this tidbit while reading Harrison Schmitt’s 1999 Johnson Space Center oral history interview. Twenty-seven years after his Apollo 17 mission, Schmitt emphasized how important Tindall’s memos were, not just at the time but for purposes of mission planning in the future (emphasis mine):

Well, Frank Borman approached me, asked me if I would do the lunar orbit flight planning for their effort. And that meant that I began to interact with [Howard W.] Tindall’s group, the Flight Operations Planning group that met weekly that really was the focus of all of the operational planning for a particular mission. They were looking at all the missions, but the one up was the one they were concentrating on. And that’s another tremendous resource.

And I’m not sure where there is a complete collection of what were called Tindallgrams. They were his summary of each of those meetings. I have a partial collection at the University of New Mexico in the files there. Whether there would be a complete collection or not, I don’t know. But somebody ought to make a very, very specific effort to get a complete collection of the FOP minutes, Tindallgrams, and to get those in some kind of form and bound. Because that is a resource that should not be lost. I can understand it’s hard to put together. I hope somebody has been able to do that.

Hallway update

I added a new 12×36″ enlargement to my refreshed hallway gallery today, a 1:3 aspect ratio crop of a high-resolution scan of the photo of the first flight of the Wright Flyer on 17 December 1903 at Kill Devil Hills, North Carolina.

Click for a larger version

Orville is piloting and that’s Wilbur at the wingtip. John Daniels, one of the five witnesses to the flight, took the photograph with Orville’s pre-positioned camera – so awed by what he saw that he almost forgot to squeeze the bulb to capture this image on the 5×7″ glass plate negative.

From the Flyer to the Apollo 16 Lunar Module Orion above it was a span of just sixty-eight years and four months.

The full-size first flight image from the Library of Congress can be found here – be aware that it’s 27MB.

Edited to add: The comments here include a discussion in some detail of the soon-to-be-released film “First Man” and HBO’s 12-part 1998 series “From the Earth to the Moon”.

Posters and framing on the cheap

Click for a larger version

Last weekend, I refreshed the pictures in my upstairs hallway, the new ones shown above. As a frame of reference, the photo shows an area of about 7×3 feet. For about a hundred dollars total, I was able to get three 16×20″ prints and one 12×36″ panorama of high-resolution Apollo-era photographs from Shutterfly and mount them in the best borderless clip frames available.

There was a time when I did my own picture mounting on foam board and framing using mail-order Nielsen #11 frame pieces and locally-sourced, custom-cut sheet glass (I never attempted matting), but these days I most often use clip frames – good ones, that is – because they’re easier, they look clean and classy, and they’re a lot cheaper than professional framing or even DIY Nielsens. The last picture I had mounted, double-matted, and framed, the “Clipper at the Gate” shown below, cost me well north of US$200 – and that didn’t include the signed print, which I had purchased several years previously. Don’t get me wrong – the framing and matting is well-done and quite attractive, but I have a lot of drawings, paintings, and photos on my walls and I am well south of a millionaire.

I was able to get those four hallway prints done both well and on the cheap thanks to four things:

  • In recent years, the negatives from the Apollo programme have been scanned with better equipment and at much higher resolution, which allows for nice-looking enlargements – not the case with the low-res images previously available. In the case of the three-foot-wide print, someone stitched together a 10,000-pixel-wide image from a panorama photo series Charlie Duke took during Apollo 16.
  • The recently completed Lunar Orbiter Image Recovery Project used current technology to produce, from the data on hundreds of carefully preserved original 1960s magtapes, awe-inspiring photos far beyond the resolution and quality NASA could produce fifty years ago. The top middle picture in the hallway is an oblique photo of Copernicus from 150 miles south of the crater that was taken by chance during a “let’s move the film forward a bit” housekeeping task on Lunar Orbiter 2.
  • A plethora of discounts, including 40% or 50% off sales that Shutterfly runs every week or two, periodic Visa Checkout deals (US$25 off the next order), and even $25 Shutterfly credits that Best Buy includes with many hard drive purchases means you can easily get prints in these bigger sizes for $12-$16 each. That’s cheap for high quality large prints.
  • Massachusetts-based Quadro Frames, which I’ve used for many years, produces the highest quality borderless clip frames I’ve seen; other, more widely-available types are mostly flimsy and ill-fitting. 16×20″ frames from Quadro are US$12.50 and it’s $20 for 12×36″. Each frame is precisely fashioned and includes a sturdy, non-bending backing board with perfectly cut, strong clip channels on the back, pristine and perfectly clear PET plastic glazing panels with peel-off protective sheets on both sides (or glass panels for just $3 more), and more than enough clips that slip into the back channel with a satisfying firm snap. Even their care in shipping to guarantee safe arrival is the best possible: I always think, “Wow, just look at that” when I open boxes from them. For some of my orders, I’ll wager it’s taken them half an hour or more to pack the materials so fastidiously. It’s a good example of corporate responsibility and pride in doing things right.

Here are the source photographs I uploaded to Shutterfly for the hallway prints. You can pause the slideshow and right-click to view and/or save any image at its full size.

This slideshow requires JavaScript.

I also got these three enlarged to 16×20″ and they’re up elsewhere in the house:

This slideshow requires JavaScript.

The explosive guillotine in the Lunar Module

1968: Apollo 11’s LM-5 ascent stage under construction at Grumman in Bethpage, Long Island. The ascent stage without fuel weighed 2,445 kg/5,390 lb; double that with fuel. Click for a larger version.

You: “Did you say ‘guillotine’?”
Me [approximating John Cleese]: “Explosive guillotine, yes.”

August 2018 update: See also the companion article Don’t get me started, in which Apollo 17 LMP Jack Schmitt describes a mind-boggling workaround they could attempt if the ascent stage launch pyrotechnics failed to fire.

See also the comments at the end of this article for a photo of a guillotine housing, sent by a fellow who machined many of them at Grumman.

LM guillotine

In missions past and present, explosive devices feature in pretty much every spacecraft because they’re a safe, reliable way to ensure that processes start, items such as antennas are deployed, and connected assemblies that need to come apart are quickly and cleanly separated.

On the Apollo missions, over 210 pyrotechnic devices were in the Saturn V stack and the Command, Service, and Lunar Modules, used for everything from extending the LM landing gear to deploying drogue and main parachutes to ensuring fuel was at the correct side of a tank in zero G – for which the word ullage (in French, ouillage) was borrowed from vintners, to whom it means the headspace between the top of the wine and the container it’s in, whether a cask or a bottle.

The Lunar Module had several devices on board:

ED Locations

Control over them was through the Explosive Devices Control Panel:

ED Control Panel

All of these devices were essential, but particularly key were the devices set off to initiate separation of the ascent and descent stages of the LM when the astronauts departed the lunar surface. This was a three-stage process that took place in the tenths of seconds before the ascent engine was lit:

  • First, fire circuit interrupters to cut off electrical signals between the stages
  • Second, fire and shear the four explosive nut and bolt assemblies that affix the ascent stage to the descent stage
  • Third, using the explosive guillotine, slice through a thick bundle of umbilical cables and wires and a water supply line that run between the two stages

Though these devices were known to be generally reliable, a certain level of trepidation about them is understandable. Blowing up some high explosives to drive a big blade through wires doesn’t exactly sound like the most controlled process even though it actually was.

“Did you know when that unit was up on the moon, and the ascent stage was going to take off, they had all those wires – fourteen miles of them – running from the ascent stage into the descent stage, and it all had to be disconnected before you took off, or you didn’t take off? That’s all there was to it. You couldn’t use wire couplings that just pulled out when you gave it a good, hard yank. Do you want to trust a wire coupling to hold through a Saturn V liftoff and all that g-force and vibration? Uh-uh. Try flying a ship with a few loose wires. So, it was all solid connections, which is why we put a guillotine inside the descent stage: to cut all the wires. Everything had to be timed just right. The explosives had to trigger the guillotine and the blade had to cut through some pretty thick cables, and at the same time, the ascent rocket engine, which was never run before, had to start.”
– Bob Ekenstierna, LM descent stage construction supervisor at Grumman

In Chariots for Apollo by Pellegrino and Stoff, a somewhat sensationalist telling of the building of the Lunar Module at Grumman*, they speak of Joe Kingfield, the director of quality control. I won’t quote them directly since they went over the top with their narrative, but Kingfield had frequent nightmares about the liftoff from the moon that involved the guillotine and one or more of the explosive bolts failing. In his dreams, the ascent stage lifted off, but, still connected by miles of wire, dragged the descent stage along the ground and eventually crashed back into the surface. In later years, Kingfield still could not bring himself to watch the footage of the lunar liftoffs taken by the Mission Control-directed TV cameras on the Lunar Roving Vehicles of Apollo 15, 16, and 17.

*Not to be confused with the unimpeachable NASA volume of the same name by Brooks, Grimwood, and Swenson; web and epub links at the link. 2018 update: You can get a free high-quality scanned PDF of this book and many other official NASA histories – see the I got mine at the GPO bookstore post for the NASA Technical Reports Server links.

Charlie Duke, Lunar Module Pilot of the Apollo 16 Orion (the LM pictured in the Finley Quality Network banner above), said that the pyrotechnics for the ascent stage separation gave him brief pause just before he and Commander John Young lifted off from the moon. When the circuit interrupters fired, then the four interstage bolts at the corners were sheared, and finally the guillotine sliced through the umbilical and water lines, the entire ascent stage suddenly dropped an inch or so. Duke thought, “Oh, sh…” but did not have time to finish that thought as the ascent engine fired and abruptly took them away from the surface back toward Ken Mattingly awaiting their return aboard the Casper Command and Service Module in lunar orbit.

You can see a fair amount of the thermal protection fly off the ascent stage as it lifts off, which happened to all of the ascent stages to some extent. In addition, panels on the rear that provided thermal protection for the Aft Equipment Bay were damaged during the liftoff, but they had done their job already. Mattingly took this photograph of the Orion before docking:

AS16-122-19535

Apollo deniers like to point to this and other photographs of the Orion damage as ineluctable proof of chicanery, but what it really means is that they prefer extending and enhancing their apparently quite enjoyable fantasies to, say, reading the post-mission report (9th link in the background material):

At lunar lift-off, four vertical thermal shields (fig. 14-26) on the aft equipment rack were torn loose from the lower standoffs and remained attached only at the upper standoffs. This occurrence was observed from the lunar-based television.

The most probable cause of the failure was ascent engine exhaust entering the cavity behind these thermal shields. A cross section of the lower edge of the shields is shown in figure 14-27. Analysis shows that the thermal shield which extends below the support tube allows a pressure buildup on the closure shield which exceeds its capability. Once the closure shield failed, the exhaust entered the cavity behind the shield, resulting in a pressure buildup exceeding the capability of the vertical thermal shields.

In the lunar surface photographs taken prior to lift-off, some of the shields appear to have come loose from the center standoff (fig. 14-28). Excessive gaps between some of the panels are evident. Both conditions could be caused by excessive pressure in the thermal blanket due to insufficient venting during boost.

The corrective action will include a redesign of the thermal shield to eliminate the projection below the support tube, as shown in figure 14-27, and to provide additional venting to the blankets as well as additional standoffs.

This anomaly is closed.

Not one problem was detected in any of the pyrotechnics during any Apollo mission. The device designs used in Apollo were later adopted by the Shuttle program, with, for instance, the Single-Bridgewire Apollo Standard Initiator (SBASI) becoming the NASA Standard Initiator (NSI).