The thousand-ring circus

On the 50th anniversary of Apollo 11, I thought readers might get a kick out of seeing this funny 1968 memo regarding a problem that needed to be fixed in the Lunar Module (it was), and learning about its extraordinary author, NASA engineer Howard W. “Bill” Tindall, Jr. I wrote about this memo five years ago with just a little information on Tindall, but I wanted to expand on that a fair amount this week because without his efforts, I’m pretty certain we would not have reached the moon before that decade was out.

Click for a larger version

I first learned of Tindall in 1989 when I read Apollo: The Race to the Moon by Murray and Cox, which I think will ever remain the definitive Apollo history from the perspective of technical people on the ground, and have since gathered the information that’s included here from 1,700 pages of his memos that the Kennedy Space Center History Office sent to me in 1999, individual memos kindly provided by the University of Houston-Clear Lake from their Johnson Space Center History Collection, some JSC oral histories, and several other books and online resources.

After his earlier work on Mercury trajectories and Gemini rendezvous techniques, Bill Tindall’s parchment-dry title was “Chief, Apollo Data Priority Coordination,” a position created by Apollo Program chief George Low that quite unusually cut across several branches of the Manned Spacecraft Center in Houston. Tindall worked with design engineers, contractors, mathematicians, programmers, mission controllers, and astronauts – everyone, really – to develop and hone the dozens of mission techniques that were used in each one of the twelve distinct phases of lunar missions. Guidance flight controller Steve Bales said of Tindall, “He had a thousand-ring circus going all the time.”

Flight Director Gene Kranz: “Tindall was pretty much the architect for all of the techniques that we used to go down to the surface of the moon. Tindall was the guy who put all the pieces together, and all we did is execute them. If there should have been a plaque left on the moon for somebody in Mission Control or Flight Control, it should have been for Bill Tindall. I respected Bill so much that when the time came for the [Apollo 11] lunar landing, the day of the lunar landing, I saw him up in the viewing room, and I told him to come on down and sit in the console with me for the landing. He didn’t want to come down, but I cleared everybody away and we had Bill Tindall there for landing, and I think that was probably the happiest day of his life. A spectacular guy.”

Late last month, the Johnson Space Center re-opened the painstakingly and beautifully restored Apollo-era Mission Operations Control Room, MOCR 2: https://arstechnica.com/science/2019/06/behind-the-scenes-at-nasas-newly-restored-historic-apollo-mission-control/. How that restoration came about is discussed in detail by JSC Historic Preservation Officer Sandra Tetley and contractor lead Adam Graves in this hour-long episode of “Houston We Have a Podcast”: https://www.nasa.gov/johnson/HWHAP/restoring-the-apollo-mission-control-center

Tindall’s frequent memos – usually two to four a week – were all dictated because Patsy Saur, his secretary, said he’d better learn how because she was not going to lose her shorthand proficiency. They were called Tindallgrams by those who eagerly awaited their common sense, humor, and perfect condensations of discussions and decisions made during the meetings he conducted – and conducted is precisely the right word. Some of those meetings went on for two or three twelve-hour days, with anywhere from half a dozen to a hundred people in the conference room discussing – or, sometimes, shouting and arguing vehemently – and coming to a consensus on every item on the agenda – or, sometimes, accepting Tindall’s final decisions via Tindallgram. Tindall, Buzz Aldrin’s equal in orbital mechanics (Aldrin’s MIT doctoral thesis was “Line-of-Sight Guidance Techniques for Manned Orbital Rendezvous”), once estimated that he spent just 10 to 20% of his time on standard mission techniques and the rest developing finely-detailed “what if” contingency plans, many of which were never needed but some of which came in very handy indeed. The increased peace of mind I’m sure he had as a result was no doubt shared by many because they all knew that there was a precise plan for just about any problem imaginable.

They were after what was right, and everybody was passionate about it. Everybody was young so they were kind of brash and there wasn’t a lot of patience anywhere. So some of those meetings were very, very colorful. Some of the characters were colorful. At the end of this, you were just inundated with all of this stuff you’ve heard. And now what?

And the next day you would get this two-, maybe three-page memorandum from Bill Tindall written in a folksy style, saying, ‘You know, we had this meeting yesterday. We were trying to ask this. If I heard you right, here’s what I think you said and here’s what I think we should do.’ And he could summarize these complex technical and human issues and put it down in a readable style that – I mean, people waited for the next Tindallgram. That was like waiting for the newspaper in the morning. They looked forward to it. I just remember that I’ve always talked to people about this amazing skill.

– Ken Mattingly, Command Module Pilot, Apollo 16

Just how complicated could Tindall’s mission techniques get? Consider that Apollo 11 Command Module Pilot Mike Collins put this CMP Solo Book on a string around his neck a few hours before Armstrong and Aldrin departed for the lunar surface (onboard audio: “Neil, I hate to bother you; could you get my solo book out of R-1 there? Big frapping book, with a bunch of updates on the cover.”). Starting on page 60 are summarized procedures – cheat sheets, if you will – for eighteen different Lunar Module rescue scenarios that Collins might have to execute if his crewmates “never made it to the lunar surface, or if they got there early or late, or departed crooked or straight” (Collins in Carrying the Fire). Some involved Collins diving the 32-ton Command-Service Module from its 60-nautical-mile lunar orbit to as low as they dared – possibly down to 35,000 feet, but I think they would have been a tad more conservative – in order to catch up to the LM if its orbit was higher and slower than the CSM’s, an example of how counter-intuitive orbital mechanics can be.

Here’s a YouTube link to an MIT “Engineering Apollo” class with the sharp and funny Collins in 2015. The interviewer/presenter is Professor David Mindell, the author of Digital Apollo.

Tindall also kept up with the latest scuttlebutt, which at times required that he step in to protect things that needed protecting. For example, when he heard that a NASA high mucky-muck said they should get rid of the Lunar Module’s rendezvous radar to save weight, and that people were beginning to take the idea seriously, Tindall took action to nip that in the bud immediately by writing this memo to George Low, the boss of all Apollo bosses. He didn’t name the official in the memo, but it was Associate Administrator for Manned Space George Mueller who made the flippant suggestion after a visit to Grumman on Long Island, where LM weight reduction was a constant focus for years. After Low read Tindall’s high-energy memo, some memos went between higher mucky-mucks and a few weeks later Mueller’s boss told him, in summary, “Yeah…no.

Sometimes fairly unlikely scenarios gnawed at him a bit – such as whether their re-entry targeting was so good that a Command Module might, by mistake and with a catastrophic result, hit the aircraft carrier that was waiting for its splashdown. His method of dealing with small worries was the same as the large ones: address all eventualities completely through thorough planning. In this case, his memo titled Let’s move the recovery forces a little. (“PAO requirements for good commercial TV” refers to the NASA Public Affairs Office.)

Another of the 1,000+ Apollo memos Tindall wrote from 1966 to 1970 was on the topic of why Apollo 11’s Eagle overshot its intended landing site by four miles. It described how incomplete venting (that is, depressurization) of the docking tunnel prior to undocking caused the Lunar Module to pop like a cork off the Command Module with just a little extra velocity, which in turn caused significant changes in its descent profile. A new rule for subsequent missions required that Mission Control confirm complete depressurization of the tunnel. A related Tindallgram on other venting sources adversely affecting the descent trajectory was titled Vent bent descent, lament!, and he wasn’t shy about making his strong feelings on those vexing vents known to all the top brass at NASA, including chief spacecraft designer – also a culprit – Max Faget, in an unusually all-caps-titled VENTS (“This will either amuse you, waste your time, or just possibly accomplish something great.”)

After a three-day-long “Mission Techniques free-for-all” not even two weeks after Apollo 11, he wrote How to land next to a Surveyor – a short novel for do-it-yourselfers. That and a follow-up memo, in which he revised his previously pessimistic targeting prognosis, detailed new mission techniques that were key to Apollo 12 Commander Pete Conrad being able to set Intrepid down just 535 feet from the Surveyor 3 spacecraft that had, two-and-a-half years earlier, soft-landed on the Ocean of Storms after bouncing twice due to a slightly-too-early engine shutdown.

Click for a larger version

Such pinpoint accuracy was life-critical for later landings, in particular Apollo 17, which landed in the Taurus-Littrow Valley, a box canyon surrounded by mountains on three sides.

Click for a larger version

Here’s an excellent 2015 Apollo 17 documentary in two parts: https://www.youtube.com/watch?v=vIGbOoZzlYI https://www.youtube.com/watch?v=SQOEC9gHpmA

Oh, yeah…for a period of about a year in 1966-67, Tindall, who grew up in Scituate, Massachusetts, flew up to Cambridge from Houston for two or three days every week to help organize, focus, and speed up – effectively manage, sometimes in a blunt manner – the MIT Instrumentation Lab’s previously somewhat free-form development of the COLOSSUS and LUMINARY software for the Apollo Guidance Computers (AGC) in the Command and Lunar Modules, respectively. (He visited often enough that he sent out a TripAdvisor-style memo every now and then.)

Early on, Lab engineers reported, to Tindall’s great alarm, that the Command Module code was about 30,000 bytes in excess of the 72,000 available in the AGC and the Lunar Module software was around 10,000 over its 72,000. 13 October 1966, the day Tindall directed them, in person, to eliminate much duplicated code that he had found, and to cut several elegant but non-essential and hence memory-wasting routines, became known to those in the Instrumentation Lab as “Black Friday.” Two weeks after Black Friday, he discussed his strategy in this memo, which began with the important point that “There are a number of us who feel that the computer programs for the Apollo spacecraft will soon become the most pacing item for the Apollo flights.” Despite the initial hard feelings at the Lab, they did what he asked, and over time came to realize just how beneficial his involvement was to their work – and best of all, that work was ready when it needed to be.

Here’s a profile of Margaret Hamilton, who, two years after the Lab’s early difficulties, became leader of the Apollo spacecraft software development effort: https://www.smithsonianmag.com/smithsonian-institution/margaret-hamilton-led-nasa-software-team-landed-astronauts-moon-180971575/

In late 1965 just before his work on Apollo began, the New York Times profiled Tindall in a brief Gemini 6/7 sidebar titled “Rendezvous Planner Howard W. Tindall, Jr.” (reprinted in the January 1966 Brown Alumni Monthly here), but Charles Fishman, who contacted me while researching his new book, One Giant Leap: The Impossible Mission That Flew Us to the Moon, says that when Tindall died in 1995, not one newspaper in the US ran an obituary. It’s even difficult to find any photographs of him bigger than a postage stamp, but here are a couple: below, one in his office (a screenshot from episode 3 of the also excellent “Moon Machines” series, playlist here: https://www.youtube.com/playlist?list=PLTu8nanTJo7GvulBxz9JT9JcXeXimM1Vr) and he’s in the center of this photo taken during Apollo 13, chin in hand, looking at papers – some probably written by him.

I’ve always thought that more people ought to know about this remarkable man. To paraphrase him, if you are still with me, hardy reader, now you do.

Bill Tindall; click for a larger version

I think it’s safe to say he thoroughly disliked inaccuracy and inexactitude, which may be reflected in the “H. Timdell” [sic] name I noticed taped to the wall behind him in that photo, the misspelling perhaps from some conference he attended. I’ve no evidence for it, but I like to think he kept it up there to point out to visitors at appropriate moments, maybe with a raised eyebrow and a little flourish of sarcasm.

We’d all get in there and defend our [computer] requirements, and then Tindall would cut them. And then we’d cuss him. And Tindall would grin, and cuss back, and laugh his loud, infectious laugh, and keep right on going.

– Apollo Flight Director Cliff Charlesworth

We weren’t working overtime, we were playing!

– Bill Tindall

Edited 9 August 2019 to add: My theory above about that misspelling on his wall is now inoperative…defunct…shot down. The Johnson Space Center History Office has kindly found and sent me the original of that official photo along with nine others of Tindall from 1965-1979, which I’ve just posted here: https://finleyquality.net/The-ringmaster. Some deductive reasoning on the uncropped version of that one that they sent reveals the much more likely source of “H. Timdell” [sic].

Edited 21 August 2019 to add: I just happened upon this tidbit while reading Harrison Schmitt’s 1999 Johnson Space Center oral history interview. Twenty-seven years after his Apollo 17 mission, Schmitt emphasized how important Tindall’s memos were, not just at the time but for purposes of mission planning in the future (emphasis mine):

Well, Frank Borman approached me, asked me if I would do the lunar orbit flight planning for their effort. And that meant that I began to interact with [Howard W.] Tindall’s group, the Flight Operations Planning group that met weekly that really was the focus of all of the operational planning for a particular mission. They were looking at all the missions, but the one up was the one they were concentrating on. And that’s another tremendous resource.

And I’m not sure where there is a complete collection of what were called Tindallgrams. They were his summary of each of those meetings. I have a partial collection at the University of New Mexico in the files there. Whether there would be a complete collection or not, I don’t know. But somebody ought to make a very, very specific effort to get a complete collection of the FOP minutes, Tindallgrams, and to get those in some kind of form and bound. Because that is a resource that should not be lost. I can understand it’s hard to put together. I hope somebody has been able to do that.

Apollo 17 documentary

I happened upon an excellent 104-minute documentary from 2015 I’d never heard of before, titled “The Apollo Experience: Apollo 17” and linked below. It’s among the best single-mission Apollo documentaries I’ve seen and I figured I should mention it here because it has to be pretty darned obscure for me to not know of it.

I have the complete lunar surface videos from the mission – the boxed set pictured at the end of this post – but this documentary puts EVA highlights in context with explanatory captions and follows the mission from training to splashdown. The archival footage throughout is of the highest quality I’ve seen and quite a treat on a fifty-inch television.

The overall quality is high enough that I forgive the engine noise the producers added to some radio transmissions, along with other low-key add-ons like electronic “beep-beep” effects. They did them in a fashion subtle enough that, while I knew immediately that they were their additions, I wasn’t compelled to say “Wut?” and put on my just-ate-a-lemon face.

Timeline’s YouTube channel is worth exploring as it has hundreds of other history documentaries. Some, like this one, come from obscure satellite channels, but it appears a large percentage of their content originally aired on Channel 4 in the UK.

Don’t get me started

The excerpt below is from the site of Apollo 17 Lunar Module Pilot and geologist Harrison “Jack” Schmitt, and it’s the most…well, invigorating description of a jump start I’ve ever read.

This goes hand-in-hand with my article on the explosive guillotine in the Lunar Module because Schmitt describes an emergency scenario that Apollo crews planned for and practiced in which the launch sequence has failed: The guillotine has not fired, the four explosive bolts holding the two stages together have not exploded, and the ascent engine has not started. This is one of several contingency methods mission planners worked out.

Bear in mind as you read Schmitt’s explanation that this would be happening after they had tossed their Portable Life Support System backpacks out onto the lunar surface to save weight during the ascent, and after they had closed up the LM and repressurized the cabin in preparation for departure from the lunar surface.

It’s not often I find something about Apollo I’ve never heard before, and this one is boggling. I bolded the last bit of the excerpt because that’s the point when the ramifications sank in and my eyebrows shot off.

Wednesday, November 8th [1972], brought on our last full Lunar Ascent Mission Simulation involving Mission Control in Houston. Six weeks hence, we hoped we would be undertaking the real thing and departing the Moon at the conclusion of a highly successful exploration effort. This “Sim” required over three straight hours in LMS2, including the debriefing with SIMSUP (Simulation Supervisor). Failure or degradation of the primary guidance or engine ignition subsystems constituted the primary concerns addressed in Ascent Simulations. We particularly worked through several scenarios involving failure of the various software-initiated means of igniting the Ascent Engine.

Schmitt in Lunar Module Simulator 2. NASA photo ap17-KSC-72PC-539

We did not have a great deal of concern about our Challenger Lunar Module, like all the others before it, having just one Ascent Engine, because, in fact, it was at least two engines that just looked like one. Only the solid metal fuel and oxidizer injector ring and the exhaust nozzle below that ring did not have identical, that is, “redundant” components that would function even if a primary component failed. No one could imagine a failure mode for these non-electronic and solid pieces of hardware.

If all internal Ascent Engine ignition options actually failed, and many such options existed to fall back on, we also had a set of jumper cables that could be used as a next to last backup to ignite the Ascent Engine. These were called the “ED/APS Emergency Jumper Cable” and would use power from an independent Pyrotechnic Battery in the Descent Stage to open the engine’s fuel and oxidizer valves and fire the pyrotechnic cable and bolt cutters that would simultaneously separate us from the Descent Stage.

To use the second of these cables, however, one of us would need to egress Challenger in order to access a regular Descent Stage battery. Integrity checks of our suit would determine which one of us would perform this emergency EVA. If Cernan’s pressure suit did not pass its pre-egress checks sufficiently to permit egress with the jumper cables, we would change positions in the cabin, a tough task on its own. As we would have already jettisoned our Portable Life Support Systems, it would be necessary to use the OPS (Oxygen Purge System) we had retained to support the EVA that Evans would perform to retrieve film canisters from America’s Scientific Equipment Bay after leaving lunar orbit for home. The 8000 psi oxygen bottle in the OPS could provide a maximum of 30 minutes of oxygen and air-cooling once activated. There would be no water cooling, however, without a PLSS.

With the Challenger’s cabin depressurized, the winner of the integrity check contest would take one end of the pair of cables out the hatch and down the ladder and move to QUAD III where a battery could be accessed. He would then tear away the Kevlar covers and attach the color-coded pair of cables to the positive and negative terminals of a battery and then return to the cabin. At the optimum liftoff time for ascent into a rendezvous sequence with Evans, Cernan would attach the cables to two circuit breakers near his left shoulder. This action would supply instant power to the two sets of hypergolic (ignite on contact) hydrazine and nitrous oxide valves in the Ascent Engine. Once power reached these valves, they would open and lock open. With opening of these valves, a signal would go to the cable and bolt cutters. We would be instantly on our way into lunar orbit, still in an un-pressurized cabin, dragging our jumper cables behind us. Once back in lunar orbit, we could clear and seal the hatch and pressurize the cabin.

The aim of this emergency EVA was to bypass relay boxes, internal wiring, and the Explosive Devices control panel in order to get power directly from a descent stage battery.

The descent stage explosive device battery, aka the pyro battery, in question was near the front, highlighted here. This procedure bypassed that in favor of a regular descent stage battery.

In later LMs, Apollo 17’s included, five descent stage batteries were at the back of the LM, shown below in an illustration from the Lunar Module LM 10 through LM 14 Vehicle Familiarization Manual [link to PDF]. Two of the five had low voltage taps; the jumper cable would be affixed to one of those.

I’m glad they never had to do this, but it actually sounds like it would work. You might think, “Yeah, except for all the cardiac arrests and such”, but you have to remember that these guys were cool customers. Witness the fact that they practiced for this instead of climbing out of the sim and seeking the nearest bar posthaste when told of the method.

Schmitt says this method was the “next to last backup”, which makes me wonder intensely what on Earth Moon the last backup was. Spit and baling wire?

Edited 24 August 2018 to add: The entire emergency EVA procedure is detailed in Apollo Operations Handbook/Lunar Module/LM 11 and Subsequent/Volume II Operational Procedures, available on the Apollo Lunar Surface Journal site. See section 5.4.25 Loss of ED Sub-system.

https://www.hq.nasa.gov/alsj/LM11HandbookVol2.pdf

I think I found the answer to my “What’s the last backup?” question there: The other choice was to quickly get to the rover – in the case of Apollo 17, parked about 158 meters away for best liftoff camera coverage – start it up, drive it back to the LM, and hook up to one of its batteries. Now that would really be a jump start for the ages, but I think far less preferable considering there was just 30 minutes of oxygen available in the OPS.

The moon is a messy place

In which I discuss moon suits, moon dust, moon models, and moon stories

“Smells like someone just fired a carbine in here.”
– Apollo 17 Commander Gene Cernan on smelling moon dust inside the Lunar Module

I finally got around to making my 1/6th scale Gene Cernan figure look a little more realistic. Brand new, it looked like this:

Apollo 17 Commander Capt. Gene Cernan, the last man on the moon, in 1/6th scale

I never quite liked the pristine look of it because the only time it was that clean was the day it arrived from the manufacturer, ILC Dover, a division of Playtex at the time the suits were designed. Yes, Playtex designed the 21-layer Apollo suits. Here’s the figure after I applied a fair amount of graphite powder using two different brush sizes:

Now you may think I went a little over the top with that, but I didn’t. My variation is probably about what Cernan’s A7LB suit looked like after two of the three seven-hour moonwalks he and Jack Schmitt made. Below you can see what his suit looked like after their third and final moonwalk on 13 & 14 December 1972 – and this was after they spent quite a while brushing each other off before re-entering the Lunar Module for the last time.

Click for a larger version

The large brush they used for the suits – and the lunar rover – was six or seven inches wide. They also had a smaller brush for camera lenses and such. You can see both in this compilation. Gene Cernan raises his gold-plated visor here while cleaning the rover’s camera lens.

Moondust is funny stuff – fine, powdery, almost like snow, and it smells of burnt gunpowder when it’s on the moon. That smell goes away on prolonged contact with normal air, so the returned samples no longer smell of anything. They don’t really know why it smells like gunpowder, but there are some theories.

Excerpts from the Apollo 17 Technical Crew Debriefing [corrected link to PDF] on 4 January 1973, during which the problems with dust came up a lot:

“Dust – I think probably one of the most aggravating, restricting facets of lunar surface exploration is the dust and its adherence to everything no matter what kind of material, whether it be skin, suit material, metal, no matter what it be and its restrictive friction-like action to everything it gets on. For instance, the simple large tolerance mechanical devices on the Rover began to show the effect of dust as the EVAs went on. By the middle or end of the third EVA, simple things like bag locks and the lock which held the pallet on the Rover began not only to malfunction but to not function at all. They effectively froze. We tried to dust them and bang the dust off and clean them, and there was just no way. The effect of dust on mirrors, cameras, and checklists is phenomenal. You have to live with it but you’re continually fighting the dust problem both outside and inside the spacecraft. Once you get inside the spacecraft, as much as you dust yourself, you start taking off the suits and you have dust on your hands and your face and you’re walking in it. You can be as careful in cleaning up as you want to, but it just sort of inhabits every nook and cranny in the spacecraft and every pore in your skin. Although I didn’t have any respiratory problems, I think the LMP, which he can comment on later, had some definite local respiratory problems right after the EVA – due to dust in the cabin.”
– Commander Gene Cernan

“Dust – We’ll just talk about in-cabin dust. After the first EVA, there was considerable dust in the cabin. It would be stirred up by movements of the suit and the gear that we had. Almost immediately upon removing my helmet, I started to pick up the symptoms that you might associate with hay fever symptoms. I never had runny eyes or runny nose. It was merely a stuffiness in the nose and maybe in the frontal sinuses that affected my speech and my respiration considerably. After about 2 hours within the cabin, those symptoms gradually disappeared. By morning of the next day they were gone completely. After the second and third EVAs, although I’m sure the dust was comparable, the symptoms were not nearly as strong as after the first EVA. That was as if I either developed a mucous protection of the affected areas or had some way or another very quickly developed an immunity to the effects of the dust.”
– Lunar Module Pilot Jack Schmitt

Here’s James Burke, prime Apollo reporter for the BBC, wearing – and removing, one by one – all the components of the A7L spacesuit. The A7L was used through Apollo 14 and the more advanced A7LB, with essentially doubled consumable capacities that allowed for seven-plus-hour moonwalks, was used on Apollo 15, 16, and 17.

That report was prepared prior to the moon landings. In 1979, Burke did an excellent ten-year anniversary documentary that explains a lot of the workings of lunar missions better than most.

I think this is a great time to start reading Spacesuit: Fashioning Apollo. I’ll do that tonight.

I also now have a lighted display case for the Dragon Lunar Approach model I have at home. In normal light, it looks like this:

In the dark with the case’s inbuilt lights turned on, this is closer to how the pair would appear in the sunlight of deep space during the translunar coast:

Finally, a treat: Several years ago on BBC Radio 4, Jeanette Winterson did a fascinating ten-part, 150-minute series about the moon from many perspectives. It’s called “The Inconstant Moon” and you can listen on her site here. Quite a pleasant way to spend a couple of hours.

Radio 4 used this graphic for the programme: