/sensors
With Zak Nienstedt and Sam Miller.
Sensor Suite for ML1 and ML2
Note: I'm going to be intentionally vague for portions of this section, both to give wide berth to my NDA, and because friends and colleagues are still working hard on the dream I still passionately believe in.
You know that feeling when a warning light appears on your car’s dashboard, and you know you should take care of it, but today you’re just too busy… That was Magic Leap about designing a sensor suite during the prototyping phase of our first product. We had stopgap solution that allowed our various teams to function. But there was a collective pushing to the back of our minds that the stopgap couldn’t possibly survive until product lock.
I had brought some concerns about lead times of custom sensors to my boss — we were running out of time to replace the stopgap. Perhaps that's why later that day I was added last minute to an unnamed meeting. I walked into a dimly lit room. It was empty save a smattering of Magic Leap's heavy hitters shifting awkwardly on their stools in silence. In retrospect, I appreciate the cliche spy movie aesthetic as that moment was my "your mission, should you choose to accept it..."
We were told that we had been hand selected by the SVP to be a strike team to cut through the institutional inertia, build consensus across teams with wildly different agendas, and come up with a world-class sensor architecture to enable Magic Leap's ambitious goals. We were warned that the people in this room would become our new best friends, and that we'd be spending many a 3AM in diners drawing on napkins. They were correct -- of those present, I wound up living with one, chasing a startup with another, and a third officiated my wedding.
We started with nothing but a long list of product requirements. We used these in conjunction with the limits to human perception and motion (measured via user studies). Working backwards from there, we had to determine what kind, which, and how many sensors to meet the functional and performance requirements. We architected sensor fusion algorithms. We balanced off-the-shelf against custom components. We determined via home-rolled simulators the required pointing angles, stereo overlaps, mechanical constraints, Field of View, f#, MTF, manufacturing tolerances, Depth of Field, lens designs, distortions, uniformities, shutter mechanisms, readout times, dynamic ranges, sensor noises, exposure times, bias currents, calibration process variations, and resolutions for each visual, depth, and inertial sensor.
Until I get approval to share technical details of the design process, a text montage of the design process will have to suffice. Envision:
Strapping tech-ed out sweat bands to unsuspecting coworkers. Couple's counseling between teams. Nerf wars (for the data of course). A helmet with pounds of cameras strapped on it. Executives whispering apologies into their cellphones on hour six of a one hour design review. Terabytes of data. Conference calls at 11pm where we planned out every second of compute on our clusters until the deadline. Making every team hate us. Making most teams love us. Swearing that one of these days I'll actually intuitively get quaternions. Conference rooms with every cm**2 of whiteboard occupied. Crowding around a computer and holding our breath as the progress bar crawls forward, calculating whether our sensor simulator we had based months of work on actually represents reality. A climactic signoff by all of senior leadership. And finally, an anticlimactic peacefulness punctuating the chaos.
While many amazing stories and friendships arose from ML1's sensor design, our biggest accomplishment is how relatively uneventful (read: mature) the ML2 design process was. We started early, were able to reuse work, adapted to the new technologies, utilized the interdepartmental relationships forged the first time, and submitted an architecture ahead of schedule with no drama or late nights. It is unremarkable, iterative, and performs damn well.
I miss the frenetic kids-in-a-garage days more than I care to admit.