Crazy kaiju-robot antics only hint at Nintendo Labo’s true potential

After writing up Nintendo’s Wednesday reveal of its new Labo playsets (coming April 20 to the US and Japan and April 27 to Europe), I realized I’d forgotten to add an important word to the article’s introduction: “what.”

More specifically, the drawn-out, question-marked version I should have shouted when the product’s reveal video played out. (“Whaaaaat?!”) I’m a big fan of Nintendo’s physical-toy era in the ’60s and ’70s, back when company legend and Game Boy creator Gunpei Yokoi came up with engineering wonders like the Ultra Hand and the Ten-Barrel Puzzle. As a result, I was immediately charmed by the physicality and toy-controller possibilities of the reveal video, which included everything from a motorcycle steering chassis to a 13-key piano to a string-loaded fishing rod—all built by players with a mix of pre-cut, pre-marked cardboard, sensing stickers, plastic, string, and more.

But then I began wondering: exactly how does everything work with Nintendo Labo? In particular, what the heck is going on with Labo’s most insane offering: a full-body robot suit?

Player, piano

Nintendo has not answered precisely how the Labo Robot Kit will work, either in its reveal videos, Labo’s official sites, or hands-on impressions posted by other outlets. With the limited info I’ve gathered, I’m taking it upon myself to offer educated guesses while commenting on why I have a serious case of Labo love.

We do know quite a bit about Labo based on hands-on reports that went live at the same time as Nintendo’s Wednesday video. The Labo Variety Kit was demonstrated at length to various outlets, and the “how” of the Variety Kit’s cardboard piano—which recognizes 13 distinct keys and a number of modulating knobs—was explained as follows by The Verge:

The Joy-Con that slots into the back [of the Labo piano] has a camera, which can see the back of the keys so that it knows which ones you’re pressing and then relays that information to the Switch. The sound-modifying knobs, meanwhile, each have distinctive stripes that are associated with their respective sounds, so that the camera can tell them apart.

To be clearer: the right-side Joy-Con, which ships with every Nintendo Switch, comes with an infrared camera and four additional infrared sensors. These are apparently enough to create Labo’s distinct combination of positional and visual data. When fed by patterns on the cardboard parts and “reflective” stickers, they can individually recognize no less than 14 simultaneous points of interactivity.

We have to wait for more direct access to all of the Labo creations (which Nintendo calls Toy-Cons) to break down exactly how every single one of them works. When we get our hands on Labo later this year, we will do just that. For now, let’s focus on the Labo Robot Kit, which was not shown off during any hands-on press previews—but appeared for long enough in Labo’s debut video to reveal juicy information.

We can start by looking at the Labo video’s footage of the cardboard piano’s assembly. Labo walks its players step-by-step through the process of building a Toy-Con, and it includes see-through 3D models of the construction every step of the way to teach users (assumedly, our world’s future engineers) exactly how this stuff works. By the end, builders have gotten the piano pretty much fully built, at which point they must insert the right-side Joy-Con into the back of the piano, so that its IR camera rig can see the piano’s full innards.

The reveal video shows Labo instructing players to test the piano’s buttons and dials, in order to test the IR sensor. This screen, which includes a touchscreen option for kids to move the camera around and see how the piano is working, actually shows 19 distinct sensing positions, including 13 purple “keys,” a pair of dials (one red, one green) directly above the keys, and four larger boxes. (We still don’t have full information about exactly which of these keys, knobs, and other elements must have sensor-boosting stickers attached.)

This wealth of data, above and beyond a mere 13-key rig, is important as we transition to the Labo Robot Kit.

Go-go Labo rangers

Nintendo’s official Labo Robot Kit site includes images of the full kit’s basic, exterior design. In the first image of the above gallery, the left-side image shows two handheld wands made entirely of cardboard, while two foot clips made of plastic are shown as connected directly to the backpack, not fully extended to the floor. That’s made clearer on the model’s feet on the right-side image. Also in that right-side image, both the handheld and foot-clipped parts are connected to string that runs through the backpack itself.

If we go back to the reveal video, we see a brief shot of the backpack’s exposed innards, before a final piece of cardboard is slapped onto the back. Four apparent levers can be seen, each with a white strip at different levels. We can tell based on the official product image that each limb’s string runs through the top of the cardboard backpack, aligned with each of these levers. As you move an arm or a leg, each white strip moves up and down.

The backpack’s back-side flap of cardboard includes a holder for the right-side Joy-Con, and again, its IR sensor points at the innards, including these white strips. Should there be any doubt that these are the backpack’s primary trackable points of data, notice the video’s footage of the Robot Kit’s accompanying game. Its robot hero has four visible levers on its back, mirroring the same design that players wear.

Additionally, the non-camera Joy-Con is mounted to a small cardboard-and-plastic headset, which a player wears for some sort of control in the game itself. This could be meant for head-tracked camera movement in the game, or for directing a player’s motion or attacks. But it’s definitely not up there just to make a kid look silly.

One question remains: what other, finer points of data might Labo’s Robot Kit be equipped to handle, a la the cardboard piano’s wealth of sensing data? The fact that Nintendo didn’t demonstrate a working version to the press could mean that there’s more to the kit; the best evidence to support this guess is the backpack’s use of three large holes on both its left and right sides. These may accommodate additional connections, either by string or some other doo-dad. There’s also the possibility, of course, that this four-lever system is not yet optimized enough for public testing.

Coming soon: a “more-power” glove?

Either way, the beauty of the IR-tracked backpack is its sheer design flexibility. If you were asked to build a fully tracked four-limb robot suit from scratch, meant to interact with a video game system, you might imagine no less than four hardware-connected sensing rigs to track basic movement—or more if you want to account for, say, ball-and-socket joints. But Nintendo will only employ one IR sensor to record four distinct limbs’ movement. There’s work to be done to engineer the rest of the physical rig, of course, but this “single-sensor” system saves users the trouble of, say, ordering laser- and wire-connected parts from Alibaba.

That’s a lot of design modularity for Labo’s future. Labo essentially declares that the Nintendo Switch already includes enough sensors and electronics. From there, Nintendo, or its fans, can engineer entirely different build-it-yourself kits based on this base with nothing but cardboard and stickers. The possibilities are wild. I’ve already imagined one idea, based on my own VR experiences. With an angled, wrist-strapped glove, Labo 2.0 could aim an IR sensor at a single human hand and, at the very least, track the basic movement of individual digits, if not the fingers’ bending at knuckles. (A second Joy-Con could join the party to determine angle and rotation, which might deliver convincing spatial hand tracking.)

Our last image gallery, directly above, includes five Toy-Cons that do not appear to have announced releases yet. Since it’s advertising these preview concepts early, Nintendo clearly sees a future in the Labo line. Its control possibilities really could go in as many wacky directions as I’ve just guessed with my own Toy-Con Glove idea.

Already before launch, Nintendo’s Labo approach feels like a much better path forward in the gimmick-filled control world than what’s come before. The era of locked-down, $90-and-up control peripherals has long passed. Players generally do not want to be saddled with a pricey, space-filling control mechanism that only works for a few games—a fact that spelled doom for everything from Kinect to Guitar Hero. The Switch has succeeded, in part, by giving players a familiar suite of controls to play as they see fit, either at home or on the go. The system also just so happens to have just enough sensing tech built in to accommodate crazy control styles… for those who want to buy add-on kits and construct their own Toy-Cons. Everyone wins. It’s brilliant, even before it exists.

Listing image by Nintendo

https://arstechnica.com/?p=1246487