Something Solid In Touchy Feely

The Age

Tuesday February 27, 2007

ADAM PENENBERG

Jefferson Han took a long look at his fingers through a glass of water, then redefined the way we will all get access to computer technology, reports Adam Penenberg.

JEFFERSON HAN, a pale, bespectacled engineer dressed in Manhattan black, faced the thousand or so attendees on the first day of TED 2006, the annual technology, entertainment, and design conference in Monterey, California. The 30-year-old was little more than a curiosity at the confab, where, as its ad copy goes, "the world's leading thinkers and doers gather to find inspiration". On that day, the thinkers and doers included Google gazillionaires Sergey Brin and Larry Page, e-tail amazon Jeff Bezos, and Bill Joy, who helped code Sun Microsystems from scratch. Titans of technology.

Then Mr Han began his presentation. His fingertips splayed on the cobalt blue 90-centimetre-wide display before him and traced playful, wavy lines. He conjured up a lava lamp and sculpted floating blobs that changed colour and shape based on how hard he pressed. ("Google should have something like this in their lobby," he joked.) With the crowd beginning to stir, he called up some vacation photos, manipulating them on the monitor as if they were actual prints on a tabletop. He expanded and shrank each image by pulling his two index fingers apart or bringing them together. A few oohs and aahs bubbled up from the floor.

Suppressing a smile, Mr Han told the assembled brains trust that he rejects the idea that "we are going to introduce a whole new generation of people to computing with the standard keyboard, mouse, and Windows pointer interface". Scattering and collecting photos like so many playing cards, he added, "This is really the way we should be interacting with the machines." Applause rippled through the room. Someone whistled.

Mr Han was far from finished. He pulled up a two-dimensional keyboard that floated slowly across the screen. "There is no reason in this day and age that we should be conforming to a physical device," he said. "These interfaces should start conforming to us." He tapped the screen to produce dozens of fuzzy white balls, which bounced around a playing field he defined with a wave of the hand. A flick of a finger pulled down a mountainous landscape derived from satellite data, and he began flying through it, using his fingertips to swoop down from a global perspective to a continental one, until finally he was zipping through narrow slot canyons. He rotated his hands like a clock's, tilting the entire field of view on its axis like an F16 in a barrel roll. He ended his nine-minute presentation by drawing a puppet, which he made dance with two fingers. He basked in the rock-star applause.

Six months later, after TED posted the video on its website, the blogosphere got wind of the presentation. Thousands posted the video on their sites or pointed to it on YouTube, where it was downloaded a quarter of a million times.

In this Googly age, it only takes a random genius or two to conceive of a technology so powerful that it can plough under the landscape and remake it in its own image. People are betting that Jefferson Han is one of them.

For as long as he can remember, Mr Han, a research scientist working out of New York University's Courant Institute, has been fascinated by technology. The son of middle-class Korean immigrants who emigrated to America in the 1970s to take over a Jewish deli in Queens, Mr Han began taking apart the family TV, VCR, "anything that was blinking", at the age of five. At summer camp, he hot-wired golf carts for nocturnal joy-rides and fixed fellow campers' busted Walkmans in exchange for soda pop. He was 12 when he built his first laser.

At Cornell University he studied electrical engineering and computer science. He skipped out on his senior year without graduating to join a start-up that bought a video-conferencing technology he developed while a student. A decade later, he's poised to change the face of computing.

Until now, the touch screen has been limited to electronic buttons that recognise one finger at a time. Mr Han's touch display redefines the way commands are given to a computer: it uses both movement and pressure - from multiple inputs, whether two fingers or 20, to convey information to the silicon brain under the display. Already industries and companies as diverse as defence contractor Lockheed Martin, CBS News, Pixar, and unnameable intelligence agencies have approached Mr Han to get hold of his invention. And, no surprise, he has formed a start-up company to market it, Perceptive Pixel.

"Touch is one of the most intuitive things in the world," he says. "Instead of being one step removed, like you are with a mouse and keyboard, you have direct manipulation. It's a completely natural reaction to see an object and want to touch it."

At a private demonstration at NYU, the 90-cm-wide (36-inch) drafting table he used at TED has evolved into a giant screen: two 2.4-metre by one metre panels. The screen is smudge resistant and, the inventor says, "peanut butter-proof".

Mr Han teaches me the one pattern I need to know - a circular motion akin to a proofreader's delete symbol, which brings up a pie-chart menu of applications. I poke at it, and suddenly I'm inside the mapping software, overlooking an arid mountain range. Spread two fingers apart, and I'm zooming through canyons. Push them together, and I'm skying thousands of feet above. I'm not just looking at three-dimensional terrain, I'm living in it: I'm wherever I want to be, instantly, in any scale, hurdling whole ridgelines with a single gesture, free-falling down to any rooftop in any city on earth. There's nothing between me and the data: no mouse, no cursor, no pull-down windows. It's seamless, immediate, easy.

We're interrupted: a producer from the Ellen DeGeneres Show called. Ever since Jefferson Han became a web phenomenon, he has been receiving all sorts of offers. An official from SPAWAR, a subdivision of the navy focused on space and naval warfare planning, queried him about collaborating. A producer from CBS News wondered how to make use of the touch screen for election coverage. A dance DJ asked if he had a product to spin music at clubs.

Before reinventing the touch screen, Mr Han was just another dotcom refugee at a crossroads. BoxTop Interactive, an e-services firm he worked for in Los Angeles, had just flamed out. He landed a research position at the Courant Institute, where he has been for the past four years.

MULTI-TOUCH isn't his only project. Modelling his work on the brain of a bee, he has been looking for ways to build a flying camera that would be able to find its way over long distances. He is also tinkering with a plan to create an autonomous robot vehicle that can traverse terrain by learning from its own experiences.

Another project is to scan materials so they are faithfully reproduced digitally. His kaleidoscope-like idea takes a fraction of the time Hollywood's best computer animators currently need.

He brought a similarly pragmatic do-it-yourself attitude to his study of touch-screen technology. He knew other researchers were working on interactive walls and tabletops, and there were a number of art pieces. But that was about it. The concept hadn't advanced much from where it was in the 1980s. "Most of it was designed with toys in mind," Mr Han says. "They weren't asking themselves what purpose it served. I wanted to create something useful."

Inspiration came from an ordinary glass of water. He noticed when he looked down on the water that light reflected differently in areas where his hand contacted the glass. Physicists call the principle "frustrated total internal reflection".

It took him just a few hours to come up with a prototype: LEDs on the side of a piece of clear acrylic, with an infra-red camera on the back. When he placed his fingers on the makeshift screen, some light ricocheted straight down and the camera captured the light image pixel for pixel. The harder he pressed, the more information the camera captured.

"It's like a thumbprint scanner, blown up in scale and encapsulating all 10 or more fingers. It converts touch to light."

He began coding software to demonstrate some of the touch screen's capabilities. Meanwhile, Philip Davidson, an NYU PhD candidate, got excited about the project and quickly became its lead software developer.

The first thing the pair did was to modify NASA World Wind, a free Google Earth-type open-source mapping program. (Mr Han figured the military would be keen on anything that works faster, since split seconds mean the difference between life and death.) Then they created the photo manipulator, which lets you upload pictures from Flickr or anywhere else on the w eb (it can also make 2-D images appear as 3-D).

A taxonomy tool makes it a cinch to navigate the illustrated branches of the Linnean classification system, from animals and plants down to every known species, and see on one screen how these families are structured and interrelated. (They are thinking of extending it to genealogy and social networks.)

Multi-dimensional graphing and charting help you visualise spreadsheet data and move them around from one point in time to another, while Shape Sketching lets you draw on-screen as easily as you can with a pencil on paper, then animate these shapes instantly.

"As computers have become more powerful, computer graphics have advanced to the point where it's possible to create photo-realistic images," Mr Han says. "The bottleneck wasn't, 'How do we make pixels prettier?' It was, 'How do we engage with them more?"'

This doesn't mean touch screens will completely replace the computer mouse, Qwerty keyboard, or traditional graphic user interface (or GUI) any more than cinema made live theatre disappear or television supplanted radio.

Jefferson Han really doesn't know how his mapping software, photo manipulator, or any of it will ultimately be used - these applications are really proofs of concept, not ends in themselves. "When unexpected uses emerge that no one ever thought of, that's when it gets exciting and takes off," says Don Norman, a professor at Northwestern University and author of Emotional Design.

Meanwhile, wherever touch-screen technology leads, the Han device will face stiff competition. Microsoft has been working on its own version, TouchLight. GE Healthcare is using TouchLight for 3-D imaging: surgeons can swipe their hands across the screen and interact with an MRI of a brain, peel away sections, and look inside for tumours (retail price: $64,000).

Mitsubishi's DiamondTouch table is a collaborative tool for business that allows a group of people to interact at the same time via touch screen. Canada-based Smart Technologies has created a nice niche selling interactive whiteboards to universities, corporations, and the US military for briefings. Panasonic has been developing wall-size touch-screen displays, as has consulting firm Accenture, whose interactive billboards are already enticing passengers at O'Hare and JFK airports.

But Jefferson Han isn't exactly worried. He says he has a lot of deals in the pipeline. With the cost of cameras and screens plummeting, it is inevitable that interactive displays will be built into walls and in stores, in schools, on subways, maybe in taxis. In fact, a screen could be as thin as a slice of wallpaper.

Not everyone is sold on the idea. Ben Shneiderman, a computer science professor at the University of Maryland and a founding director of the Human-Computer Interaction Laboratory, asks, if you are standing in front of the screen, how would people behind you be able to see what you're doing?

But criticisms like these are a million light-years from the inventor's mind. "I want to create an environment where I can create technology, get it into the hands of someone to market it, and move on to other technologies so I can keep innovating," he says.

-- FAST COMPANY

LINK

? cs.nyu.edu/~jhan/ftirtouch/index.html

© 2007 The Age

Back to News Index | Back to Home