“A pair of patent applications published by the U.S. Patent and Trademark Office on Thursday reveal Apple is investigating the use of heuristics in its mobile devices to deliver a more realistic digital representation of handwriting, as well as selective touch input that ignores extraneous touch events,” Mikey Campbell reports for AppleInsider.
“Apple’s ‘Handwriting capture techniques’ describes a set of rules used to render a digital replica of a person’s handwriting, or more specifically, how a touch-capable device translates user input into a representation visually similar to handwriting,” Campbell reports. “The second touch-related application published on Thursday was Apple’s ‘Region Activation for Touch Sensitive Surface,’ which describes a system in which extraneous touch events recognized outside of so-called ‘active regions’ are ignored.”
Much more in the full article, including Apple’s patent application illustrations, here.
Nobody seems to have noticed the ‘selective touch’ recognition, but this one tells me all I need to know about future of desktop OS, and it is that it will be entirely (multi)touch UI. As I have been saying for over a year now, over the next few years, Apple will converge iOS and Mac OS X into one system, with one set of apps for both OSes, and mouse will join typewriter in the museum of office equipment.
The primary challenge for desktop iOS is the size and position of the touchscreen. Obviously, current layout (with keyboard / mouse / display) wouldn’t work due to the well-known “gorilla arm” syndrome. However, flat, or slightly tilted display can be as large as we want it and be the ultimate in usability. Selective multi-touch is precisely the type of interface necessary for interacting with a device that resembles traditional, physical desktop, where an arm rests on the surface while fingers interact with the work.
Probably no more than five years will be enough to essentially completely transition to iOS and forever kill the original “desktop” UI paradigm. For all legacy applications, there will likely be a way to emulate the old system with a pointing device (represented by the mouse arrow), allowing for mouse hover / mouse-over, right-click and similar old concepts. With their increasingly robust desktop muscle, Apple will likely easily strong-arm Adobe, Microsoft and other big software players to once again migrate to the new environment (the way they migrated from 64k to PPC, from System 9 to OS X, from Carbon to Cocoa, from PPC to Intel, from 32-bit to 64-bit…). Granted, this migration requires re-writing the UI paradigm, but that may actually end up being easier than keeping the UI and rewriting back-end code from PPC to Intel. If they did it before, with much fewer Mac users, they surely will drag themselves into it this time around as well.
Predrag says: and mouse will join typewriter in the museum of office equipment.
I haven’t used a mouse in years. I am instead addicted to using my Apple Magic Trackpad with ALL the gestures turned on.
It’s a neat idea, but it isn’t practical.
There are a lot of very specific use cases where this would work, such as chalkboards/whiteboards, tactical displays, etc. but for the common desktop, it does not work from an ergonomics standpoint. Flattening the screen to make using your hand as comfortable as a pointing device only further complicates problems associated with bad posture or “leaning while you work”.
I’m not saying we’ve already reached the pinnacle if the desktop environment, but simply adding touch and dropping the screen is not its replacement.
For centuries, humans have worked in the same way, directly interacting with their work (writing, drawing, cutting, splicing, folding, swiping, hammering, squeezing, pinching, stretching, etc). This was always done on a flat surface in front of us. The ergonomics of this scenario have been perfected in the course of those centuries. Over the last thirty years (a blink of an eye in the history of humans), for a certain few tasks, we have abandoned this ergonomic concept for another, completely unintuitive one, where you press or move one device on the surface (keyboard / mouse), and seemingly unrelated events takes place on a different device placed about 60cm away (display). Being human and adaptive, we have learned to accept this unintuitive concept, at first taking pains to learn it, and later, as we introduced it to other humans at an earlier age, becoming more and more comfortable with it. The fact remains that the concept is still rather unintuitive, despite the effort made by people like Jobs and Apple to make it more intuitive.
The introduction of the iPhone, and especially the iPad, finally brought back the concept of interacting DIRECTLY with our work by touching, swiping, pinching and stretching. Two years of iPad and five years of iPhone clearly demonstrates the massive superiority of this UI paradigm over the traditional concept with keyboard – pointing device – display. Some of the humans are quite proficient today with the old concept, but most of the planet’s population must take time to learn how to use it and practice before becoming comfortable and proficient.
I have no doubt that ten years from now, young generations will likely wonder at ours, asking us how could we possibly get anything done efficiently with the old K-M-D (keyboard-mouse-display) interface. I’m sure there will be certain jobs for which some additional accessories may improve workflow (the way pressure-sensitive tablets with styli are used by graphic designers and ergonomic split keyboards by professional typists), but vast majority of general public will likely be more than happy to do their computing on a touch screen laid flat (or slightly tilted) on their desk, the way humans have been interacting with their work for centuries.
I certainly miss being able to use Graffiti handwriting recognition without having to crank up my old Palm IIIc. Hopefully Apple have designed something even more reliable. Apple’s Inkwell handwriting recognition is not good enough for my purposes. Apparently Apple agrees as Inkwell never progressed onto iOS devices.