Craig Federighi explains development of iPad’s Scribble feature

A computer’s ability to read handwriting, then translate it into letters and numbers it can understand, has been a challenge going back decades, but Apple seems to have found a reliable solution with Scribble in iPadOS 14 for iPad.

iPadOS 14 brings the systemwide Scribble to iPad with Apple Pencil, allowing users to write in any text field — where it will automatically be converted to typed text — making actions like replying to an iMessage or searching in Safari fast and easy. All handwriting and conversion to text happens on device, keeping it private and secure. When taking notes, Smart Selection uses on-device machine learning to distinguish handwriting from drawings, so handwritten text can easily be selected, cut, and pasted into another document as typed text. Shape recognition allows users to draw shapes that are made geometrically perfect and snap right into place when adding useful diagrams and illustrations in Notes.

Data detectors now work with handwritten text to recognize phone numbers, dates, and addresses, and offer users the ability to take actions like tapping a written number to make a call, adding an event directly to Calendar, or showing a location in Maps.

Scribble will initially offer support for English, Traditional and Simplified Chinese, and mixed Chinese and English, so users can write English and Chinese words together without needing to switch languages.

Alexander George for Popular Mechanics:

In the newest update to iPadOS, when you write with the Apple Pencil ($129), the iPad can understand your scrawl and, with Scribble, convert it to typed text. It works like most machine learning—examples inform rules that help predict and interpret a totally new request—but taps into a smarter data set and greater computing power to do what had stumped generations of previous machines.

“When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things,” says Craig Federighi, senior vice president of software engineering at Apple. “We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation.” That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple’s tech, static examples weren’t enough. They needed to see the strokes that formed each letter. “If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written.”

The massive amount of statistical calculations needed to do this are happening on the iPad itself, rather than at a data center. “It’s gotta be happening in real time, right now, on the device that you’re holding,” Federighi says. “Which means that the computational power of the device has to be such that it can do that level of processing locally.”

MacDailyNews Take: With iPad’s Scribble, we’ve come a long, long way from Apple’s Newton, the first PDA to feature handwriting recognition.

[Thanks to MacDailyNews Reader “Fred Mertz” for the heads up.]