Apple has offered details of a system for monitoring and improving machine learning (ML) applications. An internal development tool titled “Overton” handles lower-level tasks which allows engineers to focus on higher-level concepts.
For example, Overton could generate a model to supply the answer to a question that may be tricky for digital assistants like Siri to parse, such as “How tall is the President of the United States?” This sort of query requires multiple data pipelines to be sourced, with many parts to ascertain before creating the intended answer.
Normally engineers would spend most of their time working on fine-grained quality monitoring of unusual data subsets, as well as supporting said multi-component pipelines. With Overton, Apple intends to limit the amount of work an engineer needs to do, automating many of the chores and to keep monitoring elements on behalf of the engineers.
“The vision is to shift developers to higher-level tasks instead of lower-level machine learning tasks,” the paper states. “Overton can automate many of the traditional modeling choices, including deep learning architecture, and it allows engineers to build, maintain, and monitor their application by manipulating data files.
Apple researchers say that Overton has been used in production to support “multiple applications” in both near-real-time and back-of-house processing, and in that time, Overton-based apps have answered “billions” of queries in multiple languages and processed “trillions” of records…
Apple researchers say that in qualitative testing, Overton reduced errors 1.7 to 2.9 times versus production systems.
“In summary, Overton represents a first-of-its kind machine-learning lifecycle management system that has a focus on monitoring and improving application quality,” wrote the paper’s coathors. “A key idea is to separate the model and data, which is enabled by a code-free approach to deep learning. Overton repurposes ideas from the database community and the machine learning community to help engineers in supporting the lifecycle of machine learning toolkits.”
MacDailyNews Note: The name Overton is a nod to the Overton window, a concept from political theory that describes the set of acceptable ideas in public discourse, Apple’s paper explains. A corollary of this belief is that if one wishes to move the “center” of current discourse, one must advocate for a radical approaches outside the current window. Here, Apple’s radical approach is to focus only on programming by supervision and to prevent ML engineers from using ML toolkits like TensorFlow or manually selecting deep learning architectures.
Read more in Apple’s “Overton: A Data System for Monitoring and Improving Machine-Learned Products.”