New technology will allow iPhone 11 users to take video and automatically remove the background using AI-based depth analysis.
It was developed by Spektral, a Danish company that has managed to analyze video at 60 frames per second using machine learning and computer vision. According to Spektral, the tech “combines deep neural networks and spectral graph theory with the computing power of modern GPUs [which] can process images and video from the camera in real-time directly on the device.”
Now, a new iOS 13 beta has confirmed that this AI background removal tool will be coming inside the operating system. This will allow apps like Instagram, Snapchat, and Apple’s own camera app to swap backgrounds and play with 3D elements that could go in front or behind a subject in real time… Other camera upgrades include an expected three lenses on the iPhone 11 Pro models, and that ultra-wide camera will let you shoot three pictures at once in order to create the ideal shot.
MacDailyNews Take: Check out the video below from Spektral, which Apple acquired in December 2017 and about which we wrote, “Yes, Apple’s interest in augmented reality remains high. We’d guess that the tech would be/already is useful in the Camera app’s Portrait Mode, iMovie, Clips, etc.”
Over two years of advancements has been made since this video was released on May 31, 2017:
iPhone 11 is going to be all about the camera(s) and that’s going to sell a lot of units to people who thought they might be waiting until next year’s 5G iPhone.
Smart Frame is going to be a tent pole feature. Expect this batch of iPhones, the last one prior to the start of the iPhone 5G super cycle in late 2020, to focus on the triple camera array, Smart Frame, and other camera-related features as the main selling point. — MacDailyNews, August 12, 2019
I think Apple is going to use the three cameras to determine depth and other information from distances from each camera and using the different focal lengths etc.
Background removal, like the iSight cam used to do?
“Background removal” is a simple concept that covers a wide range of abilities. With iSight, you had to take a photo, and then the camera could not move, else you got bleed-through of the real background over the fake one.
To use an analogy, iSight cam background removal is like the cheap facial recognition which were used (and still used!) by some Androids, which are easily fooled with a printed photo.
Spektral’s background removal is analogous to FaceID. It accomplishes the same thing, but 1000x better, 99.9+% of the time.
Now, everyone can fake their instagram vacay posts and no one would notice.
As long as you use different clouds in the background 🙂
Well that’s about the most optimistic and fairly innocuous use of the technology – I can think of many nefarious uses too tho’ I’m no criminal and lack the mindset anyway.
Yes, yes…I know…”the camera cannot lie” was always a myth but this just shreds the idea completely.
From a philosophical point of view, we seem to be heading towards ‘the only true experience is the one you alone witness’ scenario where every other detail can be ignored. Oh wait…
Talking about Nefarious…. have you looked into ‘Deep Fake‘ technology .. its both mind boggling and scary.
Actually this technology exists since 2017, when Zinemath ( a Hungarian company) introduced zKey product to do depth – based so called “greenless” keying for broadcast television quality using a small rig around the cameras’s lenses with a built-in time of flight sensor and a dedicated depth processing workstation. A year later it was integrated with Unity 3D and Unreal and a camera track solution called zTrack to provide a full Virtual Studio and Augmented Reality for broadcasters and streamers in real-time with no solid color background for keying, using depth image and some visual analysis. The depth image and the measured object distance could be embedded in NDI (the IP version of SDI video) to be shared between other NDI compatible production and streaming solutions. Zinemath is the original developer of zLense, the first real-time broadcast quality depth processing platform introduced in 2013.
🙏👍
..a new iOS 13 beta has confirmed that this AI background removal tool will be coming inside the operating system.”
So in other words, this is not actually an iPhone 11 exclusive feature–it will be available to most recent X class iPhones..