The best smartphones of the moment are true beasts of photography and video, thanks to the fact that they have three or more rear cameras with different focal lengths, great power to process images and videos, and camera apps with a lot of modes.
What many of us wonder is what will be the next step for the industry?
Judd Heape, vice president of camera products at Qualcomm, is clear about the future of photography for smartphones and has shared it with Android Authority.
Artificial Intelligence (AI) is going to continue to be one of the main areas of interest for smartphone and chip manufacturers in the coming years, and Heape believes that AI will be the biggest area of interest for the industry. It also points to some ways in which AI will take a step forward in the coming years.
In the future, we’ll see a lot more AI ability to understand the scene, understand the difference between skin and hair, clothing and background, and that sort of thing.
And all those pixels are handled differently in real time, not just in post-processing a couple of seconds after taking the snapshot, but in real time during video recording with a camcorder.
The use of AI has come a long way since the modes first introduced in 2018, but Heape explains that AI in photography can be divided into four stages.
The first stage it is quite basic; AI is used to understand a specific thing in an image or scene. In the second stage, the AI controls the so-called 3A features, i.e. autofocus, auto white balance, and auto exposure settings.
Qualcomm believes that the industry is currently in the third stage of AI photography, in which AI is used to understand the different segments or elements of the scene.
Current technologies such as semantic segmentation and face/eye recognition leverage AI to recognize specific subjects/objects in a scene and adjust them accordingly. For example, today’s phones can recognize a face and make sure it’s well exposed, or recognize that the horizon is crooked and suggest that you hold the phone correctly.
And the fourth stage? Heape says that we are about three to five years away from reaching the holy grail, in which the AI will process the whole picture:
Imagine a world of the future where you’d say “I want the photo to look like this scene from National Geographic” and the AI engine would say “okay I’m going to adjust the colors and texture and white balance and everything to make it look better.” look and feel like this picture you just showed me.”
Sony made some interesting claims earlier this year when an executive predicted that Smartphone photos would outperform images from DSLR cameras in the next few years.
It’s easy to imagine, given the progress we’ve seen from smartphones, but phones still have smaller components due to the slimmer form factor compared to DSLRs. Qualcomm acknowledges these limitations, but he still believes that smartphones will outperform dedicated cameras.
In terms of getting close to DSLR image quality, yes. I think the image sensor is there, I think the amount of innovation that’s happening in mobile image sensors is probably faster and more advanced than what’s happening in the rest of the industry.
Furthermore, he explained that the processing power available in smartphones is a significant advantage over DSLR cameras:
The processing in Snapdragon is 10 times better than what can be found in the biggest and most powerful Nikon and Canon cameras.
That is why we are able to overcome the barrier of image quality. Because even though we have a small lens and a small image sensor, we are doing much, much more processing than even a DSLR is capable of.
The pace of chip development means we are seeing huge performance improvements in the camera. For example, the series Snapdragon 865 offered unlimited slo-mo at 960fps, the Snapdragon 8 Gen 1 offers 8K HDR video, and the series Snapdragon 888 introduced simultaneous 4K HDR recording across three cameras. Add to this ever-improving multi-frame processing and smartphones are pushing the limits of cameras in impressive ways.
Sony’s prediction about DSLR cameras came when the company introduced 1″ sensors for mobile phones. We saw the first sensor land in the 20MP Sharp Aquos R6 and 12MP Xperia Pro-I last year, while Xiaomi and Sony also worked together to bring a brand new 1-inch 50MP IMX989 sensor to the Xiaomi 12S Ultra.
The megapixel war has quietly reignited in recent years. In 2018 and 2019, 48MP cameras were a rare thing, but since then we’ve seen 108MP cameras become commonplace.
Recently, the first phone with a 200MP camera has been introduced. There are also rumors that Samsung could offer a 200MP camera inside next year’s Galaxy S23 Ultra.
Mediatek already supports 320MP cameras on the Dimensity 900 series of chips0, and Qualcomm believes this resolution could be the next leap point for the industry.
The 320 MP is probably the next point where we are going to stop
It is clear that there is room for sensors with many megapixels and sensors with larger pixels, since companies like Samsung and Xiaomi are going this way. However, Heape has his preference:
My philosophy is maybe different from some people in the industry, as I don’t necessarily think we need hundreds and hundreds of megapixels. I think we need bigger pixels, right? We have to get closer to DSLRs and maybe take the sweet spot of about 40 or 50 MP instead of going to 200 MP and 300 MP. Although the industry is pushing to go both ways.
Heape suggests that innovative technologies such as anamorphic lenses could help increase size, but it will be difficult in the short term.
In the short term, no, I don’t see us going over the inch. But in the future, yes, we can probably get to it.
Heape explains that video is currently the area of greatest interest for hardware accelerators, pointing to the bokeh engine and computer vision hardware. But it also anticipates another novelty in future products.
Very soon we will have announcements of dedicated hardware to handle, as I said, different parts of the scene. A hardware that will know what to do with the pixels of the skin, the hair, the fabric, the sky, the grass and the background. Those are the areas — and they all apply to video again — where we really see the need to add specific hardware.
More generally, we’ve also seen that video quality has come a long way in recent years, but it seems doubtful that we will see 8K/60fps soon, given that 8K video in general still seems so niche to begin with. Heape agrees, suggesting that 8K/30fps is fine for now.
I think we already have the resolution and frame rate at a point where it won’t change for a while. We are already shooting in 8K/30. We may get to 8K/60, but it will take a few years. It’s a tremendous amount of performance and, frankly, what we do run into is power consumption limitations as we level up.
If anything, we expect more refinements to come for 8K video and below. After all, Qualcomm introduced 8K HDR with the Snapdragon 8 Gen 1, while recent innovations like super steady video, AI-driven low-light video, and nighttime hyperlapses take place in 4K or 1080p.
Heape believes there is room for improvement such as a better “managing movement and understanding movement in a scene”. It points out objectives such as motion management in videos without “ghosting” effect and a treatment of motion and noise at the same time.
All of these basic principles of image quality that we address through multi-frame techniques in snapshots, we have to handle in real time for 8K/30 video.
We have already begun to see phones with selfie cameras under the screen in the last two years, and it is clear that even the latest solutions cannot compete with conventional cameras. Heape agrees that they are not an option for quality seekers due to issues with diffusion, color, and “strange” artifacts at night.
The cameras under the screen may eventually seem reasonable, but that’s about it.
I understand that people just want a very sleek and gorgeous looking edge-to-edge display with nothing to get in the way. But for people who really want to do good photography, I don’t think that’s the direction we should be going.
But the things that we’re achieving in photography now — really good zoom, scene understanding, segmentation, depth, good 3A processing — all of this only gets worse if you try to put the camera under a bunch of pixels. .
will we get that [las cámaras bajo la pantalla] good enough that a typical selfie looks reasonable? Yes, but anything beyond that is going to be very, very difficult.
The telephoto cameras they’ve been a fixture in smartphones dating back to 2016, with modern optical zooms limited to a long but fixed 10x range. Sony has also innovated in this area with the Xperia 1 IV, introducing a continuous zoom telephoto camera, capable of shooting at a variety of native zoom factors. This means that separate telephoto/periscope cameras are not needed.
Heape believes that future phones could ditch the dedicated telephoto camera in favor of a main camera that doubles as a telephoto lens.
I think best of all, it can reduce the complexity of the camera system. Whereas three cameras are now needed, such as a wide angle, an ultra wide angle and a telephoto lens.
If you can have true optical zoom, maybe you can combine wide angle and telephoto in one camera, and have only two cameras and simplify the camera system, reduce power consumption, etc.
The next few years look bright for innovation in this space, between increasing sensor sizes and megapixel counts, potentially shrinking cameras thanks to variable telephoto technology, using AI to take care of more camera tasks and more advanced hardware.