Samsung laid out its AI for All vision at the Samsung Developers Conference 2024 in San Jose this week. The company has integrated advanced AI features across its major product lines, with Galaxy AI on mobile, Bespoke AI on home appliances, and Screen AI on its TV and audio products. While there's been a greater marketing push from the company around the AI features on these devices, Samsung TVs have implemented these technologies for years, particularly with respect to upscaling content as high as 8K and delivering a vastly superior video and sound experience.
The company is in a unique position to show the industry the way forward on AI for audio and video devices, considering its domination of the TV market. There was considerable emphasis on highlighting these advancements at SDC 2024, and while to some it may seem that perhaps things could hit a saturation point here, in reality, the potential for AI on the screen has never been so significant.
That, and much more, were the topics of our conversation with Samsung's Moonsoo Kim, VP, Head of Application S/W R&D Group at the Samsung Developers Conference 2024.
Answers have been paraphrased for brevity and clarity.
Q1: How has AI transformed the user experience on Samsung TVs and soundbars?
A: Samsung has been applying AI-related features to picture and sound quality for a long time and the reason we’ve been doing this is to provide customers with an immersive experience, which is the inherent value of the big screen, something a smaller device can’t provide.
We focus on improving picture and sound quality a lot and remain driven to continue doing that.
Optimization is also an ongoing process, because we have learned that decent AI requires a lot more computing error, so in parallel there’s another process of optimizing software to be adapted to limited hardware.
Q2: What limitations, if any, are preventing AI on such devices to achieve its true potential?
A: I think it’s a bit early to think about limitations because if you’re thinking about the current form factor, definitely there could be some limitations, because TV is typically attached to the wall, what if the screen is moving? What I believe is that AI can be adapted to many other areas, including improving experiences on the big screen.
I believe sometimes limitation triggers innovation. Hardware and software requires deep integration to realize very challenging AI features and definitely when there is software innovation, the hardware follows. In this case, in generative AI, I think the software came first and we are in the process of improving the hardware continuously.
Q3: Has incorporating AI into soundbars and TVs helped increase their purchase appeal among customers, and if so, how does Samsung quantify that?
A: It depends on the needs of the customer who purchases these TVs and appliances. AI features can help our devices become more efficient, for example, even TVs can retain the same picture quality while consuming 20% less energy, so it’s beneficial to the customer and it’s also good for the environment. Nevertheless, quantifying people’s purchase decisions is a very difficult task, because customers see everything and compare.
They don’t buy something just because it advertises specific features. Our lead in the TV segment goes beyond AI. Samsung has had the number one position for 18 consecutive years because customers love the picture quality and immersive experience. This brand power has actually made it possible for us to continuously invest in AI to improve the experience on TV.
Q4: Features like Intelligent Mode are part of the AI offerings on Samsung TVs. Could you elaborate on the specific machine learning models or techniques used to adapt to changing factors and preferences to deliver a personalized experience?
A: Without speaking to specific models, we apply different technologies to improve the image quality. We process the picture with neural networks and classify each and every frame to improve it using that network that’s hard-wired on the chip.
Intelligent Mode in particular analyzes the viewing environment such as whether the light is turned on or how bright it is, etc. This kind of information can help us optimize the viewing experience. For example, if you’re watching a movie in a very dark environment, you have to adjust the brightness of the TV and if lowered significantly, you may hardly recognize the screen.
We use several techniques to ensure that viewers can enjoy the content without any loss through preserving dynamic range and optimizing brightness. Samsung has applied AI technology in combination with information from sensors, lighting conditions, and the viewing environment which is then consolidated for the AI to optimize the viewing experience without losing the detail.
Q5: Do you find that current hardware architectures in smart TVs are the limiting factor for pushing AI capabilities further, or are AI algorithms and software innovations needing to mature to better utilize the existing hardware?
A: The software side can definitely improve to bring the benefit of AI to more people. We have to lower the size of the models as currently a decent AI model requires too much space, memory and CPU. So optimization of that size will be very much needed to provide the same experience to more people.
It’s ongoing, as you can see with SLMs and the trials to minimize the size of the models. Many companies are doing research to make models smaller. At the same time, I believe that a bigger model should also integrate its experience with the screen, since that’s not the case right now.
For example, what if TV supports generic and natural conversation with people? There can be such advancements to provide the kind of experience that people have only seen in sci-fi movies until now, where people are usually interacting and talking to screens.
Q6: How does Samsung’s AI handle the balance between on-device and cloud-based processing for real-time upscaling and sound optimization? Are there specific optimizations that allow for high-quality AI features while maintaining low latency on TVs and soundbars?
A: Optimization is all about deploying the AI model to on-device, and that has certain limitations in terms of latency as we have to run and calculate everything on the device so what matters here is how we can do it better.
Anyone can run an on-device model, but the output could be different. Here, the key for Samsung is to leverage its experience in optimizing picture and sound quality so even though these devices have limited hardware resources, we can successfully make the video and audio experience a lot better.
Q7: Samsung’s AI-based color mapping technology enhances color accuracy, particularly in complex areas like skin tones. How does the AI adjust to these subtle variations in color in real-time, and what data is it processing to make these decisions?
A: The color gamut of the input signal typically isn’t as broad as that of the TV. We make optimizations considering the TV’s broader range of color gamut using AI technology. For example, when upscaling a picture it checks whether the skin tone, which is very sensitive to people, is presented in a natural way even though it is upscaled, with the facial color preserved.
Algorithms are applied to find out where exactly the face is located, extracting the facial level from the screen in real time, and different logic is applied to preserve the skin color, so that’s how we provide better balance in terms of color representation to make it as natural as possible.
Q8: Can Generative AI play a role in customizing sound profiles for users, automatically adapting the sound settings on Samsung soundbars based on the type of content and each user’s unique preferences?
A: When we render sound we optimize the sound level and it doesn’t require Generative AI in many cases. We can just apply a very simple neural network to improve the sound quality and meet users preferences. For example, bass amplification when the user is specifically watching music videos can be achieved through customization amplification of a certain range of sound based upon a simple AI algorithm.
We actually apply similar technology to amplify the voices in content where it’s not clear enough. The AI model detects the voice level, checks the noise level, and if it’s too high to easily hear the voice, it then amplifies the voice so that customers can easily hear what’s being said in the content.
Q9: What is the long-term vision of Samsung’s AI features for soundbars and TVs? How do you see it evolving in the next 3-5 years, especially in the context of strengthening the Galaxy ecosystem?
A: It’s quite difficult as the world is changing very fast and it’s hard to expect what comes next. As we have declared the new era of screen AI, what we are seeing is that adoption of AI to screen will increase and definitely could change the way we interact with TVs.
For example, AI might no longer require us to use a remote control in the future, the screen should be able to interact with you, recognize you, and it may provide you a better way of interaction, that’s how i envision the future of the screen, and perhaps soundbars can provide similar features as well. Ballie is again a good example of these concepts, it has a screen and wheels and it can come to you if you call it.
