Muting TV Commercials With OpenCV
I like to play Pokemon Go. Yes. I’m one of those who are still playing it to this day. I play with multiple accounts, the main one on my mobile phone and the others on my previous older mobile phone’s models. Recently I was going to a walk around the neighbourhood, playing Go as I usually do — when a message popped on my Go screen running on a Samsung S& mobile phone: “There is a new app version please upgrade”. I clicked into the Play Store to see an alarming message in Pokemon Go app page: “your phone model is not supported anymore…”.
Admittedly it’s an old mobile phone but it still runs perfectly ok. I used exclusively to play Go as using it for other day to day apps seems a bit too much for it’s hardware. So what should I do with it now? Maybe I’ll use it as a running picture frame? Naahh, too small. Security camera? Good option but needs to mount and power it somewhere. Pondering this question while watching some TV at that evening was interrupted abruptly by an insane 10 minutes commercial break, with a loud shouting promos that threw my thought back to reality. And then I knew — I will use it to silence those pesky commercials.
System components
- A smart TV. Our living room TV is a 65" Samsung 7 smart TV. Here is a picture of it during one of those commercial breaks:
- My old Samsung S7 mobile phone (Hey Niantic — why can’t you support it anymore for Pokemon GO?).
- My development laptop, a MacBook Pro.
Controlling the TV
The first task to research — is it possible to control the TV remotely in a programmatic way? It urns out the Samsung SmartThings portal will allow us to do that. The TV was already part of my personal SmartThings hub, and it turns out Samsung has an API that will allow you to control hub devices programatically.
We’ll need a personal access token to authorize in the API calls, so head to your Samsung account tokens page and create a new token:
Now that we have an access toke, let’s try it out. Can we get info about the TV?
A request to get my SmartThings devices returned the TV details:
As you can see we got the TV id.
Next we query the TV info:
And we can see it has the audioMute capability.
Testing mute:
It works! Coding this will be easy.
Capturing the TV screen
For this task I will use the S7 mobile as a web cam. While I used it in the past as a security camera using Alfred Home Security app — it required the viewer to run on another mobile phone but I need it to run on my laptop.
I also tried in the past the IP Webcam app. When I was active in mobile app development — one of my application had a feature to view a remote phone’s camera snaps/live stram — and I used this app for ideas and reference. The main problem I had with this app is that it was only usable within the same (WiFi) network the device was connected to. If this is not a limitation for you — you can skip to the next section. In my case I wanted to explore another system I’ve been using recently to access some of the devices in my home: PCs, laptops, Raspberry Pi’s (scattered all over the place), smart appliances (like smart switches), NAS etc.
I’m using ZeroTier. So I joined my S7 device to my ZeroTier home network (it has an Android client app). While the description of ZeroTier is out of the scope of this post — it’s suffice to say that it create a personnal VPN network that enable you to access your devices from anywhere.
My S7 now has a fixed IP which enable me to access the IP camera web interface from anywhere.
Detecting Commercials Break
In my favorite TV channel, commercials have this time remaining counter at the corner of the screen:
We’ll use it to detect when the commercials end so we can unmute the TV.
Capturing the IP web camera feed in OpenCV can be done like that:
Putting It Together
Finally, we can write the code to mute/unmute the commercials automatically:
The script runs in loop and get the live TV stream frames. For each frame it looks at the portion of the images containing the commercials text. It uses tesseract to identify texts in this image and looks for the commercials text.
If the text is found with a high level of confidence (line 32) we start counting the commercials frames. WHen a specific threshold is reached — the TV is muted.
If the text is not found for a specific time — the TV is unmuted.
Here is an example:
It works!
Takeaways:
- We learned we can use ZeroTier to access our mobile phone from anywhere in the internet.
- We learned how to control our Smart TV programmatically.
- We learned how to detect text in video frames.
What’s Next?
When I demonstrated this script to my son he immediately asked: “why go through all this ordeal and not just run it on the mobile phone?”.
Well, because I can :)
But it’s a good point. We’ll revisit that in a future post. Stay tuned…
