A century ago, film comedian Charlie Chaplin was using movie trickery to convince viewers he had almost uncanny timing. In his short comedies, he’d catch a brick without seeing it or miss getting hit in the head by a hammer even though he had no idea it was hidden on the other side of a door.
Decades later, historians discovered his trick: he would film some scenes in reverse: throwing the brick up and walking backwards through a doorway and then playing the scenes in the opposite direction — magic. The fact is, though we think we can always tell when a video is being played backwards, we can’t, but computers may be another story. Scientists at Massachusetts Institute of Technology (MIT) are now teaching computers to use life’s little physical dynamic queues to tell the difference between video running forward and reverse.
Not as easy as it sounds
They’ve developed a new algorithm that can, according to a release, detect with, on average, up to 80% accuracy whether a piece of video is running forward or backward. Researchers plan to reveal their work at the IEEE Conference on Computer Vision and Pattern Recognition later this month.
The algorithms attempt to detect subtle changes in video that could be considered earmarks of movement and direction. The goal is to see and understand what’s expected to happen in the physical world versus what’s not.
William Freeman, a professor of computer science and engineering at MIT described the work as learning the structure of the visual world. “To study shape perception, you might invert a photograph to make everything that’s black white, and white black, and then check what you can still see and what you can’t. Here we're doing a similar thing, by reversing time, then seeing what it takes to detect that change. We're trying to understand the nature of the temporal signal.”
Of the three algorithms MIT scientist developed, the one that really did the trick divides each frame of video into hundreds of thousands of squares and then slices each square into a four-by-four grid. Overly simplified, the algorithm then watches, frame-by-frame (there are typically 24 frames per second of video) for pixel movement from square-to-square.
That movement is compared to a sort of movement dictionary, which is based on information the algorithm has learned from a sampling of 4,000 girds in the video. It’s the combination of these actions or “dictionary words” that ultimately provides clues of forward or backward movement.
The researches first trained the algorithm on one set of videos and then let the algorithm analyze a third set. It was able to accurately detect if the video was running in forward or reverse up to 90% of the time. Its lowest rating was 74%.
Fakery made more real
What does this mean? Detecting direction could help computers understand how the real world should work. They could identify fake videos and maybe help build programs that could create even more realistic motion video games, computer animation and special effects.
"If we could learn statistical models describing the dynamics of the visual world, that might help us as we try to synthesize realistic visual dynamics. This work is just a step in that direction, learning how to detect the unrealistic visual dynamics of things moving backwards in time," Professor Freeman told Mashable
It also means that future Charlie Chaplins may have even better ways of fooling their audiences.
Watch this stock video of things going forward and in reverse as you ponder that.
Have something to add to this story? Share it in the comments.