A while back a friend showed me an impressive web page from Sonos, which contains an embedded video in which a lady walks around her house turing in a number of coloured lights, as each light turns on the page background gradually changes colour to become an extension of the video. It reminded me of a family of televisions from Phillips which would shine a coloured light onto the wall behind it to help make the TV show more immersive.
To make a video playing in a constrained box more immersive, by creating a coloured “overspill” that shines out from behind the box onto the page background.
Playback is achieved using the wonderful HTML 5 <video> tag, although sadly it’s not possible to interrogate the video’s frames directly in order to determine the overspill colour. However a work around is to write the current video frame to a HTML 5 <canvas> element, from there I can access the bitmap array. The good news is that the canvas doesn’t actually have to part of the page, so we can programmatically create one and use that a vehicle to get the frame bitmap.
What I’m left with then is a map of the current frame represented as an array. The array is arranged into batches of three values between 0 and 255, for the red green and blue elements respectively. So elements 0, 1 and 2 contain the red green and blue values for the first pixel (the top left corner of the image (yeah, it’s a very big array).
Now equipped with the array, I iterate through it, summing up the brightness of each of the red, green and blue elements; dividing each of these sums by the total number of pixels in the video (width x height) giving the average red, green and blue values for the image – the average frame colour.
A CSS3 radial gradient is then used to set the background colour of the page to this average value, radiating out form behind the video so that it appears to be spilling out from it.
As an extra dimension I set the opacity of the coloured element of the page so that the brightness of the video is reflected in the intensity of the overspill colour. This is achieved by summing up RBG values of every pixel and dividing that by three times the total pixel count, multiplied by 255:
(TotalRed + TotalBlue + TotalGreen) / (VideoWidth * VideoHeight * 255)
What this leaves is a number between 0 and 1, which represents how white the frame is; a perfectly while frame would be 1, whereas a perfectly back one would be 0. The final step is to subtract this number from 1 and use that as the opacity for the background colour. To prevent the image form becoming completely transparent, or too dark, it is clipped at 0.3 and 0.75. A 50ms transition delay is used in an attempt to soften sharp transitions in colour, however this can have the effect of creating a lag on such situations.
If you’re wondering about that final subtraction from one, it means that darker scenes will be paired with more intense colours. As a TV shinning colours onto the wall can only make things more bright it makes sense that brighter scenes have brighter background colours, however as this sets the colours of the page I can “shine a dark light” from the overspill, so it makes sense to match a dark scenes with a dark background, not a pale one.
Everything here should work in all modern browsers, with a few vender specific CSS properties filling the gaps in for older browsers. The only slight issue is HTML 5 video codec support; each video needs be encoded a couple of times to ensure compatibility, which is a pain.