The Real Reason Why Flash On Tablets Hasn’t Happened Yet

I was recently re-reading an old story from a 1998 edition of The Onion mocking Compaq (remember them?) for introducing a $4995 computer that can – gasp! – play TV shows downloaded from the Internet. I remember reading it in 1998 as the Great Internet Bubble was in full swing. It was funny then – who would spend $4995 on a computer and install an unthinkably expensive high-speed 1.54Mbit/s T1 Internet line just to watch TV? – but it’s even funnier now because watching TV on our computers is exactly what we’re all doing. It's now so commonplace that the TV industry is running scared of the ‘cable cutters’ who are giving up TV to get their video from the web.

Probably the most exciting advance in what The Onion once mocked as outrageously expensive is the new category of low-cost computing device: the tablet. Apple has proved with the iPad that we can have a lightweight device with a long battery life that’s great for watching video.

Unfortunately, it’s not quite complete: the majority of web video (and nearly all the TV shows from that Onion story) is provided using Adobe Flash – famously excluded from the iPad by Steve Jobs in a public spat with Adobe that were dubbed “Flash Wars.” Apple has tried to move the web away from Flash and toward open standards like HTML5 and a format that the iPad can play. But YouTube has explained why it is sticking with Flash – video codec choice (there is no universal codec that will play on all PCs, unlike Flash) and Digital Rights Management (HTML5 doesn’t have any).

While it's true that YouTube is often accessed via a dedicated tablet app (although with restricted content), video on the web is fragmenting rather than consolidating: sites like Vimeo and Mpora are gaining viewers, and I think it’s safe to say that for several years to come there are going to be major web video sites that will be using Flash.

So why aren’t we seeing Flash-enabled browsers on tablets? Plenty of manufacturers have announced Flash capability but we’ve seen little evidence of it running well. Unfortunately, it turns out that Flash Wars is more than just a bit of industry politicking: there are some very tricky technical problems to implementing Flash on an embedded processor – and it’s not going to be solved by locking programmers into a room and yelling “fix it!” at them.

Flash was designed in the mid-90s for PCs and the new Internet era to display multimedia content in a web browser. It decodes video in software, combines the video and graphics in software, synchronizes the video and audio in software, handles Internet dropouts in software, and copies the results as fast as possible into the web browser. It relies on the CPU being fast and handling RAM accesses very quickly.

This is almost the complete opposite of how digital TV set-top boxes are built for video: they have slow CPUs hooked up to dedicated video decoders, which are directly linked to the display. The CPU basically is a manager for all the intensive video processing, which goes straight to the TV. Tablets are built from embedded processors that are designed for low power consumption and low cost: the Apple iPad (running its own ARM-based A4 processor) consumes about 1.7W including the screen. Compare this to the 8W for just Intel’s Atom 330 processor. The flip-side is that the Atom runs general software applications a lot faster.

Getting Flash to run on a tablet efficiently means that the video must be handled by a hardware video decoder, like the digital TV set-top box does. And indeed, Apple’s iPad can do 720p HD video decode without breaking a sweat (as anyone who has used an iPad to watch movies knows – it hardly dents the battery at all).

But here we hit the first problem with implementing Flash: video codecs. The H.264 standard is very common. It’s used in all kinds of places, from digital HDTV to Blu-ray to iTunes. It’s widely supported in embedded processors and it’s one of the three video codecs used by Flash (the other two being VP6 from On2 – the company recently acquired by Google – and Sorenson Spark). The most common format for Flash video is H.264. So what’s the problem? Why not get the Flash software to call up drivers for video decode hardware and get that to do the work? The problem is that there’s no such thing as just ‘H.264’ – the standard was drawn up to satisfy contradictory requirements. Some wanted it to be easy to decode (for cheap and low power video hardware) but others wanted sophisticated compression (for efficient use of bandwidth).

The H.264 committee decided to please everyone by producing ‘profiles’: Baseline Profile (BP), Main Profile (MP) and High Profile (HiP). A chip salesman may well say “my chip does 1080p HD H.264” but what he mumbles behind his hand is crucial: “for Baseline Profile video”. Adobe’s Flash software for the PC supports H.264 HiP, and lots of web sites encode their video for this profile (after all, if Flash decodes it and the bandwidth is lower, why not?). Unfortunately, it means that lots of web video won’t decode on H.264 BP.

So these embedded processors have to use software to decode the video – soaking up CPU time and battery power, the very thing they shouldn’t be doing. In many cases, the processor won’t be able to decode the video fast enough and the video won’t play at all. So why don’t they just support H.264 HiP in hardware? The answer is that most of the embedded processors used in tablets today were designed a couple of years ago for the then smartphone market, and this market wasn’t asking for H.264 HiP video: everyone was expecting to sell music videos for $5 a pop from me-too online music stores that had H.264 BP video encoded for small screens.

One solution to the video codec problem is of course to get everyone in the world with a web video site to re-encode the video using H.264 BP. This I call the ‘Field Of Dreams’ approach (“if we build it, the content will come”) and as a commercial model it’s been a miserable failure countless times before (remember WAP?). I’m not holding my breath. But even if we assume the video decode problem magically goes away (maybe the ghosts of famous baseball players will help out with the re-encoding) there’s still another problem: memory bandwidth.

Graphics and video have to be merged together for display (usually so that text and icons appear over the top of video) and then synchronized with the sound (a process called A/V sync). In digital TV this is pretty straightforward and is done in hardware: the output from the video decode is mixed with the output from the graphics chip to form the picture (a particular color is chosen as transparent, and video shows through: this is the ‘green screen’ technique for superimposing one video source on top of another). It’s very efficient and the hardware to do this is very cheap.

In Flash, designed to run on PCs, this is of course done in software: each decoded video frame is copied to a piece of RAM, the graphics and text are overlaid and the result copied again into the web browser and finally the visible part of the web page is copied into the display hardware for output on the screen. This whole process has to be done 30 times a second to give good quality video. On a PC this isn’t too bad: they are designed for doing this kind of thing. For embedded processors, it’s a problem: again, it’s CPU time and power consumption that makes things run sluggishly and runs the battery down. And the bigger the screen, the more data to be handled. The solution is again to get Flash to use specially-designed hardware for doing all this work: to have dedicated hardware pathways for converting video, for layering graphics and for scaling the outputs so that the huge amount of data being handled doesn't hit the CPU and soak up memory bandwidth. Unfortunately, we are only just now seeing embedded processors with this kind of video handling hardware coming to market and Flash needs to be adapted to use this hardware.

I can’t see how we are going to see tablets that can do a good job of showing HD video from Flash web sites until we get tablets with embedded processors that are up to the job and the system software developed to use that hardware properly. If anyone tells you they have a tablet that does Flash video really well, arch your eyebrow in a Spock-like skeptical manner and ask to see it.

Join the conversation about this story »