I, too, am driven crazy by two audio sources slightly out of sync. The example that seems most crazy is when listening to HD radio in my car. In the case of poor HD signal (really “not enough packets in the buffer”) the radio mixes the HD and the analog audio together. Unfortunately many of the HD stations don’t have the HD and analog audio perfectly sync’ed, so you get this cacophony of the HD signal followed milliseconds later by the analog signal. It drives me nuts!

Of course, a more fun use is for something like Speech Jammer, which makes use of audio processing delays to really mess with your head.

I did a lot of video work early on the Amiga as well and then more recently at Swarmcast where we were enabling high def streaming over best effort bandwidth by slicing and dicing the streams. One thing when talking about audio and video is understanding latency versus jitter versus synchronization. In latency there is processing versus propagation latency. Propagation is the minimum time that the signal can transit the media given the distance, processing delay, is well, processing delay. As noted in most non-live event applications latency is almost irrelevant – the latency of a satellite fed video stream can approach several whole seconds for instance (and less than a 1/2 second of that is propagation delay as the round trip is only ~50000 miles). Jitter on the other hand is the variation in delay. Our brain will correct for latency up to some threshold (as 20-50ms are typical for video, 10-20ms for audio) but will notice jitter at a much lower threshold. We also have a brain that does lots of visual processing – so people’s brains will gloss over things like a video gap of up to 30-50ms (this is why movie projection worked). Our hearing has much less processing, so we will notice pops and clicks as short as 3-5ms. Last is the issue of synchronization. In virtually all systems (especially now with digital processing) the audio and video takes a completely different path through the system. Even basic tasks like playback of a movie on a PC represent real challenges to keep the audio and video synchronized.

In a modern system a video and audio signal typically gets “packetized” several times (2-3 at least…maybe IP on a network, and newer digital video signaling like HDMI and DisplayPort are packetized). Every round of packetization will introduce some additional latency – quite a bit if the device isn’t using cut-through transmit and receive (meaning able to start to process or transmit the data without having to buffer a full packet).

So not really driving to any point other than it is indeed very very hard to provide a really good mixed media transport of a media stream. An interesting question would be for live events would there be enough of a market for more specialized gear with low latency and tight synchronization. Can anybody thing of other applications with similar needs? Maybe audio and/or video in a multi-display situation like the office, retail or residential? I can notice differences in latency even with things like a football game on two TVs on separate set top boxes. Example – I have noticed the DirectsTV non-DVR boxes have about 1/8-1/4 second less latency than the DVR box. Having the same thing on two TVs 1/4 offset can be pretty jarring on the audio in the house. Interesting thinking.