One of the things that most (non-professional) media applications don't seem to take into account is that playback/editing is really a realtime problem, and needs to use soft realtime approaches if it's not going to suck. While I'm not going to worry about it in the initial implementation, soft realtime is definitely a direction I want to go with Moing for the sake of getting reasonably smooth playback.
Since lazy evaluation, garbage collection, and even unconstrained memory allocation are all inimical to realtime programming, that means we can't get there in Haskell. But I don't want to try and write the entire application with realtime constraints in mind. Rather, I'd want to section off a small realtime portion of the code in a language other than Haskell (probably a carefully-chosen subset of C++) and run it in separate thread(s). The two halves would be connected by (mostly) nonblocking communication channels; the realtime half would simply blindly work its way through a local "script" for the scene with all the timing and positioning information; the non-realtime half would control it by sending incremental updates to that script across a channel. This is roughly the same approach which Rosegarden uses with its separate realtime sequencer process, and I think it's a much better idea than trying to write the entire application under realtime constraints (which gets harder and harder the more code is involved).
That reminds me of a particular disagreement I had with a certain realtime programmer (I hope he is not representative of his field) about media programming. Aside from my seeing no need to write the entire application under realtime constraints, there was also the issue of what should happen if deadlines are missed: yes, if you're missing deadlines it's a bug in principle, but this isn't a critical situation where you have to self-destruct the rocket if the control system goes non-linear on you. In fact, in practice, when we're talking about media apps, the amount of work the user may give the realtime portion to do is unbounded, so if the user makes a complex enough document you're going to miss some deadlines no matter what you do. So, for media at least, I think you want to be able to handle missed deadlines gracefully, resynching as time allows, rather than crashing, entering an unusable state with buffers full of garbage, or letting work pile up indefinitely. Those latter three are not acceptable options.
Subscribe to:
Post Comments (Atom)
5 comments:
Sounds good. I'm worried about lag in the pipe, though. I can easily imagine such a pipe 'clogging' when the interactions are more complicated. Oh, I suppose thats what you mean about deadlines. Well, as long as there is a good way to do these communications, I'm game.
Yes, that'd need to be one of the things we'd have to think about from a standpoint of robustness: eventually you pass a point where resending the entire scene is faster than processing all the accumulated deltas. The easiest way to address that is probably coalescing un-received events on the sending side.
(The specific example is a good catch on your part, by the way -- the bandwidth of the pipe isn't an issue I'd originally thought of.)
You might be interested in the paper From Functional Animation
to Sprite-Based Display. It describes an animation architecture along the lines of what you mentioned -- with a general, high-level Haskell-based thread talking to a tuned, lower-level C++-based compositing & interpolation engine that runs at video refresh rate. Moreover, the implementation is correct by construction.
Thanks, that looks interesting!
Post a Comment