[Lumiera] Question about Scheduler capabilities

[Ichthyostega]

Hi Christian,

thanks for the detailed explanation (much appreciated);
for now I'll just concentrate on one detail...

Christian Thaeter schrieb:
...
> The 'hard' queue is ordered by *start* times for jobs, a job there will not 
> be scheduled *earlier* than this time. Each job can also have a 'expire' 
> time. So scheduling may happen at or later than the 'start' time but the job 
> will be handled specially (configureable) when the expire time is missed. I 
> am not finally decided how to implement this 'special' handling, likely 
> something like "abort if no other job depends on it, otherwise keep if the 
> other job still can reach its goals", likely this will be implemented by a 
> functor to make this very flexible and leave some room for experimentation 
> and design. This is what you call 'marked as failure'.

yeah, leaving this configurable for future extensions sounds like a good
idea. At the moment, I'm not aware of any situation where a dependent
job has lower timing requirements than its prerequisite jobs.

That is, I expect the last job in a dependency chain to have the most
stringent constraints; i.e. the last job, who actually delivers data
will likely be scheduled with a very tight timing window, because in
the general case I really want to avoid unnecessary buffering.

To give a catchy example: if we deliver video frames with a double
buffering scheme, likely we want the last job in the chain to run
/after/ the current back-buffer has been "opened", and up to
/just before/ the flip happens and the back-buffer is swapped into
display. (of course we need to take the timing uncertainties
into account, thus our time window will be much smaller than one frame)

Of course this works only if this "last job" doesn't do much and thus
can be expected to run quickly and predictably (e.g. just doing some
copy into the external output buffer or such or similar).

The timing deadlines for prerequisite jobs are way more "soft"; we can
use these timings to adjust the throughput of the engine etc. Just,
by going backwards from the final deadline, we can derive deadlines
for the prerequisites. And if a prerequisite isn't there at that point,
there is no way the final job can catch up, given the tight requirements
there.

Ichthyo wrote...
>> failure propagation when a job fails, either due to an job internal error, 
>> or by timing glitch, any dependent jobs need to receive that failure state.
>> Usually this means that these jobs are marked as failure too, and the 
>> propagation continues....

> yes, question: i think its easily possible to go the reverse here, that is: 
> when a job receives a failure from a dependent job and this failure is 
> because (for example) a missed deadline, but this job is still fine in time, 
> it might reschedule the dependent job "Do it again Harry!" ... to be 
> investigated

> I am a bit worried that we must watch carefully that there are no 
> failure-avalances,

Why, what would be the problem?
My expectation was rather that we don't propagate in forward direction
(i.e. we don't notify successor jobs if a predecessor fails). Rather,
for a given job now able to be scheduled, we check if all prerequisites are
there and only at that point even notice that a prerequisite was marked as
failure. This way, there can't be an avalanche of recursive calls. Rather,
we just mark this one as failure too and go ahead with the next job able
to be scheduled?

Well, that's just guessing of implementation details on my side ... ;-)


>> guaranteed execution ... some special jobs are marked as "run reliably". 
>> These need to run in any case, even when prerequisite jobs fail — moreover 
>> it could be necessary that these special jobs receive a flag indicating 
>> normal state or failure. (I don't think we ever need to propagate more 
>> specific error information)
> 
> I think we should avoid this, a prerequisite is a prerequisite, by definition
> we can go if something is missing because this just will result in an error.
> Do you have an case where you need this?

Yes, several different.
One thing is, who does the page flip, or who actually emits to the external
interface? In case of Jack, this is a separate thread which we don't control,
same for GTK. But for other output interfaces we might want to run the actual
output handling also as job (maybe). Besides that, we have to care for
de-clicking / de-flickering in case of failure.

Bottom line: for some external interfaces there might be a tiny job, which
runs reliably /after/ the delivery job, and which somehow detects if the
delivery was successful or failed

There are several other concerns too (we discussed them on one of the
last meetings): I need a way to know reliably when certain "mark points"
in the frame sequence are passed, especially for all those nasty cases
when the "playback plan" has been changed on-the fly (live changes of
looping, framerate, scrubbing, life changes to the edit). Another thing
is that we need to enqueue the next chunk of jobs reliably. Like it or not,
at some point inevitably we need to know when some things are done.

> If really required i could add some kind of 'weak' resources to jobs, but i 
> urge to avoid this. On the other hand, as i described above, if the 'failure'
>  handler is a functor, then you already can remove this failed resource from
> a target job in its handler, possibly allow the target job to become
> scheduled since no resources are missing anymore.

As said, wouldn't it be much simpler just to have a flag in the job
to determine, if it dies in case prerequisites did fail, or otherwise
if it runs anyway?

See for comparison the Actors in Erlang: some of them die if a linked
Actor dies, others get a special (asynchronous) message in that case.
Well -- we don't have messages in Lumiera, but instead we have a guaranteed
sequencing through prerequisite jobs. I can't see how anyone would be able
to build reliable asynchronous systems without any such feature.

	Hermann