[Lumiera] Scheduler and friends

[Christian Thaeter]

Ichthyostega wrote:
>>> Christian Thaeter schrieb:
>>>> * 'builder' runs, are dependencies for the frames, Ichthyo can tell you 
>>>> more...
> 
>> Ichthyostega wrote:
>>> Yes, I pretty much expect them to be triggered only by GUI actions 
>>> resulting in mutations within the high-level (object) model. The output of 
>>> a builder run is a new node graph for one or multiple segments of the 
>>> (overall) timeline.
> Christian Thaeter schrieb:
>> Do you want to trigger this build events still by the scheduler (as high pri 
>> jobs) or call it directly. I'd suggest to put it on the scheduler.
> 
> Indeed I always assumed that they will somehow be managed by the
> scheduler or at least that the backend is involved while triggering.
> Now your answer made me aware that there would be also the possibility
> to handle them directly from the "proc dispatcher" (which is just the
> entity controlling the incoming editing operations). At least
> theoretically this would be possible, and maybe even simpler
> at first sight.
> 
> But I still doubt this to be the better solution on the long run. I
> still think that we might be better off to concentrate all such scheduling
> of work jobs at one place, i.e. in the backend scheduler. Otherwise we'll
> rather end up with duplicated functionality: because triggering the
> builder runs will include a bit of monitoring of the timings; it might
> even be a good idea to include a tiny delay, in order to catch multiple
> incoming operations from the GUI.
> 
>> The advantage is that the backend can manage locks and resources...
> yes indeed. Especially regarding "locking", in this case, according to my
> general design we just need to block any incoming operations via the
> proc dispatcher. (Here I'm following an approach common in many GUI toolkits)

ack

> 
>> Further you may decide if you do a synchronous wait, thats prolly the easiest,
>> or do this asynchronously then you leverage multitasking and can build 
>> different things in parallel.
> Yeah, I thought to start out with an synchronous API for the first version,
> and then investigate what parts could work in parallel. Obviously, the first
> 2 stages of the build process can't, but the rest (which probably will to the
> bulk of work) look as if they could. Let's just wait until the first complete
> implementation "is available" ;-)

ack

> 
>> My plans are to schedule only single frame renders, and then frame per node, 
>> rendering sequences need frame jobs for each frame, but there will be some 
>> hinting not to waste resources on this.
> the pull() call is per-frame, so basically the backend is completely
> free here to do as it sees fit.
> 
> 
>> If some job it already running we may decide to just let it complete and 
>> throw the results away, in most cases this will happen to rarely to be 
>> important.
> 
> agreed, my understanding too. Anything else would need plugin support....

ack
> 
> 
>>> This might rise a concern regarding memory management; but actually I don't
>>> see any problems here: as long as any halting or aborting of jobs is done
>>> in a sane manner (i.e. cooperative, so the ongoing task gets a chance to 
>>> release its context and stack), the ref-counting will do the hard work for 
>>> us :-)  Even if an object got removed from the high-level model, it will 
>>> stay alive until any task relying on it has finished.
> 
>> I am not completely convinced about refcounting. There are numerous problems 
>> with it. Each acquisition is a write which is bad for performance, its hard 
>> to debug and when you have cycles (there may be feedback loops and graphs 
>> spaning the timeline) it just gets hairy. Sooner or later we can just add a 
>> small garbagge collector for nodes and their resources, lets see.
> 
> :-) we shouldn't forget that a garbage collector is plagued by the same
> problems, and even more of them.
> Well, I did a sloppy wording and should state it more precise. I was mainly
> referring to assets and the objects within the high-level model, where up
> to now I rely on refcounting, and didn't run in any problems yet.
> 
> The nodes within the low-level model are a different story: here memory
> management relies on the memory pools provided by the backend. All the nodes,
> wiring descriptors, buffer pointer stacks (and maybe some other families of
> small objects to come) will be bulk-deallocated for a complete segment.
> There is no need for ongoing allocations once the build process is done.
> By definition there can't be any connection to neighbouring segments. (This
> causes a bit of duplication, but if this shows to be a noticeable problem,
> then I'll switch over to the Flyweight pattern, which is just a localised
> change within the builder)

ack
> 
> 
> Maybe I should add a note on dependencies and prerequisite frames while
> rendering. We use pull rendering, which means that the top level (output)
> frame doesn't "know" all its dependencies, just its immediate predecessor.
exactly
> Of course there could be an API on each node to call down to assemble a
> list or range of all predecessor nodes. And I assume to use such an API
> when the scheduler needs hints in advance. Caching this information is
> also an option.
yes, well I think the schedule can collect this recursively, lets see.
> But actually, when rendering a frame, these prerequisite frames are just
> "discovered" in the course of calling down. On certain "cache points",
> the pull() call asks the backend/cache for a specific prerequisite frame,
> and if this call fails, it will issue the necessary recursive pull()
> call to its predecessor node(s).

Moreover for temporal effects, the current frame depends on a possibly
infinite sequence of previous frames (ugh, ever thought about effects
which depend on future frames .. and then have that both applied to one
track?). For the final render where we start at frame 1 and work onward,
this is not a problem. But when previewing and scrolling though the
project we can't render all previous frames. I plan to add some quality
weight which is passed up to dependencies. Each plugin can implement a
static factor on how much it thinks the some input will contribute to
the output and then some lower limit on how exact we go for preview
rendering.


> These "cache points" are wired up by the builder, and it's just a
> question of parametrisation on what nodes this will happen. We might
> set up things so as to insert a "cache point" on every node, or just
> on some nodes known to be expensive, or even not at all. Moreover,
> when the pull() call queries the backend/cache for prerequisite
> frames, in a distributed renderfarm there could also be a dispatched
> job to be scheduled on a remote node.....

My current plan is to have caches as files on disk (and an index) and
for a simple first implementation I just want some sliding buffer for
example for frames 1-24 look like the following:

-----------cachefile---------
01 02 03 04 05 06 07 08
   09 10 11 12 13 14 15 16
      17 18 19 20 21 22 23 24

That is every 8th frame rewinds to n-7 and starts overwriting there and
we have a gurantee that ever 8th frame is cached (in real we may do that
with every 32th or whatever frame, 8 is just for this example)
Some other overlay schemes are also possible and have to be investigated.

> While this principle may sound simple, the actual situation is a bit
> more complicated, as many nodes deliver multiple channels of output.
> E.g. a codec/demuxer will output video and audio. The implementation
> of the render nodes, as it stands now, explicitly relies on the
> presence of the cache to handle these. The mechanics of the pull()
> call will just request suitable buffers from the backend (and might
> provide the necessary hints at this point). But after finishing
> the process() call at this node, the "superfluous" buffers will
> just signalled to be released, supposing the cache will know
> to keep them around to be re-used when we hit them on calculation
> of the respective other channels.
> other channels.

So it will be your job to give nodes unique (but maybe regular)
identifiers and then open caches for them, this are then this 'cache
points' you speaking about. We need some heuristic to determine which
are the cache points. Some cost model comes in mind, for example costly
nodes or the inputs before something get mixed would make sense to
cache. You then have to provide some hints and parameters to this caches
(unique name, sliding window, other parametes). Then each request needs
some unique identifier which determines the frame at the current state,
I think a 'generation counter' which is incremented for each mutation
will do this fine, this is then stored in the index. It might be a bit
tricky to ensure proper invalidation down the stream, but should be
doable. And dont forget about the 'masks' i once spoke about which
optimize the render process by masking out all parts which don't
contribute to the final output.

The other thing is the memory for nodes which is not at such cache
points. I want to keep that in some shared cache files (each for similar
sized frame), perhaps in some hash like configuration that for each
query are a few alternative locations and not such sliding window.
You'll likely get some similar interface to open it as with the cache files.


	Christian