[Lumiera] New guy trying to understand the proc model

[Ichthyostega]

Hello Peter,

first of all -- thanks for your interest in this topic. You are welcome!
Feel free to ask questions, because posing and answering questions actually
helps to improve our understanding of the subject matter and is a valuable
form of contribution.

On 09.03.2017 18:40, Peter Lama wrote:
> I can see how Placements would work well for a timeline, but can it also 
> represent node compositing? I'm wondering how it would handle effects that 
> could have multiple inputs and/or outputs.

The interesting question is what it means "to represent"...?

On a technical level, the answer is, of course, yes. The whole point with
placements is to cover and unify several somewhat tricky features you'd like to
use in a more elaborate edit. But if "to represent" means to get it in a shape
which a human understands and likes to work on, the situation is not so clear.
It would be easy to cook up something based on the "mechanics" of placements,
which would behave in a highly puzzling and confusing way. So the point is
rather, can we come up with a representation in the user interface, which
makes such a feature usable and fun to work with?

At this point I should clarify some relations. Lumiera has three Layers, and
this is so for good reason, because the concerns addressed in each of these
Layers are quite different. The question you pose here spans all three layers.

If we ask the question: "Does Lumiera support node editing?", then the answer
would be "No!". With a special twist: The render engine of Lumiera is *based*
on nodes, but we do not expose them to the user, like other applications do.
To use a metaphor, node editing is very similar to assembly programming.
When you first encounter this kind of programming, it looks very cool, it
looks like giving you enormous powers. Yet in practice it is quite cumbersome
even to do the most simple stuff in assembly. It does not scale to large
projects, because, when working on that low level, everything is explicit.

This is the reason why we try to build a "high-level-model". To gain the
ability to handle some things implicitly, akin to what a high level
programming language does. There is a compiler (in Lumiera, we call it
"Builder"), which walks this high-level model and builds the corresponding
node network, which can then be performed by the engine. This is where the
placements come into play. Incidentally, most of this stuff is not yet
implemented ;-) We're kind of paving our way towards that area.

But even the high-level-model is not tangible for the user. It is a data
structure. To allow working on it, we need the third layer, the UI. We need
to translate the structures of the high-level-model into the commonly known
interface metaphors, like boxes you can drag with the mouse, windows with
scrollbars, areas you can expand, nonmodal property panes where you can
tweak settings etc. At that level, we /could/ (personally I think, we should)
create a special UI where you can drag connection lines between boxes to
setup those properties of placement which are related to signal routing.
But creating such an UI is nowhere near trivial, since you'd need maybe
four or five dimensions, but you've got just two dimensions in an UI.


On 09.03.2017 18:40, Peter Lama wrote:
> I don't think I fully understand what all a Placement can do. I would 
> appreciate it if you could explain.

A placement addresses or links together several different topics.

(1) memory management. A placement is a ref-counting smart-ptr.

(2) identity. A placement is like an object instance. It carries
    an unique hash ID (we call that a LUID) and there is an index,
    allowing to retrieve any placement in the session by ID.

(3) scopes. Placements can be attached to another placement. Anything
    attached at a placement is considered to be /within/ the scope
    created by that parent placement. Tracks are within the Sequence.
    Tracks can be within Tracks. Clips are within Tracks. Effects
    are within Clips, or within Tracks, or within a Pipe. Since
    scopes are nested, you can fetch a global definition from an
    outer scope, or you can shadow such a definition with a more
    local definition.

(4) constraints. A placement holds an ordered list of constraint definitions.
    We call them "LocatingPins". What is actually constrained, is essentially
    open. At a minimum, each placement starts out with a time position and
    an output designation. There /can/ be LocatingPins in the list to constrain
    these, e.g. "must be 200 frames after the start of Sequence XYZ", or
    "sound output should go to music subgroup". But LocatingPins /might/
    introduce additional variables, which means additional degrees of freedom.
    Like, a LocatingPin might say "sound position is 20° to the left and 45°
    elevation" or "stereoscopic window position is at 5m distance", or
    "colour grading uses profile moonlight"

(5) query resolution. You can ask a placement to resolve itself for a given
    parameter / dimension: "find out your absolute time position". The placement
    first applies its own locating pins. If the result is not determined, it
    passes the query up to the placement of the enclosing scope. If the result
    is not determined when we reach the root placement, the query is passed on
    to generic rules and definitions in the session. Like, by default, we route
    output to the master bus.

[side note: (1), (2), (3) is implemented. (4) is drafted, (5) is future work]

Now, when the Builder is triggered, it collects all placements and creates
a priority queue of tasks. This way, it finds out at what point to start.
It then prompts the corresponding placement at that points to resolve
themselves. Which, of course, creates additional tasks, to be enqueued.
Especially of interest here is, that, during that process, a placement
might issue a "WiringPlug". Simply put, when the placement finds out that
it ought be connected to xyz, say "music subgroup", it issues the demand
"I want to be connected to music subgroup". At that point we do not know
what "music subgroup" is. But, as we proceed, at some point (hopefully)
we'll encounter another placement (likely a pipe), which issues a
"WiringClaim": this placement (directed by its LocatingPins) demands
"I am music subgroup". Now this allows the Builder to issue a WiringRequest.
Probably you've got the idea by now. Progressively we'll order the elements,
and at some point we're able to build up connection paths, starting from
the media sources. This is where we begin to drop off processing nodes,
which are wired backwards (each node knows only its predecessors).
We end up with a small number of exit nodes, which can be "pulled"
to start the actual rendering.


I'd better stop at that point. Peter, I am well aware that I haven't really
answered your question but hopefully I was able to indicate the direction
towards which to head for a solution. Probably you've also gotten an intuitive
understanding, that this system of placements can embody anything a nodal
editing system can express, and it can even embody more. But the real challenge
is how to shape an interface to enable users and allow them to use these powers.
And this is a challenge which lies ahead, is not solved yet.

Cheers!
Hermann Vosseler
(aka "Ichthyo")