Resolve "create edges in the workflow"

Closes #2243 (closed)

assigned to @ladinesa

added 1 commit

• baae2050 - Implementing sorting and linking of tasks

Compare with previous version

requested review from @jrudz

@jrudz this is my proposed implementation of the sorting and linking of tasks. Instead of explicitly introducing linked tasks, I added time_start and time_end in order to resolve overlapping and sequential tasks which can then be linked together appropriately. we can also simply introduce task.order

@aalbino

Hey @ladinesa, thx I think that start time and end time thing is too brittle to solely lean on that.

In experiments people are not filling these fields, or these fields are not always available in the raw files..

So for this reason I think it would be better to get the possibility of explicit task linking

Another reason is that users might want to link tasks in a more complicated way than a timeline sequence

Isn't this assuming a linear workflow graph?

Even for a linear workflow, defining start and end times does not really make sense in the context of this tool that creates the worklow yaml. There you really just want to generate a general directed graph to represent your workflow: https://fairmat-nfdi.github.io/nomad-utility-workflows/how_to/create_custom_workflows.html

If your proposal ended up being the only functionality then what I would do in this tool would be to automatically generate arbitrary start and end times so that the tasks would be ordered properly by NOMAD. I guess this is possible, but it seems a bit convoluted.

Could we have a series of ways to make connections performed according to priority:

1. explicit input/output matches (current implentation) -- links specific sections to the edges
2. explicit edge specifications (as I proposed above) -- links mainfiles/entries to the edges
3. automated ordering according to start/end times -- for cases where no explicit edges are given

Another problem with 3 is that there are some use cases with "floating nodes", i.e., the user may not want to have a fully connected workflow graph but rather say that a series of independent tasks were performed (especially in a sub workflow), so I would be a bit careful about auto-generating edges.

can we simply have then a link with task as section?

added 1 commit

• b73a48f3 - Resolve task inputs from linked task

Compare with previous version

added 1 commit

• 877c86b7 - Linting fix

Compare with previous version

At a quick glance, I believe this would work for me.

So, including an input whose section is a task results in an in-edge from this task. OR including an output whose section is a task results in an out-edge to this task. Correct?

@aalbino does this work for you? I think then you could easily set these during your normalization/mapping.

yes, except that i made a mistake in the normalization, it should always read from the task outputs. i can also remove the normalization actually and put the resolution in the visualizer, this is cleaner.

I don't know if this is the best solution.

This is what we currenlty do as a patch to show the arrows..

@hnaesstroem ?

added 1 commit

• a712c135 - Remove task normalization

Compare with previous version

Hi, @ladinesa. Should we test something about this feature?

yes, maybe test if a link is created when you provide a task as input to a task.

Are you thinking to this as the final thing? I'm not supporting it 100%..

@hnaesstroem how about you

Let's maybe discuss it at the meeting tomorrow at 10?

added 1 commit

• 7e24fe8c - Resolve link from outputs

Compare with previous version

marked this merge request as ready

merged

mentioned in commit 9eff3e95

mentioned in issue #2270