Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to delivery data from same Producer Factory with different params #274

Closed
conderls opened this issue Oct 19, 2022 · 2 comments
Closed

Comments

@conderls
Copy link

If I have a producer Factory whose result depends on param label,

class ProducerFactory(label: String) extends Factory[String] {
   override def deliveryId = "delivery_id"
   ...
  override def get() = label
}

I may use the data from ProducerFactory in another Factory by

class ConsumerFactory(label: String) extends Factory[String] {
   @Delivery(id = "delivery_id")
   private val label: String = ""
   ...
}

I can override deliveryId in ProducerFactory according to the param label, e.g. override def deliveryId = s"delivery_id_$label" for sure.

  1. I am wondering that, the ConsumerFactory is generic enough, If I have Producers with deliveryId: "delivery_id_a", "delivery_id_b", how can I using ConsumerFactory to consume data from these Producers? As I cannot set deliveryId in annotation according to the param label
   @Delivery(id = s"delivery_id_$label")
   private val label: String = "" // ERROR raised: annotation argument needs to be a constant;
  1. right now, I am creating new pipeline to use ProducerFactory and ConsumerFactory with fixed deliveryId; but how to clean the pipeline if there are some cached/persisted data.

Originally posted by @conderls in #263

@glc-mfelja
Copy link

glc-mfelja commented Nov 16, 2022

Hello @conderls,
In order to generate the workflow based on relationships between factories and steps, the predictability of dependencies is mandatory, so dynamic delivery ids will broke this requirement.
I don't think this is something that we can acheive, at least not with the current design of the framework.
@qxzzxq if you want to add something ? if not you can close the ticket

@qxzzxq
Copy link
Member

qxzzxq commented Nov 17, 2022

Hi @conderls,
As @glc-mfelja says, the workflow must be deterministic at runtime. SETL does not support dynamic delivery. This is also due to the fact that Java/Scala's annotation does not support dynamically generated strings.

@qxzzxq qxzzxq closed this as completed Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants