Task genrator debug api #9058

saurabhd336 · 2022-07-14T17:41:08Z

Save task generator info into an inmemory map, and provide a controller API to fetch the details for a given table
design doc: https://docs.google.com/document/d/1bCn13k57WLSaIAQGfATP75VS3p6V_ievlBi_HoOtmZ8/edit?usp=sharing

Release Notes

Added an API GET /tasks/generator/{tableNameWithType}/{taskType}/debug which returns the success time / error message for the 5 most recent task generator runs.

saurabhd336 · 2022-07-14T17:43:39Z

@npawar @mcvsubbu I've raised this PR in place of the now closed PR #9043

mcvsubbu · 2022-07-14T20:05:34Z

Overall, I am a plus 1 on more minion debuggability.

My view is that we should add APIs to minions similar to that of servers, and pull debugging info from there and display it.

npawar · 2022-07-14T20:08:03Z

Overall, I am a plus 1 on more minion debuggability.

My view is that we should add APIs to minions similar to that of servers, and pull debugging info from there and display it.

We already have minion APIs (look for /debug endpoint in Tasks tab). This is for scheduler errors, and the scope of scheduler remains limited to the controller, hence thought it makes most sense as an API here

npawar · 2022-07-15T05:26:34Z

...controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java

+    } catch (Exception e) {
+      for (TableConfig tableConfig : enabledTableConfigs) {
+        try {
+          StringWriter errors = new StringWriter();


it seems we will do the same thing for each table config, and then put the same stacktrace for every table?

npawar · 2022-07-15T05:33:05Z

pinot-common/src/main/java/org/apache/pinot/common/utils/helix/FakePropertyStore.java

@@ -37,7 +37,13 @@ public FakePropertyStore() {

  @Override
  public ZNRecord get(String path, Stat stat, int options) {
-    return _contents.get(path);
+    ZNRecord znRecord = _contents.get(path);


is this change needed?

npawar · 2022-07-15T05:33:21Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+  /**
+   * Returns the current information version, it should be consistent with the corresponding {@link ZNRecord} version.
+   */
+  public int getVersion() {


prolly don't need this method either

npawar · 2022-07-15T05:36:30Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+    _tableNameWithType = tableNameWithType;
+    _taskType = taskType;
+    // sort and keep the most recent several error messages
+    _mostRecentErrorRunMessage = new TreeMap<>();


it seems this constructor is only ever called from within this same class, with empty list and map. Do we need all this sorting logic?

If at all we move to persistent storage, we might need to serialize / deserialize instances of this class using an allArgsConstructor

codecov-commenter · 2022-07-15T09:09:06Z

Codecov Report

Merging #9058 (35d8326) into master (f0a4b44) will decrease coverage by 1.31%.
The diff coverage is 26.38%.

❗ Current head 35d8326 differs from pull request most recent head 9fa0078. Consider uploading reports for the commit 9fa0078 to get more accurate results

@@             Coverage Diff              @@
##             master    #9058      +/-   ##
============================================
- Coverage     70.06%   68.74%   -1.32%     
+ Complexity     4743     4656      -87     
============================================
  Files          1832     1838       +6     
  Lines         96971    97224     +253     
  Branches      14620    14640      +20     
============================================
- Hits          67939    66840    -1099     
- Misses        24305    25680    +1375     
+ Partials       4727     4704      -23

Flag	Coverage Δ
integration1	`26.25% <20.84%> (-0.14%)`	⬇️
integration2	`?`
unittests1	`66.85% <1.16%> (-0.06%)`	⬇️
unittests2	`15.30% <10.86%> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ache/pinot/broker/broker/AccessControlFactory.java	`73.33% <ø> (-6.67%)`	⬇️
...broker/broker/ZkBasicAuthAccessControlFactory.java	`0.00% <ø> (ø)`
...e/pinot/broker/broker/helix/BaseBrokerStarter.java	`76.99% <ø> (-2.82%)`	⬇️
.../BrokerResourceOnlineOfflineStateModelFactory.java	`55.81% <ø> (ø)`
...quota/HelixExternalViewBasedQueryQuotaManager.java	`67.71% <ø> (-1.35%)`	⬇️
...che/pinot/broker/routing/BrokerRoutingManager.java	`85.31% <ø> (ø)`
...elector/SegmentLineageBasedSegmentPreSelector.java	`100.00% <ø> (ø)`
.../segmentpreselector/SegmentPreSelectorFactory.java	`100.00% <ø> (ø)`
...oker/routing/segmentpruner/EmptySegmentPruner.java	`87.93% <ø> (ø)`
...mentpruner/MultiPartitionColumnsSegmentPruner.java	`73.91% <ø> (ø)`
... and 225 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

klsince · 2022-07-15T16:18:11Z

pinot-common/src/main/java/org/apache/pinot/common/minion/InMemoryTaskManagerStatusCache.java

+
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one


format: move this header to top

mcvsubbu · 2022-07-15T16:57:44Z

Overall, I am a plus 1 on more minion debuggability.
My view is that we should add APIs to minions similar to that of servers, and pull debugging info from there and display it.

We already have minion APIs (look for /debug endpoint in Tasks tab). This is for scheduler errors, and the scope of scheduler remains limited to the controller, hence thought it makes most sense as an API here

I don't see a minion debug REST endpoint in the code. Am I missing something?

mcvsubbu

@saurabhd336 thanks a lot for your contribution. It has been hard to debug minions, so I totally understand your pain point. However, I have a few general questions:

What is the use case where you think this will help? (we have some use cases where we saw problems, and instead of introducing one new API per use case, I am looking to make sure we do something that works for more than one. @sajjad-moradi can review from our standpoint).
I know @npawar referred to a debug API on the minion, but I could not locate one. Please correct me if I am wrong. I think we should issue an API to the minions as well (liek we do with servers) and get the latest status form them and display it in the controller.

mcvsubbu · 2022-07-15T19:10:20Z

...roller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java

+      @ApiParam(value = "Table name with type", required = true) @PathParam("tableNameWithType")
+          String tableNameWithType) {
+    BaseTaskGeneratorInfo
+        taskGeneratorMostRecentRunInfo = _taskManagerStatusCache.fetchTaskGeneratorInfo(tableNameWithType, taskType);


What if the table is not mastered on this controller?

@saurabhd336 thanks a lot for your contribution. It has been hard to debug minions, so I totally understand your pain point. However, I have a few general questions:

What is the use case where you think this will help? (we have some use cases where we saw problems, and instead of introducing one new API per use case, I am looking to make sure we do something that works for more than one. @sajjad-moradi can review from our standpoint).

I know @npawar referred to a debug API on the minion, but I could not locate one. Please correct me if I am wrong. I think we should issue an API to the minions as well (liek we do with servers) and get the latest status form them and display it in the controller.

Hey Subbu, this is the one for Minion tasks: /tasks/task/{taskName}/debug. In the list of subtasks returned, there's an "info" field which contains the exception (if any) encountered by this task

ah good catch about the lead controller! this won't work as is, we'll have to think of how to handle that

for 2. I feel it's not very convenient when one auto scales minion workers frequently based on pending tasks. The debug API would be either not reachable when no workers are around or get no past task states to check when new workers are added

But we have helix APIs for that, right? Has that avenue been fully explored? We are migrating to Helix 1.x) very soon (we are getting to production with that), so please check in 1.x pages what is available for task management and debugging, thanks. We will also do the same.
cc: @sajjad-moradi

There are two parts:

generating task configurations which happens in pinot code

submitting the task via TaskDriver which is in Helix code

Subbu has a valid point that helix should provide some info on failure on the 2nd part - task submission. The first part though is before reaching helix and it's good if we have an API to capture issues in task generation in pinot code.

Also scheduling tasks can be done by either:

periodic task generator job on lead controller, or

task REST endpoint which can be on any controller

So this means the approach in this PR doesn't work as expected.

Ack. The status can indeed be stored across different controllers. I was trying the following two approaches
Upon hitting this API,

The controller makes a HTTP call to every other controller to get their respective in-memory debug statuses, waits for the responses and finally responds back with a list of all the responses.

Use the helix messaging framework to broadcast a message to each controller, and via the AsyncCallback, collate all the replies and respond back (https://github.com/saurabhd336/pinot/pull/9/files)

I'll test and evaluate both approaches and update this PR accordingly.

saurabhd336 · 2022-07-20T08:14:12Z

@mcvsubbu @sajjad-moradi @npawar @klsince I've tried addressing the issue with the debug data being present across different controllers by letting the main controller query every other controller for the status data.
Please have a look at the updated PR

klsince

LGTM

klsince · 2022-07-20T17:41:34Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+   * @param ts A timestamp
+   */
+  public void addSuccessRunTs(long ts) {
+    _mostRecentSuccessRunTS.add(ts);


Q: any need to keep _mostRecentSuccessRunTS sorted?

Simplifies the eviction is all. Also the APi response is easier to work with (the most recent timestamp is the one of the top)

Does this indicate a successful run (completion) of the task, or a successful generation of the task? If the task fails in minion, does your API address it?

This only indicates the successful generation of a the task. This API is exclusively for task generation

klsince · 2022-07-20T17:48:08Z

...controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java

+      pinotTaskConfigs = taskGenerator.generateTasks(enabledTableConfigs);
+      for (TableConfig tableConfig : enabledTableConfigs) {
+        try {
+          _taskManagerStatusCache.saveTaskGeneratorInfo(tableConfig.getTableName(), taskGenerator.getTaskType(),


nit: add a saveTaskGeneratorInfoQuietly() to save the try-catch block for brevity.

With the in-memory storage, I don't see any exceptions that can be thrown. Have removed the try catch block

Jackie-Jiang · 2022-07-20T21:54:40Z

pinot-common/src/main/java/org/apache/pinot/common/minion/InMemoryTaskManagerStatusCache.java

+public class InMemoryTaskManagerStatusCache extends TaskManagerStatusCache<TaskGeneratorMostRecentRunInfo> {
+
+  private static class TaskGeneratorCacheKey {
+    String _tableNameWithType;


(minor) make them final

Jackie-Jiang · 2022-07-20T21:58:18Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+  private final String _taskType;
+  private final String _tableNameWithType;
+  // the timestamp to error message map of the most recent several error runs
+  @Nonnull


(minor) We don't usually use Nonnull. Everything without annotation are treated non-null

Jackie-Jiang · 2022-07-20T21:58:59Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+ * several error run
+ * message.


(minor)

Suggested change

* several error run

* message.

* several error run messages.

Jackie-Jiang · 2022-07-20T21:59:34Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+  @VisibleForTesting
+  static final String MOST_RECENT_SUCCESS_RUN_TS = "mostRecentSuccessRunTs";
+  @VisibleForTesting
+  static final String MOST_RECENT_ERROR_RUN_MESSAGE = "mostRecentErrorRunMessage";


Seems not used?

This was being used when we had ZK based storage. Removed

Jackie-Jiang · 2022-07-20T22:03:19Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+    _tableNameWithType = tableNameWithType;
+    _taskType = taskType;
+    // sort and keep the most recent several error messages
+    _mostRecentErrorRunMessage = new TreeMap<>();


Since we decided to not use ZNode to store the information, suggest simplifying it to only take tableNameWithType and taskType. We may change it when we decide to persist the info in the future.

Jackie-Jiang · 2022-07-20T22:06:32Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+  private final String _tableNameWithType;
+  // the timestamp to error message map of the most recent several error runs
+  @Nonnull
+  private final TreeMap<Long, String> _mostRecentErrorRunMessage;


Suggested change

private final TreeMap<Long, String> _mostRecentErrorRunMessage;

private final TreeMap<Long, String> _mostRecentErrorRunMessages;

Consider storing timestamp as java.sql.Timestamp so that the response can show human readable time

Jackie-Jiang · 2022-07-20T22:15:50Z

...roller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java

+      try {
+        result.add(JsonUtils.stringToJsonNode(resp));
+      } catch (IOException e) {
+        LOGGER.error("Failed to parse ");


Jackie-Jiang · 2022-07-20T22:25:16Z

...controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java

+    List<PinotTaskConfig> pinotTaskConfigs;
+    try {
+      pinotTaskConfigs = taskGenerator.generateTasks(enabledTableConfigs);
+      for (TableConfig tableConfig : enabledTableConfigs) {


Let's add a TODO here to separate the generated task/exception for each table. Currently even though some table doesn't have anything scheduled (e.g. enough tasks are scheduled from previous tables), we will still put success TS for it

Didn't get this. This method is only called for enabledTableConfigs i.e. only for tables for which the particular task is enabled?

Had a discussion offline. Added the todo

Jackie-Jiang

LGTM

Jackie-Jiang · 2022-07-21T21:28:25Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+  public TreeMap<String, String> getMostRecentErrorRunMessages() {
+    TreeMap<String, String> result = new TreeMap<>();
+    _mostRecentErrorRunMessages.forEach((timestamp, error) -> result.put(
+        OffsetDateTime.ofInstant(Instant.ofEpochMilli(timestamp), ZoneId.of("UTC")).toString(), error));


(minor)

Suggested change

OffsetDateTime.ofInstant(Instant.ofEpochMilli(timestamp), ZoneId.of("UTC")).toString(), error));

OffsetDateTime.ofInstant(Instant.ofEpochMilli(timestamp), ZoneOffset.UTC).toString(), error));

We can also consider putting ZoneId.systemDefault() which might be more readable, but it might also be confusing when the client is in a different timezone from the server. IMO both ways have pros and cons, so your call

I feel keeping everything UTC might be more user friendly especially when pinot clusters are deployed in different geographies. Keeping it UTC for now.

Jackie-Jiang · 2022-07-21T21:29:27Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskManagerStatusCache.java

+import java.util.function.Consumer;
+
+
+public abstract class TaskManagerStatusCache<T extends BaseTaskGeneratorInfo> {


Suggest changing it to an interface

mcvsubbu · 2022-07-26T18:57:06Z

...roller/src/main/java/org/apache/pinot/controller/api/resources/PinotTaskRestletResource.java

+    }
+
+    // Call all controllers
+    List<InstanceConfig> controllers = _pinotHelixResourceManager.getAllControllerInstanceConfigs();


you don't need to send request to all controllers. Just to the lead controller is enough

Since POST '/tasks/schedule' API lets one schedule tasks for a given table on any controller (non leader included), we do need to query all controllers here.

Perhaps I was not clear. You can field the request in any controller (like you do). If that is not the lead for the table, then locate the leader controller (see the class PinotLeadControllerRestletResource for how to do that) and then forward the request to just the lead controller.

Ah ok, so the POST actually has any controller schedule the job for a table ? Hmm... maybe we should fix hat too. But I see what you are saying here. My bad.

mcvsubbu · 2022-07-26T18:59:36Z

I had asked some questions about the use case you are tackling. Can you please answer those? How frequently are you generating tasks (in your use case)? Is a metric bump not enough for your use case? Can log processing help solving this use case?

saurabhd336 · 2022-07-27T08:48:43Z

@mcvsubbu

@npawar @Jackie-Jiang ? I might have just very rough, possibly inaccurate numbers.
I feel the need of a control plane level API within pinot to give an overall view into current and past state of minion tasks is of importance to us. Task generator being a key part of the entire minion task flow. While metrics can help to some extent, having details like failure stack traces etc might be difficult. This api avoids having to tally metrics and debug logs from a separate log processing system.
I suppose it could. But integrating the log processing framework into pinot APIs themselves might be a bit of a challenge. Having a system table for these kind of usecases might be the right way forward, such that pinot itself can store and serve debug data and status metrics for each of the components / flows. Essentially move from in-memory storage of logs and metrics into the system table

@npawar @Jackie-Jiang to add more

mcvsubbu · 2022-07-27T16:27:53Z

@mcvsubbu

@npawar @Jackie-Jiang ? I might have just very rough, possibly inaccurate numbers.

I feel the need of a control plane level API within pinot to give an overall view into current and past state of minion tasks is of importance to us. Task generator being a key part of the entire minion task flow. While metrics can help to some extent, having details like failure stack traces etc might be difficult. This api avoids having to tally metrics and debug logs from a separate log processing system.

I agree there is a need to debug minions. However this API does not provide any new insights other than what Helix already provides. So, let us work off of some concrete examples -- what is the use case in which you think this will help?

I suppose it could. But integrating the log processing framework into pinot APIs themselves might be a bit of a challenge. Having a system table for these kind of usecases might be the right way forward, such that pinot itself can store and serve debug data and status metrics for each of the components / flows. Essentially move from in-memory storage of logs and metrics into the system table

Every installation has its own way of processing logs. Cloud installations typically pipe the logs into a log processing service to get insights into it. On-prem installations have either their own way of piping the logs, or ways to get onto the host and look for the logs.

@npawar @Jackie-Jiang to add more

The reason I am reluctant to add more APIs is that the more we add, the more we need to keep backward compatibility on. My question to @saurabhd336 is : Have you looked at what Helix has to offer? If so, what are the short-comings?

npawar · 2022-07-27T20:11:48Z

@mcvsubbu

@npawar @Jackie-Jiang ? I might have just very rough, possibly inaccurate numbers.

I feel the need of a control plane level API within pinot to give an overall view into current and past state of minion tasks is of importance to us. Task generator being a key part of the entire minion task flow. While metrics can help to some extent, having details like failure stack traces etc might be difficult. This api avoids having to tally metrics and debug logs from a separate log processing system.

I suppose it could. But integrating the log processing framework into pinot APIs themselves might be a bit of a challenge. Having a system table for these kind of usecases might be the right way forward, such that pinot itself can store and serve debug data and status metrics for each of the components / flows. Essentially move from in-memory storage of logs and metrics into the system table

@npawar @Jackie-Jiang to add more

for 1: typically for users of RealtimeToOfflineTask, tasks get generated hourly. For SegmentGenerationAndPushTask, it can be way more frequently, depending on the the number of times files are generated in the source dir. MergeRollupTasks, are less frequent, but still several a day. There might be others, but this is what we see most commonly setup by users in oss. In a typical setup, all of these would be configured. It becomes quite confusing for users to have to find the exact exception in the logs, especially because some logs are in controller (scheduler related) and some in minion (task execution related). This API will help us make the feedback loop quicker, especially when we add this into the new Minion tab on the Pinot Admin UI

Regarding info already in Helix, these scheduler related exceptions are not present in the Helix generated metadata.

mcvsubbu · 2022-07-27T20:37:09Z

pinot-common/src/main/java/org/apache/pinot/common/minion/BaseTaskGeneratorInfo.java

+/**
+ * Base abstract class for task generator info.
+ */
+public abstract class BaseTaskGeneratorInfo {


Do you want to add implements Serializable

mcvsubbu · 2022-07-27T20:44:06Z

pinot-common/src/main/java/org/apache/pinot/common/minion/TaskGeneratorMostRecentRunInfo.java

+   * @param ts A timestamp
+   */
+  public void addSuccessRunTs(long ts) {
+    _mostRecentSuccessRunTS.add(ts);


Does this indicate a successful run (completion) of the task, or a successful generation of the task? If the task fails in minion, does your API address it?

mcvsubbu · 2022-07-27T20:49:59Z

@mcvsubbu

@npawar @Jackie-Jiang ? I might have just very rough, possibly inaccurate numbers.

I feel the need of a control plane level API within pinot to give an overall view into current and past state of minion tasks is of importance to us. Task generator being a key part of the entire minion task flow. While metrics can help to some extent, having details like failure stack traces etc might be difficult. This api avoids having to tally metrics and debug logs from a separate log processing system.

I suppose it could. But integrating the log processing framework into pinot APIs themselves might be a bit of a challenge. Having a system table for these kind of usecases might be the right way forward, such that pinot itself can store and serve debug data and status metrics for each of the components / flows. Essentially move from in-memory storage of logs and metrics into the system table

@npawar @Jackie-Jiang to add more

for 1: typically for users of RealtimeToOfflineTask, tasks get generated hourly. For SegmentGenerationAndPushTask, it can be way more frequently, depending on the the number of times files are generated in the source dir. MergeRollupTasks, are less frequent, but still several a day. There might be others, but this is what we see most commonly setup by users in oss. In a typical setup, all of these would be configured. It becomes quite confusing for users to have to find the exact exception in the logs, especially because some logs are in controller (scheduler related) and some in minion (task execution related). This API will help us make the feedback loop quicker, especially when we add this into the new Minion tab on the Pinot Admin UI

Regarding info already in Helix, these scheduler related exceptions are not present in the Helix generated metadata.

I still think log processing should be the answer to this (and other similar PRs that may come up in the future). We should not be adding a new API for every error condition we may encounter in the system (and log something).

@siddharthteotia , @npawar , @Jackie-Jiang , @snleee, @kishoreg, @mayankshriv what do you think? If the PMCs don't have any objection to this then I can live with this, but I am willing to bet that more such PRs will come up because logs are difficult to read.

saurabhd336 mentioned this pull request Jul 14, 2022

Task generator debug api #9043

Closed

npawar reviewed Jul 15, 2022

View reviewed changes

saurabhd336 force-pushed the taskGenratorDebugApi branch from 6420c75 to fd0e6bd Compare July 15, 2022 08:14

klsince reviewed Jul 15, 2022

View reviewed changes

mcvsubbu reviewed Jul 15, 2022

View reviewed changes

saurabhd336 force-pushed the taskGenratorDebugApi branch from fd0e6bd to e9dc106 Compare July 18, 2022 09:12

zhtaoxiang and others added 17 commits July 20, 2022 11:08

save task generator info to ZK

0186642

fix test case

ae25c2e

fix format

4336064

fix unit tests

da71ba8

Add task generator debug API

86b14a3

Review comments

29d1e2f

Change functionName

1177190

Move to inmemory cache for task generator

af6b497

Add 404 response

79a7ed1

Lint fix

efd4dfe

Change base class

9e33055

Comments

35d736f

Add http call to fetch responses

a5fe721

Fix build

18b03d3

Lint fix

8c5456a

Change to single object

4980e0d

Add scheme to url

aa80094

saurabhd336 force-pushed the taskGenratorDebugApi branch from 92e0153 to aa80094 Compare July 20, 2022 05:51

Lint

009ba40

klsince approved these changes Jul 20, 2022

View reviewed changes

Jackie-Jiang reviewed Jul 20, 2022

View reviewed changes

Saurabh Dubey added 3 commits July 21, 2022 13:44

Review comments

0d23c99

Change response timeformat

ec3c848

Add json ordering annotation

08d9c6a

Jackie-Jiang approved these changes Jul 21, 2022

View reviewed changes

Saurabh Dubey added 2 commits July 22, 2022 11:49

Review comments

f7a95b9

Change TimeZone offset

9fa0078

saurabhd336 force-pushed the taskGenratorDebugApi branch from 522743f to 9fa0078 Compare July 22, 2022 06:48

Jackie-Jiang added the release-notes Referenced by PRs that need attention when compiling the next release notes label Jul 22, 2022

Jackie-Jiang merged commit 463bf97 into apache:master Jul 22, 2022

mcvsubbu reviewed Jul 26, 2022

View reviewed changes

mcvsubbu reviewed Jul 27, 2022

View reviewed changes



		/**
		* Licensed to the Apache Software Foundation (ASF) under one

	private final TreeMap<Long, String> _mostRecentErrorRunMessage;
	private final TreeMap<Long, String> _mostRecentErrorRunMessages;

	OffsetDateTime.ofInstant(Instant.ofEpochMilli(timestamp), ZoneId.of("UTC")).toString(), error));
	OffsetDateTime.ofInstant(Instant.ofEpochMilli(timestamp), ZoneOffset.UTC).toString(), error));

		import java.util.function.Consumer;


		public abstract class TaskManagerStatusCache<T extends BaseTaskGeneratorInfo> {

Task genrator debug api #9058

Task genrator debug api #9058

Conversation

saurabhd336 commented Jul 14, 2022 • edited Loading

Release Notes

saurabhd336 commented Jul 14, 2022

mcvsubbu commented Jul 14, 2022

npawar commented Jul 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 15, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcvsubbu commented Jul 15, 2022

mcvsubbu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sajjad-moradi Jul 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saurabhd336 commented Jul 20, 2022

klsince left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jackie-Jiang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saurabhd336 Jul 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcvsubbu commented Jul 26, 2022

saurabhd336 commented Jul 27, 2022

mcvsubbu commented Jul 27, 2022

npawar commented Jul 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcvsubbu commented Jul 27, 2022

saurabhd336 commented Jul 14, 2022 •

edited

Loading

codecov-commenter commented Jul 15, 2022 •

edited

Loading

sajjad-moradi Jul 17, 2022 •

edited

Loading

saurabhd336 Jul 22, 2022 •

edited

Loading