[Scheduled Reports] Handling create/execute jobs with scheduling in mind

[tsullivan]

ExportTypeRegistry is what provides the "create job" and "execute job" functions for each Reporting integration instance. This issue outlines how the ExportTypeRegistry will be carried over in Reporting when it uses Task Manager for queuing.

The flow of creating a reporting job has differences between "ad-hoc" reports and scheduled reports. When reports are scheduled and it is time to execute the job, Task Manager goes straight to the "Execute Job" flow.

Another difference between ad-hock and scheduled reports, is that ad-hoc reports will have a "pending" phase where they are waiting for a Kibana instance, but scheduled reports will not have that phase.

Create Job (For Ad-Hoc)

Generating a report begins with an incoming user POST request. The request body has contains a keyword field for the exportType (payload.jobType) for type of job, which is used by the ExportTypeRegistry to find the corresponding ExportTypeDefinition.

The create job request handler (referred to as controller) gives the request ExportTypeDefinition's schedule function, stores the "pending" report in historical data. The controller then saves the document in the historical index and calls Task Manager to fire an ad-hoc reporting:execute task.

To support only Ad-Hoc, there does not have to be much info in the parameters of the scheduled task. We would be able to just track the pending report's ID so it can be claimed and executed later. However, to support both ad-hoc and scheduled reports in the simplest way, the task parameters should have the full information about the report, including the encrypted headers.

Create Job (For Scheduled)

Creating a job for scheduled reports refers to creating the task parameters that are passed to Task Manager, and are part of the task instance context when the task runs. Task parameters for scheduled reports contain complete information about the report, plus encrypted headers: enough context to execute a reporting job from scratch each time the schedule is triggered.

A sample of task instance parameters (same for ad-hoc and scheduled):

• Schedule name (report name)
• URL of the page (PNG/PDF) or saved object ID/search source (CSV)
• File type (PNG/PDF/CSV)
• time range in the time picker
• "created_by" username (assuming users will only be able to see the schedules they created)
• encrypted request headers, or API token
• Array of actions (slack, email, etc)
• Recurrence (time unit string or cron expression)

Execute Job

The run function of the reporting:execute task type receives incoming tasks triggered by Task Manager, and passes the job to the corresponding ExportTypeDefinition execute job function.

As mentioned above, reports will come to the same execute flow whether they are ad-hoc or scheduled, and have the full information about the report in task parameters. There will be slight differences in the flow between ad-hoc and scheduled reports:

• For ad-hoc reports, the task parameters will have a document ID reference pointing to a pending job that needs to be updated to "processing."
  • It would be convenient to get the report info from the pending document. To keep it simple though,the Report info should always come from task params.
• For scheduled reports, execution must begin with calculating the next runAt time.
  • Get the next timestamp by adding the recurrence interval to the current time
  • Alternatively, an interval field can be given in the scheduling data, but we would not be able to support custom intervals

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)