Job schedule pause_status from YAML Configuration Ignored upon Deployment

[alexerm]

In Databricks CLI v0.207.0, when defining a job in a YAML configuration file and setting the pause_status to UNPAUSED, the job is expected to be deployed to the Databricks platform in an active state. However, the job is always deployed in a paused state, requiring manual intervention to resume the job each time it's deployed. This behavior diverges from the expected behavior as defined by the pause_status setting in the YAML configuration.

Here's a snippet from a YAML configuration demonstrating the issue:

resources:
  jobs:
    example_job:
      name: example_job
      schedule:
        quartz_cron_expression: '0 0 5 ? * *'
        timezone_id: Europe/Berlin
        pause_status: UNPAUSED  # TODO: Doesn't work, need to resume job manually 

The commented TODO line indicates the problem. Despite pause_status: UNPAUSED being set, the job schedule is always deployed in a paused state.

Upon reviewing the relevant section of code in the Databricks CLI, it appears that the pause_status setting from the YAML configuration is being ignored, and the PauseStatus is being unconditionally set to Paused. Here's the snippet from the process_target_mode.go file:

if r.Jobs[i].Schedule != nil {
	r.Jobs[i].Schedule.PauseStatus = jobs.PauseStatusPaused
}

Expected Behavior:

• The job schedule should be deployed in an active state when pause_status: UNPAUSED is set in the YAML configuration.

Actual Behavior:

• The job schedule is always deployed in a paused state, regardless of the pause_status setting in the YAML configuration.

Disclaimer: I have recently begun working with this project and there's a possibility that my understanding of the issue might be incorrect. I appreciate any feedback or guidance to help clarify the situation and am open to suggestions to improve the deployment process. Thank you for your understanding and assistance.

Fix Suggestion:
It might be necessary to modify the code to check the pause_status setting from the YAML configuration and apply it accordingly during the deployment process. A potential fix could be to update the process_target_mode.go file to ensure PauseStatus is set based on the pause_status value from the YAML configuration:

if r.Jobs[i].Schedule != nil {
    if r.Jobs[i].Schedule.PauseStatus == "UNPAUSED" {
        r.Jobs[i].Schedule.PauseStatus = jobs.PauseStatusUnpaused
    } else {
        r.Jobs[i].Schedule.PauseStatus = jobs.PauseStatusPaused
    }
}
// Repeat similar condition checks for Continuous and Trigger

This issue requires users to manually resume the job each time it's deployed, which is error-prone and can lead to delays in job execution. It's crucial to have the pause_status setting respected to ensure smooth deployment and operation of job schedules on the Databricks platform. This behavior seems to be a bug or an unexpected oversight that needs addressing to align the deployment process with the configuration settings provided by users.

[pietern]

Thanks for reporting the issue. The active pausing of deployed jobs is a feature of mode: development. The rationale behind it is that while doing development, periodically running jobs is not what you'd want.

You can disable this behavior by removing mode: development from your target definition, but this also removes other defaults it implies, such as the name prefix of the resources under management.

cc @lennartkats-db

[alexerm]

Thank you for clarifying this, @pietern. It is indeed working fine in prod mode. While I can understand the rationale behind defaulting to a paused state in development mode to avoid inadvertently running jobs, the hardcoded nature of this behavior could be a hindrance to flexibility. Honestly, it was quite confusing, and I spent more time than I'd like to admit trying to figure out why my jobs weren't activating.

Would it be possible to allow a configuration override for the pause_status in development mode? This way, developers who wish to have their jobs active in development mode can do so with explicit configuration, while others can stick with the default paused state. This could potentially save other developers some confusion and time in the future.

Thank you for your time and consideration.

[lennartkats-db]

Thanks for the report @alexerm! Making it overridable makes sense. I think the main issue you're calling out is the discoverability of this feature. What we'd like to do is add a "description" to the job so this behavior is described in the Jobs UI.

[lennartkats-db]

@alexerm One more question, could you clarify your use case? Why would you want to run your personal development copy on a schedule? Is it possible your goal was to do a production deployment?

[pietern]

This was fixed in #885 and released in v0.210.0.