fix(import/export): Improve Jinja handling logic

[Vitor-Avila]

Since the CLI supports Jinja templating to dynamically modify the asset contents during the import, it automatically escapes existing Jinja templating syntax from assets during the export, so that they are ignored by the CLI renderer until import (and properly preserved to be handled by Superset).

Recently the query_context was introduced to chart exports, which is a JSON structure converted to a string. The CLI was incorrectly escaping the JSON structure ({"key_one: {"key_two: "value" {{ '}}' }}) as if it was Jinja templating syntax during the export, causing the import to fail.

This PR improves the Jinja escaping logic, trying to load any string field as a JSON (the most common field is the query_context but we have other JSON fields that are exported as strings) so that only Jinja syntax is escaped.

It also improves the Jinja rendering logic during import, by performing a YAML -> JSON conversion in case a TemplateSyntaxError is raised.

[betodealmeida]

The CLI was incorrectly escaping the JSON structure ({"key_one: {"key_two: "value" {{ '}}' }}) as if it was Jinja templating syntax during the export, causing the import to fail.

Can you explain why this would fail? I thought on import we'd apply Jinja rendering, so that this:

{"key_one: {"key_two: "value" {{ '}}' }}

Would get rendered to this before import:

{"key_one: {"key_two: "value" }}

Is the root problem that we don't apply Jinja rendering to certain fields like query_context but apply the escaping when exporting?

[Vitor-Avila]

The CLI was incorrectly escaping the JSON structure ({"key_one: {"key_two: "value" {{ '}}' }}) as if it was Jinja templating syntax during the export, causing the import to fail.

Can you explain why this would fail? I thought on import we'd apply Jinja rendering, so that this:

{"key_one: {"key_two: "value" {{ '}}' }}

Would get rendered to this before import:

{"key_one: {"key_two: "value" }}

Is the root problem that we don't apply Jinja rendering to certain fields like query_context but apply the escaping when exporting?

It's actually a problem on rendering the query_context -- If you trigger a sync command using a YAML with just this line:

query_context: '{"queries": [{"custom_form_data":{{{ '}}' }}]}'

The import throws this error:

File "/Users/vitoravila/code/backend-sdk/src/preset_cli/cli/superset/sync/native/command.py", line 101, in render_yaml
    template = Template(input_.read())
  File "/Users/vitoravila/.pyenv/versions/local-cli/lib/python3.9/site-packages/jinja2/environment.py", line 1208, in __new__
    return env.from_string(source, template_class=cls)
  File "/Users/vitoravila/.pyenv/versions/local-cli/lib/python3.9/site-packages/jinja2/environment.py", line 1105, in from_string
    return cls.from_code(self, self.compile(source), gs, None)
  File "/Users/vitoravila/.pyenv/versions/local-cli/lib/python3.9/site-packages/jinja2/environment.py", line 768, in compile
    self.handle_exception(source=source_hint)
  File "/Users/vitoravila/.pyenv/versions/local-cli/lib/python3.9/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

The query_context content is exported as a str (not as a JSON) so I'm un-sure why this is happening. The solution I found was updating the escaping logic to only escape actual field values, and then still during the import for some cases it would throw a TemplateSyntaxError, but then converting the YAML to JSON would work.

I'm open to testing other approaches if you have any ideas!

[betodealmeida]

Ah, the problem is that this:

query_context: '{"queries": [{"custom_form_data":{{{ '}}' }}]}'

Should be parsed by Jinja as:

query_context: '{"queries": [{"custom_form_data":{TEMPLATE_START '}}' TEMPLATE_END]}'

But instead is parsed as:

query_context: '{"queries": [{"custom_form_data":TEMPLATE_START{ '}}' TEMPLATE_END]}'

Which is invalid. So the problem is that when exporting we're replacing every } with {{ '}}' }}, but then we end with invalid Jinja2.

[betodealmeida]

Let's merge this and then we can improve the export later.