[BUG] use of tpldir variable does not allow the formula to be wrapped

[eedgar]

Your setup

Formula commit hash / release tag

starting from e2e1be1

Versions reports (master & minion)

Salt Version:
Salt: 3002.1

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: Not Installed
Python: 3.8.5 (default, Jul 28 2020, 12:59:40)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2

System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-52-generic
system: Linux
version: Ubuntu 20.04 focal

Pillar / config used

trying to extend the formula by creating a new www/init.sls and having the contents look like this
{% from "apache/map.jinja" import apache with context %}
include:

• apache
  apache-config-custom-global:
  file.managed:
  • name: {{ apache.confdir }}/custom-global.{{ apache.confext }}
  • source: salt://www/files/custom-global.conf
  • mode: 644
  • makedirs: True
  • template: {{ apache.get('template_engine', 'jinja') }}
  • context:
    apache: {{ apache|json }}
  • require:
    • pkg: apache-package-install-pkg-installed
  • watch_in:
    • module: apache-service-running-restart
  • require_in:
    • module: apache-service-running-restart
    • module: apache-service-running-reload
    • service: apache-service-running

gives this error
local:
Data failed to compile:
----------
Rendering SLS 'base:www' failed: Jinja error: www/defaults.yaml
/var/cache/salt/minion/files/base/apache/map.jinja(5):
---
# -- coding: utf-8 --
# vim: ft=jinja

{%- set tplroot = tpldir.split('/')[0] %}
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}    <======================
{%- import_yaml tplroot ~ "/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}
{%- import_yaml tplroot ~ "/oscodenamemap.yaml" as oscodename %}
[...]

the workaround

{% set curr_tpldir = tpldir %}
{% set tpldir = 'apache' %}
{% from "apache/map.jinja" import apache with context %}
{% set tpldir = curr_tpldir %}
include:

• apache
  apache-config-custom-global:
  file.managed:
  • name: {{ apache.confdir }}/custom-global.{{ apache.confext }}
  • source: salt://www/files/custom-global.conf
  • mode: 644
  • makedirs: True
  • template: {{ apache.get('template_engine', 'jinja') }}
  • context:
    apache: {{ apache|json }}
  • require:
    • pkg: apache-package-install-pkg-installed
  • watch_in:
    • module: apache-service-running-restart
  • require_in:
    • module: apache-service-running-restart
    • module: apache-service-running-reload
    • service: apache-service-running

---

Bug details

Describe the bug

replace the tpldir/tplroot logic with hardcoded apache paths

Steps to reproduce the bug

Expected behaviour

Attempts to fix the bug

Additional context

[baby-gnu]

Interesting point.

I must admit that I always extend formulas by providing new sls under the same salt:// path.

With the following master configuration:

fileserver_backend:
  - roots
  - gitfs

I can create /srv/salt/apache/www/foo.sls and salt will lookup that sls with the following URL salt://apache/www/foo.sls.

I'll make some try tomorrow and post my finding.

[baby-gnu]

It took a little bit longer but I finally made little tests with relative imports and it's not trivial:

• using relative import of map.jinja in .sls files is working (at least with my 3001 test bed)
• using relative import of files in map.jinja is not working because the relative pathes are expanded relatively to the .sls file

So, salt 'testmachine2' state.show_sls foo.test-relative-import works but salt 'testmachine2' state.show_sls foo.subcomponent.test-relative-import does not.

To make my test, I created the following files:

• foo/test-relavite-import.sls

  {%- from "./map.jinja" import mapdata with context %}
  
  foo/test-relative-import:
    test.nop:
      - name: {{ mapdata }}

• foo/subcomponent/test-relative-import.sls

  {%- from "../map.jinja" import mapdata with context %}
  
  foo/subcomponent/test-relative-import:
   test.nop:
     - name: {{ mapdata }}

• foo/map.jinja

  {%- from "./test-relative-sub-import.jinja" import submapdata %}
  
  {%- set mapdata = submapdata %}

• foo/test-relative-sub-import.jinja

  {%- set submapdata = {'foo': 'bar'} %}

• bar/test-relative-import.sls

  {%- from "foo/map.jinja" import mapdata with context %}
  
  bar/test-relative-import:
    test.nop:
      - name: {{ mapdata }}

• bar/subcomponent/test-relative-import.sls

  {%- from "foo/map.jinja" import mapdata with context %}
  
  bar/subcomponent/test-relative-import:
    test.nop:
      - name: {{ mapdata }}

Here are the outputs:

• salt 'testmachine2' state.show_sls foo.test-relative-import

  testmachine2.example.net:
      ----------
      foo/test:
          ----------
          __env__:
              base
          __sls__:
              foo.test-relative-import
          test:
              |_
                ----------
                name:
                    ----------
                    foo:
                        bar
              - nop
              |_
                ----------
                order:
                    10000
  

• salt 'testmachine2*' state.show_sls foo.subcomponent.test-relative-import

  testmachine2.example.net:
      - Rendering SLS 'base:foo.subcomponent.test-relative-import' failed: Jinja error: ./test-relative-sub-import.jinja
        /var/cache/salt/minion/files/base/foo/map.jinja(1):
        ---
        {%- from "./test-relative-sub-import.jinja" import submapdata %}    <======================
        
        {%- set mapdata = submapdata %}
        ---
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 400, in render_jinja_tmpl
            output = template.render(**decoded_context)
          File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
            return original_render(self, *args, **kwargs)
          File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
            return self.environment.handle_exception(exc_info, True)
          File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
            reraise(exc_type, exc_value, tb)
          File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
            raise value.with_traceback(tb)
          File "<template>", line 1, in top-level template code
          File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1073, in make_module
            return TemplateModule(self, self.new_context(vars, shared, locals))
          File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1152, in __init__
            body_stream = list(template.root_render_func(context))
          File "/var/cache/salt/minion/files/base/foo/map.jinja", line 1, in top-level template code
            {%- from "./test-relative-sub-import.jinja" import submapdata %}
          File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 204, in get_source
            raise TemplateNotFound(template)
        jinja2.exceptions.TemplateNotFound: ./test-relative-sub-import.jinja
  

• salt 'testmachine2*' state.show_sls bar.test-relative-import

  testmachine2.example.net:
      ----------
      bar/test-relative-import:
          ----------
          __env__:
              base
          __sls__:
              bar.test-relative-import
          test:
              |_
                ----------
                name:
                    ----------
                    foo:
                        bar
              - nop
              |_
                ----------
                order:
                    10000
  

• salt 'testmachine2*' state.show_sls bar.subcomponent.test-relative-import

  testmachine2.example.net:
      ----------
      bar/subcomponent/test-relative-import:
          ----------
          __env__:
              base
          __sls__:
              bar.subcomponent.test-relative-import
          test:
              |_
                ----------
                name:
                    ----------
                    foo:
                        bar
              - nop
              |_
                ----------
                order:
                    10000
  

This could be solved if we find a consistent way to find the path of the current included/imported file instead of current sls file:

• having the real path of the imported map.jinja (in my test, tlpdir is set to bar when importing it in bar/test-relative-import.sls using with context but stay foo without the context (now I wonder why we require the context)
• remove the trailing /map.jinja from that path
• import relatively to that directory

Is there a way to achive this consistently across salt versions? I'm a bit lost about what is possible or not [1], [2] :-/

@myii: do you have any hint?

Regards.

[baby-gnu]

I made a little test:

• take openssh-formula and openntpd-formula which have the _mapdata verification
• removing all the with context when importing from map.jinja
• run the bin/kitchen verify for all supported platforms

The Inspec _mapdata_spec.rb see no difference in the mapdata contents…

The load with context is only useful when the imported .jinja require to access variables set in the .sls.

• without the with context, map.jinja can't use the something variable

  {%- set something = "this is something" %}
  
  {#- Get the `tplroot` from `tpldir` #}
  {%- set tplroot = tpldir.split('/')[0] %}
  {%- from tplroot ~ "/map.jinja" import mapdata %}

• with the with context, map.jinja can use the something variable

  {%- set something = "this is something" %}
  
  {#- Get the `tplroot` from `tpldir` #}
  {%- set tplroot = tpldir.split('/')[0] %}
  {%- from tplroot ~ "/map.jinja" import mapdata with context %}

I think we could say that map.jinja must be loaded without with context and then, map.jinja lookup files relatively to it's tpldir.

[baby-gnu]

I made more tests and I think this solution may be interesting:

{#- Make sure `map.jinja` is imported without the context #}
{#- Import `with context` override the `tplfile` and `tpldir` variables #}
{%- if not tplfile.endswith("/map.jinja") %}
{{ raise("Import error: map.jinja must be imported without context. tplfile='" ~ tplfile ~ "'") }}
{%- endif %}

{#- `tplroot` is `tpldir`: the directory where is `map.jinja` #}
{%- set tplroot = tpldir %}

It's working with v5 map.jinja for openvpn-formula for all kitchen supported platforms but not with relative imports like ../map.jinja because tpldir is ../ in this case.

This means that for the above example, I must remove all the with context and replace ../map.jinja by foo/map.jinja in foo.subcomponent.test-relative-import.

I could even add another test and raise in map.jinja:

{%- if tplfile.startswith("../") %}
{{ raise("Import error: map.jinja must be imported with absolute path. tplfile='" ~ tplfile ~ "'") }}
{%- endif %}

This will permit to load map.jinja of whatever formula from any other without troubles.

[kritzi-at]

tpldir does not work with older versions.

for example it does not work Salt Version 3000.6

[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 169, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 404, in render_jinja_tmpl
    buf=tmplstr)
SaltRenderError: Jinja variable 'tpldir' is undefined
/var/cache/salt/minion/files/base/apache/map.jinja(4):
---

# -*- coding: utf-8 -*-
# vim: ft=jinja

{%- set tplroot = tpldir.split('/')[0] %}    <======================
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}
[...]
---

This is a big issue for non zero-day admins

[baby-gnu]

tpldir does not work with older versions.

for example it does not work Salt Version 3000.6

Hello @kritzi-at, that's strange because I tested on all supported platforms with a test bed test-tplvars-formula:

• test-tplvars-formula/test-tplvars/test.sls

  {%- from "test-tplvars/test-tplvars.jinja" import jinjafile, jinjadir %}
  
  test-tplvars-variable:
    file.managed:
      - name: /tmp/tplvars.txt
      - source: salt://test-tplvars/tplvars.txt.jinja
      - template: jinja
      - context:
          jinjafile: {{ jinjafile }}
          jinjadir: {{ jinjadir }}

• test-tplvars-formula/test-tplvars/tplvars.txt.jinja

  tplfile: {{ jinjafile }}
  tpldir: {{ jinjadir }}

• test-tplvars-formula/test-tplvars/test-tplvars.jinja

  {%- if not tplfile.endswith('/test-tplvars.jinja') %}
  {{ raise("Import error: map.jinja must be imported without context") }}
  {%- endif %}
  
  {%- set jinjafile = tplfile %}
  {%- set jinjadir = tpldir %}

• test-tplvars-formula/test/integration/default/controls/tplvars.rb

  # frozen_string_literal: true
  
  control "test-tplvars/test" do
    title 'check value of tplvars for imported jinja file'
  
    describe file('/tmp/tplvars.txt') do
      it { should be_file }
      its('content') { should include "tplfile: test-tplvars/test-tplvars.jinja" }
      its('content') { should include "tpldir: test-tplvars" }
    end
  end

I use kitchen files from template-formula with minor changes for this pseudo formula:

test-tplvars-formula/kitchen.yml file

# -*- coding: utf-8 -*-
# vim: ft=yaml
---
# For help on this file's format, see https://kitchen.ci/
driver:
  name: docker
  use_sudo: false
  privileged: true
  run_command: /lib/systemd/systemd

# Make sure the platforms listed below match up with
# the `env.matrix` instances defined in `.travis.yml`
platforms:
  ## SALT `tiamat`
  - name: debian-10-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:debian-10
  - name: debian-9-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:debian-9
  - name: ubuntu-2004-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:ubuntu-20.04
  - name: ubuntu-1804-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:ubuntu-18.04
  - name: ubuntu-1604-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:ubuntu-16.04
  - name: centos-8-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:centos-8
  - name: centos-7-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:centos-7
  - name: amazonlinux-2-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:amazonlinux-2
  - name: oraclelinux-8-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:oraclelinux-8
  - name: oraclelinux-7-tiamat-py3
    driver:
      image: saltimages/salt-tiamat-py3:oraclelinux-7

  ## SALT `master`
  - name: debian-10-master-py3
    driver:
      image: saltimages/salt-master-py3:debian-10
  - name: ubuntu-2004-master-py3
    driver:
      image: saltimages/salt-master-py3:ubuntu-20.04
  - name: ubuntu-1804-master-py3
    driver:
      image: saltimages/salt-master-py3:ubuntu-18.04
  - name: centos-8-master-py3
    driver:
      image: saltimages/salt-master-py3:centos-8
  - name: fedora-32-master-py3
    driver:
      image: saltimages/salt-master-py3:fedora-32
  - name: fedora-31-master-py3
    driver:
      image: saltimages/salt-master-py3:fedora-31
  - name: opensuse-leap-152-master-py3
    driver:
      image: saltimages/salt-master-py3:opensuse-leap-15.2
      run_command: /usr/lib/systemd/systemd
    # Workaround to avoid intermittent failures on `opensuse-leap-15.2`:
    # => SCP did not finish successfully (255):  (Net::SCP::Error)
    transport:
      max_ssh_sessions: 1
  - name: amazonlinux-2-master-py3
    driver:
      image: saltimages/salt-master-py3:amazonlinux-2
  - name: gentoo-stage3-latest-master-py3
    driver:
      image: saltimages/salt-master-py3:gentoo-stage3-latest
      run_command: /sbin/init
  - name: gentoo-stage3-systemd-master-py3
    driver:
      image: saltimages/salt-master-py3:gentoo-stage3-systemd

  ## SALT `3001`
  - name: debian-10-3001-py3
    driver:
      image: saltimages/salt-3001-py3:debian-10
  - name: debian-9-3001-py3
    driver:
      image: saltimages/salt-3001-py3:debian-9
  - name: ubuntu-2004-3001-py3
    driver:
      image: saltimages/salt-3001-py3:ubuntu-20.04
  - name: ubuntu-1804-3001-py3
    driver:
      image: saltimages/salt-3001-py3:ubuntu-18.04
  - name: centos-8-3001-py3
    driver:
      image: saltimages/salt-3001-py3:centos-8
  - name: centos-7-3001-py3
    driver:
      image: saltimages/salt-3001-py3:centos-7
  - name: fedora-32-3001-py3
    driver:
      image: saltimages/salt-3001-py3:fedora-32
  - name: fedora-31-3001-py3
    driver:
      image: saltimages/salt-3001-py3:fedora-31
  - name: opensuse-leap-152-3001-py3
    driver:
      image: saltimages/salt-3001-py3:opensuse-leap-15.2
      run_command: /usr/lib/systemd/systemd
    # Workaround to avoid intermittent failures on `opensuse-leap-15.2`:
    # => SCP did not finish successfully (255):  (Net::SCP::Error)
    transport:
      max_ssh_sessions: 1
  - name: amazonlinux-2-3001-py3
    driver:
      image: saltimages/salt-3001-py3:amazonlinux-2
  - name: oraclelinux-8-3001-py3
    driver:
      image: saltimages/salt-3001-py3:oraclelinux-8
  - name: oraclelinux-7-3001-py3
    driver:
      image: saltimages/salt-3001-py3:oraclelinux-7
  - name: gentoo-stage3-latest-3001-py3
    driver:
      image: saltimages/salt-3001-py3:gentoo-stage3-latest
      run_command: /sbin/init
  - name: gentoo-stage3-systemd-3001-py3
    driver:
      image: saltimages/salt-3001-py3:gentoo-stage3-systemd

  ## SALT `3000.3`
  - name: debian-10-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:debian-10
  - name: debian-9-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:debian-9
  - name: ubuntu-1804-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:ubuntu-18.04
  - name: centos-8-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:centos-8
  - name: centos-7-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:centos-7
  - name: fedora-31-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:fedora-31
  - name: opensuse-leap-152-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:opensuse-leap-15.2
      run_command: /usr/lib/systemd/systemd
    # Workaround to avoid intermittent failures on `opensuse-leap-15.2`:
    # => SCP did not finish successfully (255):  (Net::SCP::Error)
    transport:
      max_ssh_sessions: 1
  - name: amazonlinux-2-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:amazonlinux-2
  - name: gentoo-stage3-latest-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:gentoo-stage3-latest
      run_command: /sbin/init
  - name: gentoo-stage3-systemd-3000-3-py3
    driver:
      image: saltimages/salt-3000.3-py3:gentoo-stage3-systemd
  - name: ubuntu-1804-3000-3-py2
    driver:
      image: saltimages/salt-3000.3-py2:ubuntu-18.04
  - name: ubuntu-1604-3000-3-py2
    driver:
      image: saltimages/salt-3000.3-py2:ubuntu-16.04
  - name: arch-base-latest-3000-3-py2
    driver:
      image: saltimages/salt-3000.3-py2:arch-base-latest
      run_command: /usr/lib/systemd/systemd

  ## SALT `2019.2`
  - name: centos-6-2019-2-py2
    driver:
      image: saltimages/salt-2019.2-py2:centos-6
      run_command: /sbin/init
  - name: amazonlinux-1-2019-2-py2
    driver:
      image: saltimages/salt-2019.2-py2:amazonlinux-1
      run_command: /sbin/init

provisioner:
  name: salt_solo
  log_level: debug
  salt_install: none
  require_chef: false
  formula: test-tplvars
  salt_copy_filter:
    - .kitchen
    - .git

verifier:
  # https://www.inspec.io/
  name: inspec
  sudo: true
  # cli, documentation, html, progress, json, json-min, json-rspec, junit
  reporter:
    - cli

suites:
  - name: default
    provisioner:
      state_top:
        base:
          '*':
            - test-tplvars.test
    verifier:
      inspec_tests:
        - path: test/integration/default

After running ./bin/kitchen verify, I have only errors for centos-6 and amazonlinux-1 which could no be created.

./bin/kitchen list

Instance                                  Driver  Provisioner  Verifier  Transport  Last Action    Last Error
default-debian-10-tiamat-py3              Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-debian-9-tiamat-py3               Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-2004-tiamat-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-1804-tiamat-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-1604-tiamat-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-centos-8-tiamat-py3               Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-centos-7-tiamat-py3               Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-amazonlinux-2-tiamat-py3          Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-oraclelinux-8-tiamat-py3          Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-oraclelinux-7-tiamat-py3          Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-debian-10-master-py3              Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-2004-master-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-1804-master-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-centos-8-master-py3               Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-fedora-32-master-py3              Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-fedora-31-master-py3              Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-opensuse-leap-152-master-py3      Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-amazonlinux-2-master-py3          Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-gentoo-stage3-latest-master-py3   Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-gentoo-stage3-systemd-master-py3  Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-debian-10-3001-py3                Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-debian-9-3001-py3                 Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-2004-3001-py3              Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-1804-3001-py3              Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-centos-8-3001-py3                 Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-centos-7-3001-py3                 Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-fedora-32-3001-py3                Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-fedora-31-3001-py3                Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-opensuse-leap-152-3001-py3        Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-amazonlinux-2-3001-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-oraclelinux-8-3001-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-oraclelinux-7-3001-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-gentoo-stage3-latest-3001-py3     Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-gentoo-stage3-systemd-3001-py3    Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-debian-10-3000-3-py3              Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-debian-9-3000-3-py3               Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-1804-3000-3-py3            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-centos-8-3000-3-py3               Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-centos-7-3000-3-py3               Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-fedora-31-3000-3-py3              Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-opensuse-leap-152-3000-3-py3      Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-amazonlinux-2-3000-3-py3          Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-gentoo-stage3-latest-3000-3-py3   Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-gentoo-stage3-systemd-3000-3-py3  Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-1804-3000-3-py2            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-ubuntu-1604-3000-3-py2            Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-arch-base-latest-3000-3-py2       Docker  SaltSolo     Inspec    Ssh        Verified       <None>
default-centos-6-2019-2-py2               Docker  SaltSolo     Inspec    Ssh        <Not Created>  Kitchen::ShellOut::ShellCommandFailed
default-amazonlinux-1-2019-2-py2          Docker  SaltSolo     Inspec    Ssh        <Not Created>  Kitchen::ShellOut::ShellCommandFailed

[kritzi-at]

I have now upgraded to salt-minion 3002.2
Error is still the same:

[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 5, in top-level template code
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1073, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1152, in __init__
    body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/apache/map.jinja", line 4, in top-level template code
    {%- set tplroot = tpldir.split('/')[0] %}
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 430, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'tpldir' is undefined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 260, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 505, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'tpldir' is undefined
/var/cache/salt/minion/files/base/apache/map.jinja(4):
---
# -*- coding: utf-8 -*-
# vim: ft=jinja

{%- set tplroot = tpldir.split('/')[0] %}    <======================
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}
[...]
---

When i use slsutil.renderer the tpldir seems to be defined, but not as expected:

# salt-call slsutil.renderer salt://apache/map.jinja
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 5, in top-level template code
  File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 198, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: ./defaults.yaml

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 260, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 543, in render_jinja_tmpl
    "Jinja error: {}{}".format(exc, out), line, tmplstr, trace=tracestr
salt.exceptions.SaltRenderError: Jinja error: ./defaults.yaml
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 5, in top-level template code
  File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 198, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: ./defaults.yaml

; line 5

---
# -*- coding: utf-8 -*-
# vim: ft=jinja

{%- set tplroot = tpldir.split('/')[0] %}
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}    <======================
{%- import_yaml tplroot ~ "/osarchmap.yaml" as osarchmap %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}
{%- import_yaml tplroot ~ "/oscodenamemap.yaml" as oscodename %}
[...]
---

[baby-gnu]

Hello @kritzi-at, can you try a salt 'minion' state.apply test-tplvars.test with my tests files?

Note that apache-formula is working on different environment as you can see in the last pipeline run, even on a 2017.7.

[kritzi-at]

Hello @baby-gnu

Test script works fine on Ubuntu with minion 3002.2

# salt-call state.apply test-tplvars.test
local:
----------
          ID: test-tplvars-variable
    Function: file.managed
        Name: /tmp/tplvars.txt
      Result: True
     Comment: File /tmp/tplvars.txt updated
     Started: 16:05:23.627108
    Duration: 75.168 ms
     Changes:
              ----------
              diff:
                  New file
              mode:
                  0644

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  75.168 ms
# cat /tmp/tplvars.txt
tplfile: test-tplvars/test-tplvars.jinja
tpldir: test-tplvars

and on CentOS with minion 3000.6

# salt-call state.apply test-tplvars.test
local:
  Name: /tmp/tplvars.txt - Function: file.managed - Result: Changed Started: - 16:05:00.786759 Duration: 74.754 ms

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  74.754 ms
# cat /tmp/tplvars.txt
tplfile: test-tplvars/test-tplvars.jinja
tpldir: test-tplvars

[baby-gnu]

Thanks @kritzi-at, so now I'm wondering why applying apache-formula is not working for you.

Can you explain a little your setup and how you run the formula?

[kritzi-at]

Hello @baby-gnu, i found the issue.
The problem was related to the template_file i set in the pillar.
The template file was custom made and included apache/map.jinja without using it.

Removing the line fixed everything.

[baby-gnu]

More tests to see the differences between with context and without context : https://gist.github.com/baby-gnu/c8c6d1b423ac64b88011ffba2b4e884d

So, only tplfile is different and with context the tplroot variable is propagated from the .sls to the imported .jinja file.