Identity json files are treated differently when starting the tunnel via command line vs systemd

[nickchen120235]

Issue

When starting the tunnel using command line ziti-edge-tunnel run --identity-dir /opt/openziti/etc/identities --verbose 4 --dns-ip-range 100.64.0.1/24, the identity json files created by ziti-edge-tunnel enroll are correctly loaded.

The relevant logs are

(2555)[        0.046]   DEBUG ziti-sdk:config.c:58 ziti_load_config() trying to load config from file[/opt/openziti/etc/identities/wan-client.json]                                                                                                           
(2555)[        0.046]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/opt/openziti/etc/identities/wan-client.json]                                                                                              
(2555)[        0.046]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/opt/openziti/etc/identities/wan-client.json]                                                                                                         
(2555)[        0.046]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1136 load_id_cb() identity[/opt/openziti/etc/identities/wan-client.json] loaded

The DNS query succeeded

root@openziti-wan-client:~# dig http-server.ziti @100.64.0.2

; <<>> DiG 9.18.24-1-Debian <<>> http-server.ziti @100.64.0.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25187
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;http-server.ziti.              IN      A

;; ANSWER SECTION:
http-server.ziti.       60      IN      A       100.64.0.3

;; Query time: 0 msec
;; SERVER: 100.64.0.2#53(100.64.0.2) (UDP)
;; WHEN: Mon Jul 15 15:02:02 CST 2024
;; MSG SIZE  rcvd: 61

However, when starting the tunnel using systemd systemctl start ziti-edge-tunnel.service, the identity json files seem to be recognized as enroll JWT tokens.

journalctl -xeu ziti-edge-tunnel shows

Jul 15 14:30:13 openziti-wan-client ziti-edge-tunnel[1776]: (1776)[        0.016]   DEBUG ziti-sdk:config.c:58 ziti_load_config() trying to load config from file[/opt/openziti/etc/identities/wan-client.json]
Jul 15 14:30:13 openziti-wan-client ziti-edge-tunnel[1776]: (1776)[        0.016]   ERROR ziti-sdk:model_support.c:67 parse_tokens() jsmn_parse() failed: -2

And indeed the DNS query fails

root@openziti-wan-client:~# systemctl start ziti-edge-tunnel.service 
root@openziti-wan-client:~# dig http-server.ziti @100.64.0.2

; <<>> DiG 9.18.24-1-Debian <<>> http-server.ziti @100.64.0.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 35339
;; flags: qr rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;http-server.ziti.              IN      A

;; Query time: 0 msec
;; SERVER: 100.64.0.2#53(100.64.0.2) (UDP)
;; WHEN: Mon Jul 15 15:01:14 CST 2024
;; MSG SIZE  rcvd: 45

Expected Behavior

Starting the tunnel with default systemd service file should behave the same as starting using command line.

Additional Information

Shipped systemd service file

root@openziti-wan-client:~# systemctl cat ziti-edge-tunnel.service 
# /lib/systemd/system/ziti-edge-tunnel.service
[Unit]
Description=Ziti Edge Tunnel
After=network-online.target

[Service]
Type=simple
EnvironmentFile=/opt/openziti/etc/ziti-edge-tunnel.env
User=ziti
UMask=0007
AmbientCapabilities=CAP_NET_ADMIN
ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh
ExecStart=/opt/openziti/bin/ziti-edge-tunnel run --verbose=${ZITI_VERBOSE} --dns-ip-range=${ZITI_DNS_IP_RANGE} --identity-dir=${ZITI_IDENTITY_DIR}
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

env file

root@openziti-wan-client:~# cat /opt/openziti/etc/ziti-edge-tunnel.env 
# all enrollment tokens named *.jwt are consumed and replaced with identity JSON files to be loaded at startup
ZITI_IDENTITY_DIR='/opt/openziti/etc/identities'

# reserved dynamic IP range for proxied services
ZITI_DNS_IP_RANGE='100.64.0.1/10'

# the log level specified in /var/lib/ziti/config.json has higher precedence than this env var; delete or modify that
# file or set via IPC "ziti-edge-tunnel set_log_level --loglevel DEBUG"
ZITI_VERBOSE=4

identity json file

{
        "ztAPI":"https://controller.openziti:1280",
        "id":{
                "cert":"<REDACTED>",
                "key":"<REDACTED>",
                "ca":"<REDACTED>"
                }
        }

[scareything]

Hello and thank you for the detailed report!

I was able to reproduce this issue with ziti-edge-tunnel v1.0.3, and I was also able to work around it by ensuring that the files in the identity directory are readable by the ziti user that the systemd unit uses to run ziti-edge-tunnel.

Apparently the code that checks if the identity file was able to be successfully opened is not working as expected. I'll dig into this further, but wanted to provide the potential workaround in the meantime.

[qrkourier]

If not practical to handle gracefully or correct within the ambient capabilities granted by systemd to ZET, an alternative is a privileged timer unit to correct owners and modes.

[nickchen120235]

and I was also able to work around it by ensuring that the files in the identity directory are readable by the ziti user that the systemd unit uses to run ziti-edge-tunnel.

Thank you for the workaround! I didn't realize it was a permission issue.