knife vault : Problem escaping string in JSON

[dheerajd-msys]

Description

knife vault update some-vault serverabc -J data.json

where data.json contains:

{
"id": "serverabc",
"agent_service_password": "blahblah\blah",
"database_service_password": "blahblah"
}

When viewing this as json, the single backslash in the "agent_service_password" object does not appear

Chef Version

chef --version

Chef Workstation version: 0.11.21

Chef Infra Client version: 15.4.45

Chef InSpec version: 4.18.0

Chef CLI version: 2.0.0

Test Kitchen version: 2.3.4

Cookstyle version: 5.10.13


Related Issue

https://getchef.zendesk.com/agent/tickets/24443

Replication Case

Create a json with example data:

{
"id": "serverabc",
"agent_service_password": "blahblah\blah",
"database_service_password": "blahblah"
}

Run knife vault show some-vault serverabc -F json and the \ will not appear

Expected Behaviour

Backslash should be able to appear in string if escaped properly.

[sanga1794]

Hi @tas50 I have worked on some part of this issue, My analysis is as below

1. Issue is reproducible while creating vault(adding values from command itself)

2. Got root cause, its happening beaause of JSON.parse() and its known issue.
   e.g.

   irb(main):030:0> x = '{"username": "root", "password": "abc\abc"}'
   => "{"username": "root", "password": "abc\abc"}"

   irb(main):031:0> JSON.parse(x)
   => {"username"=>"root", "password"=>"abcabc"}

   irb(main):032:0> YAML.safe_load(x)
   => {"username"=>"root", "password"=>"abc\abc"}

3. So tried to resolve particular scenario using YAML.safe_load() (as shown in point 2)

4. But again problem arises with the loading, this saved knife-vault item with format json, its showing some unicode characters like
   {
   "id": "root",
   "username": "root",
   "password": "abc\u0007bc"
   }

   if we use format as yaml it showing correctly,

   We used FFI_Yajl::Parser to fetch data from databag

@tas50 , Please share your thoughts so that we can proceed?

[btm]

@sanga1794 you need to escape the backslash (more than you might think), because backslash is a special escape character.

in this example, it looks like there is a backslash in the password but that is actually \a not \ and a.

irb(main):030:0> x = '{"username": "root", "password": "abc\abc"}'
=> "{"username": "root", "password": "abc\abc"}"

To understand that visually, look at \n:

irb(main):021:0> string = "\n\n\n"
irb(main):022:0> string
=> "\n\n\n"
irb(main):023:0> puts string



=> nil

The string "looks like" it has three backslashes in it, but those are escape characters, combined with n that means newline, not the characters \ and n.

In this example:

{
"id": "serverabc",
"agent_service_password": "blahblah\blah",
"database_service_password": "blahblah"
}

That should probably be this, to get one backslash:

{
"id": "serverabc",
"agent_service_password": "blahblah\\\\blah",
"database_service_password": "blahblah"
}

You can also see this work out by creating a hash in ruby with a properly escaped backslash (only has to be \\ in a ruby string, and you'll see that JSON.generate produces \\\\:

irb(main):016:0> creds_hash = {"username"=>"root", "password"=>"abc\\abc"}
irb(main):017:0> creds_hash
=> {"username"=>"root", "password"=>"abc\\abc"}
irb(main):018:0> creds_json = JSON.generate(creds_hash)
irb(main):019:0> creds_json
=> "{\"username\":\"root\",\"password\":\"abc\\\\abc\"}"
irb(main):020:0> JSON.parse(creds_json)
=> {"username"=>"root", "password"=>"abc\\abc"}

If the JSON file is being created by hand, the user has to be aware of the requirement that any \ character is represented by \\\\. We can't do much to fix that directly. We can 1) document it and 2) warn about it.

ChefConfig::PathHelper.printable? may be useful here. You can see how we use it to warn a user that they may have the wrong number of escape characters in ChefConfig::PathHelper.validate_path.

https://github.com/chef/chef/blob/master/chef-config/lib/chef-config/path_helper.rb

Maybe you could do the same thing in ChefVault::Mixin::Helper.values_from_json and have ChefVault::Log.warn if the unparsed or parsed JSON contains non-printable characters (which is what something like \a would be). I'm not sure if parsed or unparsed is the right place to check yet. it would be easier to help the user if we checked the parsed json, because we could print the k and value that were the concern in the warning.

[btm]

Here's further example:

irb(main):039:1* def self.printable?(string)
irb(main):040:2*   if string =~ /[^[:print:]]/
irb(main):041:2*     false
irb(main):042:2*   else
irb(main):043:2*     true
irb(main):044:1*   end
irb(main):045:0> end
=> :printable?
irb(main):046:0> printable?("foo")
=> true
irb(main):047:0> printable?("abc\abc")
=> false