buffer: remove error for malformatted hex string #12012

Trott · 2017-03-23T23:40:21Z

Improve accuracy of the error message when a hex string of an incorrect
length is send to .write() or .fill().

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

Affected core subsystem(s)

buffer

Trott · 2017-03-23T23:41:50Z

Rather than attempting to adjust the behavior of our code to match the error message (as in #3773) this takes the approach of adjusting the error message to more accurately report the issue that is found. (Of course, we can always implement more complete and robust error checking like in #3773 at a later date if deemed desirable and performant.)

Trott · 2017-03-24T01:36:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/7011/

targos

Maybe a RangeError is appropriate?

targos · 2017-03-24T06:04:22Z

test/parallel/test-buffer-alloc.js

@@ -512,7 +512,8 @@ assert.strictEqual(Buffer.from('=bad'.repeat(1e4), 'base64').length, 0);

 // Test single hex character throws TypeError


nit: comment should be updated

Updated, thanks!

joyeecheung · 2017-03-24T06:33:17Z

src/node_buffer.cc

@@ -616,7 +616,7 @@ void Fill(const FunctionCallbackInfo<Value>& args) {
      enc == UCS2 ? str_obj->Length() * sizeof(uint16_t) : str_obj->Length();

  if (enc == HEX && str_length  % 2 != 0)
-    return env->ThrowTypeError("Invalid hex string");
+    return env->ThrowError("Invalid hex string length");


Maybe we can be a bit more explicit about it? Something like String length must be a multiple of 2 if encoding is "hex"?

Perhaps hex string length must be a multiple of 2

I'm starting to think this check isn't really worthwhile and the right thing to do is remove it. This error is still misleading in that if you get it, one would reasonably expect there's more validation going on for hex strings than just a length check. One would reasonably expect that we check for valid hex chars. But we don't. We happily allow out-of-range characters just as long as the string length is divisible by two.

The upshot is puzzling behavior like this:

> Buffer.from('xx', 'hex') <Buffer > > Buffer.from('x', 'hex') TypeError: Invalid hex string at Buffer.write (buffer.js:769:21) at fromString (buffer.js:213:18) at Function.Buffer.from (buffer.js:105:12) at repl:1:8 at ContextifyScript.Script.runInThisContext (vm.js:23:33) at REPLServer.defaultEval (repl.js:339:29) at bound (domain.js:280:14) at REPLServer.runBound [as eval] (domain.js:293:12) at REPLServer.onLine (repl.js:536:10) at emitOne (events.js:101:20) > Buffer.from('abxx', 'hex') <Buffer ab> > Buffer.from('abx', 'hex') TypeError: Invalid hex string at Buffer.write (buffer.js:769:21) at fromString (buffer.js:213:18) at Function.Buffer.from (buffer.js:105:12) at repl:1:8 at ContextifyScript.Script.runInThisContext (vm.js:23:33) at REPLServer.defaultEval (repl.js:339:29) at bound (domain.js:280:14) at REPLServer.runBound [as eval] (domain.js:293:12) at REPLServer.onLine (repl.js:536:10) at emitOne (events.js:101:20) >

It would be a whole lot less puzzling if this happened:

> Buffer.from('xx', 'hex') <Buffer > > Buffer.from('x', 'hex') <Buffer > > Buffer.from('abxx', 'hex') <Buffer ab> > Buffer.from('abx', 'hex') <Buffer ab> >

Principle Of Least Astonishment to me says we either accept anything and do the best we can or else we do reasonably rigorous error checking.

Trott · 2017-03-24T18:20:13Z

Maybe a RangeError is appropriate?

If I got a RangeError, I would expect it to be something like the string contained characters that were invalid hex digits (/[^a-fA-F0-9]/). This error we're talking about here seems more like "invalid formatting" than "out of acceptable range" to me...

Trott · 2017-03-24T18:30:03Z

Reprinting an above comment down here for more visibility:

I'm starting to think this check isn't really worthwhile and the right thing to do is remove it. This error is still misleading in that if you get it, one would reasonably expect there's more validation going on for hex strings than just a length check. One would reasonably expect that we check for valid hex chars. But we don't. We happily allow out-of-range characters just as long as the string length is divisible by two.

The upshot is puzzling behavior like this:

> Buffer.from('xx', 'hex')
<Buffer >
> Buffer.from('x', 'hex')
TypeError: Invalid hex string
    at Buffer.write (buffer.js:769:21)
    at fromString (buffer.js:213:18)
    at Function.Buffer.from (buffer.js:105:12)
    at repl:1:8
    at ContextifyScript.Script.runInThisContext (vm.js:23:33)
    at REPLServer.defaultEval (repl.js:339:29)
    at bound (domain.js:280:14)
    at REPLServer.runBound [as eval] (domain.js:293:12)
    at REPLServer.onLine (repl.js:536:10)
    at emitOne (events.js:101:20)
> Buffer.from('abxx', 'hex')
<Buffer ab>
> Buffer.from('abx', 'hex')
TypeError: Invalid hex string
    at Buffer.write (buffer.js:769:21)
    at fromString (buffer.js:213:18)
    at Function.Buffer.from (buffer.js:105:12)
    at repl:1:8
    at ContextifyScript.Script.runInThisContext (vm.js:23:33)
    at REPLServer.defaultEval (repl.js:339:29)
    at bound (domain.js:280:14)
    at REPLServer.runBound [as eval] (domain.js:293:12)
    at REPLServer.onLine (repl.js:536:10)
    at emitOne (events.js:101:20)
>

It would be a whole lot less puzzling if this happened:

> Buffer.from('xx', 'hex')
<Buffer >
> Buffer.from('x', 'hex')
<Buffer >
> Buffer.from('abxx', 'hex')
<Buffer ab>
> Buffer.from('abx', 'hex')
<Buffer ab>
>

Principle Of Least Astonishment to me says we either accept anything and do the best we can or else we do reasonably rigorous error checking.

addaleax · 2017-03-24T22:58:28Z

accept anything and do the best we can

That’s mostly what we do for base64, so doing it for hex too sounds good to me

Trott · 2017-03-25T05:02:11Z

That’s mostly what we do for base64, so doing it for hex too sounds good to me

Done! PTAL

addaleax · 2017-03-25T07:00:22Z

@targos @cjihrig You probably want to re-review too now that this PR does something quite different :)

targos · 2017-03-25T07:09:13Z

test/parallel/test-buffer-alloc.js

-// - https://github.com/nodejs/node/issues/6770
-assert.throws(() => Buffer.from('A', 'hex'), TypeError);
+// Test single hex character is discarded.
+assert.strictEqual(Buffer.from('A', 'hex').length, 0);


Can you add a test where the resulting length is >0 ?
Like Buffer.from('abx', 'hex')

@targos Good idea. Added!

Remove error message when a hex string of an incorrect length is sent to .write() or .fill(). Fixes: nodejs#3770

jgeewax · 2017-03-26T15:10:59Z

(Just chiming in as this relates to #4877)

It's clear that there's precedent here with base64 (it clearly just swallows any invalid characters):

> Buffer.from('not !**# base64 at all_... !!!!', 'base64').toString('base64')
'notbase64atall//'

Would it make sense to also add "strict mode" where invalid values are rejected?

In other words, I'd guess there is a not-insignificant group of folks out there who would prefer to be told when garbage goes into creating a new Buffer (since they're saying what the format should be). A strict mode wouldn't hurt those of us who just want the buffer to do "just do the best it can", and is super helpful to those of us that want to be notified when we accidentally send invalid values into the Buffer.

A few options that seem reasonable:

Flags like RegExp: Buffer.from(input, 'hex', 's')
Boolean for strict-mode: Buffer.from(input, 'base64', true)
A separate encoding type: Buffer.from(input, 'hex-strict')

Not sure if it matters also, but short-cut validation methods that check whether the Buffer.from(input, 'base64').toString('base64') == input could be useful (this might be out of scope of the Buffer class though...)

Trott · 2017-03-26T19:38:25Z

@jgeewax I definitely see that sort of error-checking safety as useful behavior and while I wish it was what Node.js did, it does seem like there's no reason that validating input like that can't be implemented as a userland module. EDIT: And if a userland module can show a way to do it in a performant way, that might even be a path into Node.js core. My sense right now is that most of the objections/hesitance about expanding the error checking is the performance hit incurred.

addaleax · 2017-03-26T23:57:20Z

@jgeewax If we were designing Node from scratch, we’d probably implement one of your suggestions (which all make sense from an API perspective), but I think they all come with some kind of negative performance impact that we would want to avoid now.

Trott · 2017-03-28T04:50:08Z

CI: CI: https://ci.nodejs.org/job/node-test-pull-request/7068/

Trott · 2017-03-28T18:35:42Z

Landed in 682573c

Remove error message when a hex string of an incorrect length is sent to .write() or .fill(). PR-URL: nodejs#12012 Fixes: nodejs#3770 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

buffer behaviour change caused by: nodejs/node#12012 landed in: nodejs/node@682573c11

Trott added buffer Issues and PRs related to the buffer subsystem. semver-major PRs that contain breaking changes and should be released in the next major version. labels Mar 23, 2017

Trott added this to the 8.0.0 milestone Mar 23, 2017

nodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. labels Mar 23, 2017

addaleax approved these changes Mar 24, 2017

View reviewed changes

targos approved these changes Mar 24, 2017

View reviewed changes

Trott force-pushed the hex-fix branch from 6d668a7 to b26d5e4 Compare March 24, 2017 06:15

joyeecheung reviewed Mar 24, 2017

View reviewed changes

cjihrig approved these changes Mar 24, 2017

View reviewed changes

Trott force-pushed the hex-fix branch from b26d5e4 to ea79e24 Compare March 25, 2017 05:01

Trott changed the title ~~buffer: better error for malformated hex string~~ buffer: remove error for malformated hex string Mar 25, 2017

Trott force-pushed the hex-fix branch from ea79e24 to d6a406d Compare March 25, 2017 05:21

Trott changed the title ~~buffer: remove error for malformated hex string~~ buffer: remove error for malformatted hex string Mar 25, 2017

addaleax approved these changes Mar 25, 2017

View reviewed changes

targos approved these changes Mar 25, 2017

View reviewed changes

lpinca approved these changes Mar 25, 2017

View reviewed changes

cjihrig approved these changes Mar 25, 2017

View reviewed changes

buffer: remove error for malformatted hex string

afa3e1b

Remove error message when a hex string of an incorrect length is sent to .write() or .fill(). Fixes: nodejs#3770

Trott force-pushed the hex-fix branch from d6a406d to afa3e1b Compare March 25, 2017 17:49

jgeewax mentioned this pull request Mar 26, 2017

Clarify Buffer hex string error message #4877

Closed

thefourtheye approved these changes Mar 28, 2017

View reviewed changes

Trott closed this Mar 28, 2017

jasnell mentioned this pull request Apr 4, 2017

8.0.0 Release Proposal #12220

Closed

leonardodino added a commit to leonardodino/basic-crypto that referenced this pull request Oct 31, 2017

fix build on node 8

412772f

buffer behaviour change caused by: nodejs/node#12012 landed in: nodejs/node@682573c11

shuse2 mentioned this pull request Feb 28, 2018

Add multiple node.js version support - Closes #561 LiskArchive/lisk-elements#562

Merged

3 tasks

melloc mentioned this pull request Jul 31, 2018

Buffer no longer performs length check for hex strings in node v8.x TritonDataCenter/node-sshpk#52

Closed

Trott deleted the hex-fix branch January 13, 2022 22:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buffer: remove error for malformatted hex string #12012

buffer: remove error for malformatted hex string #12012

Trott commented Mar 23, 2017

Trott commented Mar 23, 2017

Trott commented Mar 24, 2017

targos left a comment

targos Mar 24, 2017

Trott Mar 24, 2017

joyeecheung Mar 24, 2017

jasnell Mar 24, 2017

Trott Mar 24, 2017 •

edited

Loading

Trott commented Mar 24, 2017

Trott commented Mar 24, 2017

addaleax commented Mar 24, 2017

Trott commented Mar 25, 2017

addaleax commented Mar 25, 2017

targos Mar 25, 2017

Trott Mar 25, 2017

jgeewax commented Mar 26, 2017

Trott commented Mar 26, 2017 •

edited

Loading

addaleax commented Mar 26, 2017

Trott commented Mar 28, 2017

Trott commented Mar 28, 2017

		@@ -512,7 +512,8 @@ assert.strictEqual(Buffer.from('=bad'.repeat(1e4), 'base64').length, 0);

		// Test single hex character throws TypeError

buffer: remove error for malformatted hex string #12012

buffer: remove error for malformatted hex string #12012

Conversation

Trott commented Mar 23, 2017

Checklist

Affected core subsystem(s)

Trott commented Mar 23, 2017

Trott commented Mar 24, 2017

targos left a comment

Choose a reason for hiding this comment

targos Mar 24, 2017

Choose a reason for hiding this comment

Trott Mar 24, 2017

Choose a reason for hiding this comment

joyeecheung Mar 24, 2017

Choose a reason for hiding this comment

jasnell Mar 24, 2017

Choose a reason for hiding this comment

Trott Mar 24, 2017 • edited Loading

Choose a reason for hiding this comment

Trott commented Mar 24, 2017

Trott commented Mar 24, 2017

addaleax commented Mar 24, 2017

Trott commented Mar 25, 2017

addaleax commented Mar 25, 2017

targos Mar 25, 2017

Choose a reason for hiding this comment

Trott Mar 25, 2017

Choose a reason for hiding this comment

jgeewax commented Mar 26, 2017

Trott commented Mar 26, 2017 • edited Loading

addaleax commented Mar 26, 2017

Trott commented Mar 28, 2017

Trott commented Mar 28, 2017

Trott Mar 24, 2017 •

edited

Loading

Trott commented Mar 26, 2017 •

edited

Loading