Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add proper DN construction #55

Merged
merged 10 commits into from
Nov 2, 2017
Merged

Conversation

csucu
Copy link

@csucu csucu commented Oct 27, 2017

Added RDN construction for TLS authentication.

@@ -847,6 +853,14 @@ func (s *Session) Login(cred *Credential) error {
defer socket.Release()

credCopy := *cred
if cred.Certificate != nil && cred.Username != "" {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably breaks the deployment for people that specify the Username themselves.
Perhaps
if cred.Mechanism == "MONGODB-X509" && cred.Username == "" {
credCopy.Username = getRFC2253NameString(cred.Certificate)
}
is enough.

@domodwyer and @szank you know more about mgo auth then me.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cred.Certificate is new so this is fine

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still shouldn't the check rely on the mechanism specified? rather than the presence of a certificate?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say. If the database is "$external" and the mechanism is "MONGODB-X509" then use the certificate. Overwrite the user name in the copy of the credentials. Otherwise use credentials as is.

Users might want to use both forms of authentication, for example. This would be the safest way.
Otherwise you could create an LoginWithCert method and make everyone's life easier.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason to pass a cert is to use it for authentication - if present, it's using cert-based auth, there's no reason to then set the mechanism to something else...

This code should set the username/mechanism/database as appropriate when a cert is given.

session.go Outdated
// getRFC2253NameString converts from an ASN.1 structured representation of the certificate
// to a UTF-8 string representation(RDN) and returns it.
func getRFC2253NameString(certificate *x509.Certificate) string {
var RDNElementsASN1 = pkix.RDNSequence{}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/RDNElementsASN1/rdnSequence

The elements in there are not ASN1 anymore.

session.go Outdated
// to a UTF-8 string representation(RDN) and returns it.
func getRFC2253NameString(certificate *x509.Certificate) string {
var RDNElementsASN1 = pkix.RDNSequence{}
asn1.Unmarshal(certificate.RawSubject, &RDNElementsASN1)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to not swallow the error and return it to the caller as this indicates a broken/bad certificate and would be good to know.

session.go Outdated
asn1.Unmarshal(certificate.RawSubject, &RDNElementsASN1)

var RDNElementsString = []string{}
for i := len(RDNElementsASN1) - 1; i >= 0; i-- {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to iterate from the back? DN ordering?
If so this will need a comment

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is. Something about AD field ordering.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the sequence is in reverse order, in the RFC it says "the output consists of the string encodings of each RelativeDistinguishedName in the RDNSequence, starting with the last element of the sequence and moving backwards toward the first."

session.go Outdated
for _, attribute := range RDNElementsASN1[i] {
var shortAttributeName = rdnOIDToShortName(attribute.Type)
if len(shortAttributeName) > 0 {
var attributeValueString = attribute.Value.(string)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not immediately clear to me, why certain DN elements need sanitizing and the ones we don't understand don't. The logic here could be much tidier if we sanitize everything.
@szank @domodwyer ?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to the RFC if the element OID is not one of the "well known ones" you hex encode the DER representation of the element. Please see my comment below.
The safest option here is to return an error on any RDN attribute type we don't know about (any outside the "well known ones".

Users can still use such certs, they just need to use openssl to return a proper string representation of the RDN.
Also, IIRC in mgo 3.4 the server itself converts the RDN to string, so setting the user name in the client is not required.

session.go Outdated

var RDNElementsString = []string{}
for i := len(RDNElementsASN1) - 1; i >= 0; i-- {
var nameAndValueList = []string{}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a critical path I think, but as a tip:

var nameAndValueList = make([]string,len(RDNElementsASN1[i))

session.go Outdated
attributeValueString = "\\" + attributeValueString
}
// escape trailing space (unless the trailing space is also the first (unescaped) character)
if len(attributeValueString) > 2 && attributeValueString[len(attributeValueString)-1] == ' ' {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szank mgo doesn't handle trailing spaces in RDN?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this is RFC requirement. Nothing to do with mgo. Leading and trailing spaces needs to be escaped.
But if the RDN contains one character, " ", then we don't want to escape it twice.

session.go Outdated
@@ -847,6 +853,14 @@ func (s *Session) Login(cred *Credential) error {
defer socket.Release()

credCopy := *cred
if cred.Certificate != nil && cred.Username != "" {
return fmt.Errorf("Failed to login, both certifcate and credentials are given.")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more efficient to use errors.New() when you don't require a formatter, and errors typically (should) start with a lowercase letter.

Have a quick read of https://github.com/golang/go/wiki/CodeReviewComments though the existing mgo code doesn't exactly follow it

session.go Outdated
@@ -825,6 +828,9 @@ type Credential struct {
// Mechanism defines the protocol for credential negotiation.
// Defaults to "MONGODB-CR".
Mechanism string

// Certificate defines an x509 certificate for authentication at login.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation here should specify the constraints - specifically:

  • The Username field is populated from the cert and should not be set
  • The Mechanism field should be MONGODB-X509 or not set (automatically populate in this case)
  • The Source field should be $external or not set (auto populate too)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

session.go Outdated
var nameAndValueList = []string{}
for _, attribute := range RDNElementsASN1[i] {
var shortAttributeName = rdnOIDToShortName(attribute.Type)
if len(shortAttributeName) > 0 {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Invert conditional and continue to avoid the if...else...

session.go Outdated
}

// escape , = + < > # ;
var replacer = strings.NewReplacer(",", "\\,", "=", "\\=", "+", "\\+", "<", "\\<", ">", "\\>", "#", "\\#", ";", "\\;")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialise replacer outside of the loop and reuse

session.go Outdated

// getRFC2253NameString converts from an ASN.1 structured representation of the certificate
// to a UTF-8 string representation(RDN) and returns it.
func getRFC2253NameString(certificate *x509.Certificate) string {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should have unit tests covering at the very least, the examples in section 5 of the RFC - if you make a function that accepts the unmarshaled pkix.RDNSequence and does the actual work you should be able to unit test by the logic of this func.

If you do the above, getRFC2253NameString() becomes a helper func accepting a x509.Certificate that calls the actual logic func - maybe the helper func can be getRFC2253NameStringFromCert() that calls getRFC2253NameString() or something - up to you.

session.go Outdated
if len(shortAttributeName) > 0 {
var attributeValueString = attribute.Value.(string)
// escape leading space or #
if strings.IndexAny(attributeValueString, " #") == 0 {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strings.HasPrefix() would be more efficient here.

session.go Outdated
}

// escape , = + < > # ;
replacer = strings.NewReplacer(",", "\\,", "=", "\\=", "+", "\\+", "<", "\\<", ">", "\\>", "#", "\\#", ";", "\\;")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list of strings to replace does not change, strings.NewReplacer can be initialised once outside of all the loops.

session.go Outdated
}
var attributeValueString = attribute.Value.(string)
// escape leading space or #
if strings.HasPrefix(attributeValueString, " #") == true {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says space or #, the HasPrefix checks for space and #.
Should be
if strings.HasPrefix(attributeValueString, " ") || strings.HasPrefix(attributeValueString, "#") {

Also == true comparison in if statements are redundant.

session.go Outdated
attributeValueString = "\\" + attributeValueString
}
// escape trailing space (unless the trailing space is also the first (unescaped) character)
if len(attributeValueString) > 2 && attributeValueString[len(attributeValueString)-1] == ' ' {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is wrong. It should escape a trailing space, unless it is already escaped.
if strings.HasSuffix(attributeValueString, " ") && !strings.HasSuffix(attributeValueString, "\ ") {

tadukurow
tadukurow previously approved these changes Oct 31, 2017
session.go Outdated
//The elements in the sequence needs to be reversed when converting them
for i := len(*RDNElements) - 1; i >= 0; i-- {
var nameAndValueList = make([]string,len((*RDNElements)[i]))
var replacer = strings.NewReplacer(",", "\\,", "=", "\\=", "+", "\\+", "<", "\\<", ">", "\\>", "#", "\\#", ";", "\\;")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the replacer can be outside the outer loop

}
var attributeValueString = attribute.Value.(string)
// escape leading space or #
if strings.HasPrefix(attributeValueString, " ") || strings.HasPrefix(attributeValueString, "#") {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strings.HasPrefix(attributeValueString, "#") can be removed as the replacer already takes care of that one.

session.go Outdated
if strings.HasPrefix(attributeValueString, " ") || strings.HasPrefix(attributeValueString, "#") {
attributeValueString = "\\" + attributeValueString
}
// escape trailing space (unless the trailing space is also the first (unescaped) character)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change comment to read
// escape trailing space, unless it's already escaped

continue
}
var attributeValueString = attribute.Value.(string)
// escape leading space or #

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment needs updating

tadukurow
tadukurow previously approved these changes Oct 31, 2017
szank
szank previously requested changes Oct 31, 2017
@@ -847,6 +853,14 @@ func (s *Session) Login(cred *Credential) error {
defer socket.Release()

credCopy := *cred
if cred.Certificate != nil && cred.Username != "" {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am 99.99% sure that mongodb 3.4 doesn't need it. It will extract the user name from the client cert.

session.go Outdated
@@ -825,6 +828,9 @@ type Credential struct {
// Mechanism defines the protocol for credential negotiation.
// Defaults to "MONGODB-CR".
Mechanism string

// Certificate defines an x509 certificate for authentication at login.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -847,6 +853,14 @@ func (s *Session) Login(cred *Credential) error {
defer socket.Release()

credCopy := *cred
if cred.Certificate != nil && cred.Username != "" {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say. If the database is "$external" and the mechanism is "MONGODB-X509" then use the certificate. Overwrite the user name in the copy of the credentials. Otherwise use credentials as is.

Users might want to use both forms of authentication, for example. This would be the safest way.
Otherwise you could create an LoginWithCert method and make everyone's life easier.

session.go Outdated
asn1.Unmarshal(certificate.RawSubject, &RDNElementsASN1)

var RDNElementsString = []string{}
for i := len(RDNElementsASN1) - 1; i >= 0; i-- {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is. Something about AD field ordering.

session.go Outdated
attributeValueString = "\\" + attributeValueString
}
// escape trailing space (unless the trailing space is also the first (unescaped) character)
if len(attributeValueString) > 2 && attributeValueString[len(attributeValueString)-1] == ' ' {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this is RFC requirement. Nothing to do with mgo. Leading and trailing spaces needs to be escaped.
But if the RDN contains one character, " ", then we don't want to escape it twice.

session.go Outdated
attributeValueString = replacer.Replace(attributeValueString)
nameAndValueList = append(nameAndValueList, fmt.Sprintf("%s=%s", shortAttributeName, attributeValueString))
} else {
nameAndValueList = append(nameAndValueList, fmt.Sprintf("%s=%X", attribute.Type.String(), attribute.Value.([]byte)))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the RFC:
If the AttributeValue is of a type which does not have a string
representation defined for it, then it is simply encoded as an
octothorpe character ('#' ASCII 35) followed by the hexadecimal
representation of each of the bytes of the BER encoding of the X.500
AttributeValue. This form SHOULD be used if the AttributeType is of
the dotted-decimal form.

Dotted decimal is OID, in human terms. We are missing the # before hex encoding tho.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, this code was not designed to handle cases like newline char 0x0A (\n) either. I have written a internal method that was good at converting our own cert's RDNs that weren't supposed to contain any stupid input, and had limited list of RDN field types.

This is the reason for few decisions here, like the one from line 5257. I am not sure I am handling unknown OIDS correctly ( even if we add the '#' sign there.

session.go Outdated
for _, attribute := range RDNElementsASN1[i] {
var shortAttributeName = rdnOIDToShortName(attribute.Type)
if len(shortAttributeName) > 0 {
var attributeValueString = attribute.Value.(string)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to the RFC if the element OID is not one of the "well known ones" you hex encode the DER representation of the element. Please see my comment below.
The safest option here is to return an error on any RDN attribute type we don't know about (any outside the "well known ones".

Users can still use such certs, they just need to use openssl to return a proper string representation of the RDN.
Also, IIRC in mgo 3.4 the server itself converts the RDN to string, so setting the user name in the client is not required.

session.go Outdated
// from the certificate to a UTF-8 string representation(RDN) and returns it.
func getRFC2253NameString(RDNElements *pkix.RDNSequence) string {
var RDNElementsString = []string{}
var replacer = strings.NewReplacer(",", "\\,", "=", "\\=", "+", "\\+", "<", "\\<", ">", "\\>", "#", "\\#", ";", "\\;")
Copy link

@szank szank Oct 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the RFC # has to be escaped only if it is the first character in the string.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, i'm not sure about this one, because in section 3 it has it listed as one of the special chars that needs to be escaped i think, unless i'm reading the grammer wrong. "special = "," / "=" / "+" / "<" / ">" / "#" / ";" " I also had looked at the way its done in java's openjdk and they escape the # throughout the string.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's slightly confusing, but Section 3 is for parsing a string back to DN. the section above states only leading '#' should be escaped, so I'd say '#' has to be dropped from the replacer and the HasPrefix('#') be added back in.

Copy link

@domodwyer domodwyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code is much cleaner, but I think we should set the source, mechanism & DB when using a cert for auth.

@@ -847,6 +853,14 @@ func (s *Session) Login(cred *Credential) error {
defer socket.Release()

credCopy := *cred
if cred.Certificate != nil && cred.Username != "" {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason to pass a cert is to use it for authentication - if present, it's using cert-based auth, there's no reason to then set the mechanism to something else...

This code should set the username/mechanism/database as appropriate when a cert is given.

session.go Outdated

// Certificate defines an x509 certificate for authentication at login.
// If providing a certificate:
// The Username field is populated from the cert and should not be set

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You would have to implement the "not specified" case and set the field for the user - it's just a helper thing.

session.go Outdated
@@ -847,6 +857,17 @@ func (s *Session) Login(cred *Credential) error {
defer socket.Release()

credCopy := *cred
if cred.Certificate != nil && cred.Username != "" {
return errors.New("failed to login, both certifcate and credentials are given")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

certificate is spelt wrong - sorry!

session.go Outdated
}
var attributeValueString = attribute.Value.(string)
// escape leading space
if strings.HasPrefix(attributeValueString, " ") {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all much clearer now - good job 👍

What happens if it's prefixed by two spaces though? I agree it's an edge case, I'd be inclined to to fix my cert rather than the code.

Copy link
Author

@csucu csucu Nov 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently it just escapes the last space, and the rest are left. The rfc isn't very clear on what todo with more then one trailing space. Should i be escaping all of them?

Copy link

@tadukurow tadukurow Nov 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The escaping of leading and trailing spaces, is to prevent trimming. By escaping the leading and trailing space, the trimming is no longer an issue, regardless of how many spaces it has after/before that.

Copy link

@tadukurow tadukurow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Just a small update to check for leading '#' and not escape everywhere and it should be good.

@domodwyer domodwyer merged commit 663dfe5 into development Nov 2, 2017
@domodwyer domodwyer deleted the feature/rdnconstruction branch November 2, 2017 14:52
@domodwyer domodwyer mentioned this pull request Jan 15, 2018
domodwyer added a commit that referenced this pull request Jan 15, 2018
Includes:

* Reduced memory in bulk operations (#56)
* Native x509 authentication (#55)
* Better connection recovery (#69)
* Example usage (#75 and #78)

Thanks to:

* @bachue
* @csucu 
* @feliixx


---
[Throughput overview](https://user-images.githubusercontent.com/9275968/34954403-3d3253dc-fa18-11e7-8eef-0f2b0f21edc3.png)

Select throughput has increased by ~600 requests/second with slightly increased variance:
```
x => r2017.11.06-select-zipfian-throughput.log 
y => 9acbd68-select-zipfian-throughput.log 

     n   min   max median  average   stddev      p99
x 3600 49246 71368  66542 66517.26 2327.675 70927.01
y 3600 53304 72005  67151 67145.36 2448.534 71630.00

          62000       64000       66000        68000       70000       72000    
 |----------+-----------+-----------+------------+-----------+-----------+-----|
                              +---------+--------+                              
1          -------------------|         |        |--------------------          
                              +---------+--------+                              
                                 +---------+---------+                          
2    ----------------------------|         |         |--------------------      
                                 +---------+---------+                          
Legend: 1=data$x, 2=data$y

At 95% probablitiy:
===> average is statistically significant     (p=0.000000, diff ~628.094444)
===> variance is statistically significant    (p=0.002398)

```

* [insert-latency.txt](https://github.com/globalsign/mgo/files/1632474/insert-latency.txt)
* [insert-throughput.txt](https://github.com/globalsign/mgo/files/1632475/insert-throughput.txt)
* [select-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632476/select-zipfian-latency.txt)
* [select-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632477/select-zipfian-throughput.txt)
* [update-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632478/update-zipfian-latency.txt)
* [update-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632479/update-zipfian-throughput.txt)

Note: latencies are approximations calculated from grouped data
libi pushed a commit to libi/mgo that referenced this pull request Dec 1, 2022
Includes:

* Reduced memory in bulk operations (globalsign#56)
* Native x509 authentication (globalsign#55)
* Better connection recovery (globalsign#69)
* Example usage (globalsign#75 and globalsign#78)

Thanks to:

* @bachue
* @csucu 
* @feliixx


---
[Throughput overview](https://user-images.githubusercontent.com/9275968/34954403-3d3253dc-fa18-11e7-8eef-0f2b0f21edc3.png)

Select throughput has increased by ~600 requests/second with slightly increased variance:
```
x => r2017.11.06-select-zipfian-throughput.log 
y => 9acbd68-select-zipfian-throughput.log 

     n   min   max median  average   stddev      p99
x 3600 49246 71368  66542 66517.26 2327.675 70927.01
y 3600 53304 72005  67151 67145.36 2448.534 71630.00

          62000       64000       66000        68000       70000       72000    
 |----------+-----------+-----------+------------+-----------+-----------+-----|
                              +---------+--------+                              
1          -------------------|         |        |--------------------          
                              +---------+--------+                              
                                 +---------+---------+                          
2    ----------------------------|         |         |--------------------      
                                 +---------+---------+                          
Legend: 1=data$x, 2=data$y

At 95% probablitiy:
===> average is statistically significant     (p=0.000000, diff ~628.094444)
===> variance is statistically significant    (p=0.002398)

```

* [insert-latency.txt](https://github.com/globalsign/mgo/files/1632474/insert-latency.txt)
* [insert-throughput.txt](https://github.com/globalsign/mgo/files/1632475/insert-throughput.txt)
* [select-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632476/select-zipfian-latency.txt)
* [select-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632477/select-zipfian-throughput.txt)
* [update-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632478/update-zipfian-latency.txt)
* [update-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632479/update-zipfian-throughput.txt)

Note: latencies are approximations calculated from grouped data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants