[Spanner] Running spanner emulator causes UNAUTHENTICATED error

[taka-oyama]

When I try to connect to the Spanner emulator, I get the following error.

Fatal error: Uncaught Google\Cloud\Core\Exception\ServiceException: {
    "message": "Established channel does not have a sufficient security level to transfer call credential.",
    "code": 16,
    "status": "UNAUTHENTICATED",
    "details": []
} in /app/vendor/google/cloud-core/src/GrpcRequestWrapper.php:257
Stack trace:
#0 /app/vendor/google/cloud-core/src/GrpcRequestWrapper.php(146): Google\Cloud\Core\GrpcRequestWrapper->convertToGoogleException(Object(Google\ApiCore\ApiException))
#1 /app/vendor/google/cloud-core/src/GrpcTrait.php(79): Google\Cloud\Core\GrpcRequestWrapper->send(Array, Array, Array)
#2 /app/vendor/google/cloud-spanner/src/Connection/Grpc.php(301): Google\Cloud\Spanner\Connection\Grpc->send(Array, Array)
#3 /app/vendor/google/cloud-spanner/src/Instance.php(264): Google\Cloud\Spanner\Connection\Grpc->getInstance(Array)
#4 /app/vendor/google/cloud-spanner/src/Instance.php(230): Google\Cloud\Spanner\Instance->reload(Array)
#5 /app/test.php(11): Google\Cloud\Spanner\Instance->exists()
#6 {main}
  thrown in /app/vendor/google/cloud-core/src/GrpcRequestWrapper.php on line 257

This only happens with grpc v1.36.
Grpc v1.35 does not have this issue.

Environment details

• OS: Debian
• PHP version: 7.4
• Package name and version: grpc v1.36

Steps to reproduce

I've created a simple sample code to reproduce the error below.

https://github.com/taka-oyama/grpc-bug-test

[skuruppu]

@AVaksman would you or someone in your team be able to please take a look?

[gsouf]

Not an isolated case. Same issue here after we upgraded our ubuntu containers

[gsouf]

@taka-oyama we fixed by downgrading grpc to version 1.35.0

sudo pecl uninstall grpc
sudo pecl install grpc-1.35.0

[taka-oyama]

I noticed cloud-spanner v1.36 included #3912.
I upgraded to 1.36 and retried the test but I'm still seeing the same error.
Was this suppose to fix the issue?
Is there something that needs to be done on the client side?

[skuruppu]

@taka-oyama unfortunately the latest Spanner release will not fix the problem. The underlying issue is in gRPC v1.36. @larkee filed an issue in the gRPC repo (grpc/grpc#25810). I pinged the issue just now as well.

[skuruppu]

@AVaksman this issue is meant to be fixed against gRPC v1.37. Could you please confirm by upgrading the version locally just to confirm that the customer can upgrade their gRPC version to get the fix?

[gsouf]

@skuruppu I upgrade to version 1.37.0 of gRPC:

sudo pecl uninstall grpc
sudo pecl install grpc-1.37.0

pecl list
Installed packages, channel pecl.php.net:
=========================================
Package Version State
grpc    1.37.0  stable

However I still get the error mentioned in this ticket when running the emulator with PHP.

[skuruppu]

@gsouf I'm so sorry. I didn't type in the minor version number in my comment earlier :(

Would you please be able to try installing grpc-1.37.1?

[gsouf]

@skuruppu it seems to work well with version 1.37.1 thanks 👍🏽

[skuruppu]

@taka-oyama would you also be able to confirm if the issue is fixed on your side if you update to grpc-1.37.1?

[taka-oyama]

@skuruppu I was able to confirm that it works with 1.37.1.
Thank you! I will close the issue.

[taka-oyama]

This is happening again with grpc v1.45.0.

I've reopened this issue to alert everyone here.

grpc/grpc#29233

[taka-oyama]

Is anyone looking into this?

[saranshdhingra]

Hi @taka-oyama
Thanks a lot for reporting this issue.
I will be looking into this issue.

Once I am able to replicate it, I'll update here.

[taka-oyama]

Hi, I think the sample code provided can easily replicate the error. Please have a look.

[taka-oyama]

Seems like they have some insight on what's going on.
grpc/grpc#29233 (comment)

[saranshdhingra]

Hi @taka-oyama
Thanks a lot for providing an easy setup for me to verify.

I have asked in the issue 29233 if I can help with the fix process.

Meanwhile, I tried your setup with grpc-1.37.1 and it still works.
Will it be possible for you to rollback to that version until a fix is released for the current version?

[taka-oyama]

Status update.

Waiting for #5224 to be released.

I have also confirmed that this was working with 1.44.0 as well so I am using that in the mean time.

[saranshdhingra]

Hi @taka-oyama
I was able to verify that this has been fixed with google/cloud-spanner:1.49 (and grpc: 1.45)

I am closing this, but if you find if it is not fixed, feel free to reopen this issue :)