Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(NODE-4696): add FaaS env information to client metadata #3626

Merged
merged 33 commits into from
Apr 12, 2023
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
c623356
FAAS metadata commit
baileympearson Apr 3, 2023
1b86b2d
fix lint
baileympearson Apr 3, 2023
ed44549
misc fixes
baileympearson Apr 3, 2023
da62ec4
remove truncated client metadata
baileympearson Apr 3, 2023
a98a423
chore: everything but truncation
baileympearson Apr 3, 2023
ef58d3f
chore: misc updates
baileympearson Apr 3, 2023
51ae18b
misc changes
baileympearson Apr 3, 2023
6853019
use `Int32`
baileympearson Apr 3, 2023
c81552a
Merge branch 'main' into NODE-4696-faas-metadata-handshake-5x
nbbeeken Apr 5, 2023
cc1bf22
refactor(NODE-4696): use an additive approach toward metadata limit
nbbeeken Apr 4, 2023
21dc9e3
test: env omission
nbbeeken Apr 5, 2023
b8108a7
fix
nbbeeken Apr 5, 2023
4be609f
test: make sure there's a failure with too large a document
nbbeeken Apr 5, 2023
70da1d6
fix: tests
nbbeeken Apr 5, 2023
871d208
fix: lb mode error
nbbeeken Apr 5, 2023
6bccf1e
test: os type omission, env omission, merge faas logic
nbbeeken Apr 5, 2023
01b9140
inheritdoc unsupported
nbbeeken Apr 6, 2023
757cc0b
perf: only serialize new element for fits check
nbbeeken Apr 7, 2023
8f28668
fix: vercel having aws env vars
nbbeeken Apr 7, 2023
beba14a
fix: gcp bool
nbbeeken Apr 7, 2023
d82591d
style: early return = no else
nbbeeken Apr 7, 2023
85a4551
chore: add comment about vercel
nbbeeken Apr 7, 2023
52000e2
Merge branch 'main' into NODE-4696-faas-metadata-handshake-5x-spec-up…
nbbeeken Apr 7, 2023
d20c333
fix: update error message
nbbeeken Apr 10, 2023
69c01d7
fix: rm vercel_url
nbbeeken Apr 10, 2023
5cc2efb
titles
nbbeeken Apr 11, 2023
6c82746
undo cmap change
nbbeeken Apr 11, 2023
83f5e6c
titles2
nbbeeken Apr 11, 2023
6c4d67d
lint
nbbeeken Apr 11, 2023
2c9b4c4
moved throwing tests
nbbeeken Apr 11, 2023
f7b1ea3
fix before/after inside loop
nbbeeken Apr 11, 2023
92c0a18
Merge branch 'main' into NODE-4696-faas-metadata-handshake-5x-spec-up…
nbbeeken Apr 12, 2023
ec0372f
fix: test locations
nbbeeken Apr 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/cmap/connect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import {
MongoRuntimeError,
needsRetryableWriteLabel
} from '../error';
import { Callback, ClientMetadata, HostAddress, ns } from '../utils';
import { Callback, HostAddress, ns } from '../utils';
import { AuthContext, AuthProvider } from './auth/auth_provider';
import { GSSAPI } from './auth/gssapi';
import { MongoCR } from './auth/mongocr';
Expand All @@ -28,6 +28,7 @@ import { AuthMechanism } from './auth/providers';
import { ScramSHA1, ScramSHA256 } from './auth/scram';
import { X509 } from './auth/x509';
import { CommandOptions, Connection, ConnectionOptions, CryptoConnection } from './connection';
import type { ClientMetadata } from './handshake/client_metadata';
import {
MAX_SUPPORTED_SERVER_VERSION,
MAX_SUPPORTED_WIRE_VERSION,
Expand Down
2 changes: 1 addition & 1 deletion src/cmap/connection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ import { applySession, ClientSession, updateSessionFromResponse } from '../sessi
import {
calculateDurationInMs,
Callback,
ClientMetadata,
HostAddress,
maxWireVersion,
MongoDBNamespace,
Expand All @@ -46,6 +45,7 @@ import {
} from './command_monitoring_events';
import { BinMsg, Msg, Query, Response, WriteProtocolMessageType } from './commands';
import type { Stream } from './connect';
import type { ClientMetadata } from './handshake/client_metadata';
import { MessageStream, OperationDescription } from './message_stream';
import { StreamDescription, StreamDescriptionOptions } from './stream_description';
import { getReadPreference, isSharded } from './wire_protocol/shared';
Expand Down
236 changes: 236 additions & 0 deletions src/cmap/handshake/client_metadata.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,236 @@
import * as os from 'os';
import * as process from 'process';

import { BSON, Int32 } from '../../bson';
import { MongoInvalidArgumentError } from '../../error';
import type { MongoOptions } from '../../mongo_client';

// eslint-disable-next-line @typescript-eslint/no-var-requires
const NODE_DRIVER_VERSION = require('../../../package.json').version;

/**
* @public
* @see https://github.com/mongodb/specifications/blob/master/source/mongodb-handshake/handshake.rst#hello-command
*/
export interface ClientMetadata {
driver: {
name: string;
version: string;
};
os: {
type: string;
name?: NodeJS.Platform;
architecture?: string;
version?: string;
nbbeeken marked this conversation as resolved.
Show resolved Hide resolved
};
platform: string;
application?: {
name: string;
};
/** FaaS environment information */
env?: {
name: 'aws.lambda' | 'gcp.func' | 'azure.func' | 'vercel';
timeout_sec?: Int32;
memory_mb?: Int32;
region?: string;
url?: string;
};
}

/** @public */
export interface ClientMetadataOptions {
driverInfo?: {
name?: string;
version?: string;
platform?: string;
};
appName?: string;
}

/** @internal */
export class LimitedSizeDocument {
private document = new Map();
/** BSON overhead: Int32 + Null byte */
private documentSize = 5;
constructor(private maxSize: number) {}

/** Only adds key/value if the bsonByteLength is less than MAX_SIZE */
public ifItFitsItSits(key: string, value: Record<string, any> | string): boolean {
// The BSON byteLength of the new element is the same as serializing it to its own document
// subtracting the document size int32 and the null terminator.
const newElementSize = BSON.serialize(new Map().set(key, value)).byteLength - 5;

if (newElementSize + this.documentSize > this.maxSize) {
return false;
}

this.documentSize += newElementSize;

this.document.set(key, value);

return true;
}

toObject(): ClientMetadata {
return BSON.deserialize(BSON.serialize(this.document), {
promoteLongs: false,
promoteBuffers: false,
promoteValues: false,
useBigInt64: false
}) as ClientMetadata;
}
}

type MakeClientMetadataOptions = Pick<MongoOptions, 'appName' | 'driverInfo'>;
/**
* From the specs:
* Implementors SHOULD cumulatively update fields in the following order until the document is under the size limit:
* 1. Omit fields from `env` except `env.name`.
* 2. Omit fields from `os` except `os.type`.
* 3. Omit the `env` document entirely.
* 4. Truncate `platform`. -- special we do not truncate this field
*/
export function makeClientMetadata(options: MakeClientMetadataOptions): ClientMetadata {
const metadataDocument = new LimitedSizeDocument(512);

const { appName = '' } = options;
// Add app name first, it must be sent
if (appName.length > 0) {
const name =
Buffer.byteLength(appName, 'utf8') <= 128
? options.appName
: Buffer.from(appName, 'utf8').subarray(0, 128).toString('utf8');
metadataDocument.ifItFitsItSits('application', { name });
}

const { name = '', version = '', platform = '' } = options.driverInfo;

const driverInfo = {
name: name.length > 0 ? `nodejs|${name}` : 'nodejs',
version: version.length > 0 ? `${NODE_DRIVER_VERSION}|${version}` : NODE_DRIVER_VERSION
};

if (!metadataDocument.ifItFitsItSits('driver', driverInfo)) {
throw new MongoInvalidArgumentError(
'Unable to include driverInfo name and version, metadata cannot exceed 512 bytes'
);
}

const platformInfo =
platform.length > 0
? `Node.js ${process.version}, ${os.endianness()}|${platform}`
: `Node.js ${process.version}, ${os.endianness()}`;

if (!metadataDocument.ifItFitsItSits('platform', platformInfo)) {
throw new MongoInvalidArgumentError(
'Unable to include driverInfo platform, metadata cannot exceed 512 bytes'
);
}

// Note: order matters, os.type is last so it will be removed last if we're at maxSize
const osInfo = new Map()
.set('name', process.platform)
.set('architecture', process.arch)
.set('version', os.release())
.set('type', os.type());

if (!metadataDocument.ifItFitsItSits('os', osInfo)) {
for (const key of osInfo.keys()) {
osInfo.delete(key);
if (osInfo.size === 0) break;
if (metadataDocument.ifItFitsItSits('os', osInfo)) break;
}
}

const faasEnv = getFAASEnv();
if (faasEnv != null) {
if (!metadataDocument.ifItFitsItSits('env', faasEnv)) {
for (const key of faasEnv.keys()) {
faasEnv.delete(key);
if (faasEnv.size === 0) break;
if (metadataDocument.ifItFitsItSits('env', faasEnv)) break;
}
}
}

return metadataDocument.toObject();
}

/**
* Collects FaaS metadata.
* - `name` MUST be the last key in the Map returned.
*/
export function getFAASEnv(): Map<string, string | Int32> | null {
const {
AWS_EXECUTION_ENV = '',
AWS_LAMBDA_RUNTIME_API = '',
FUNCTIONS_WORKER_RUNTIME = '',
K_SERVICE = '',
FUNCTION_NAME = '',
VERCEL = '',
AWS_LAMBDA_FUNCTION_MEMORY_SIZE = '',
AWS_REGION = '',
FUNCTION_MEMORY_MB = '',
FUNCTION_REGION = '',
FUNCTION_TIMEOUT_SEC = '',
VERCEL_REGION = ''
} = process.env;
nbbeeken marked this conversation as resolved.
Show resolved Hide resolved

const isAWSFaaS = AWS_EXECUTION_ENV.length > 0 || AWS_LAMBDA_RUNTIME_API.length > 0;
const isAzureFaaS = FUNCTIONS_WORKER_RUNTIME.length > 0;
const isGCPFaaS = K_SERVICE.length > 0 || FUNCTION_NAME.length > 0;
const isVercelFaaS = VERCEL.length > 0;

// Note: order matters, name must always be the last key
nbbeeken marked this conversation as resolved.
Show resolved Hide resolved
const faasEnv = new Map();

// When isVercelFaaS is true so is isAWSFaaS; Vercel inherits the AWS env
if (isVercelFaaS && !(isAzureFaaS || isGCPFaaS)) {
if (VERCEL_REGION.length > 0) {
faasEnv.set('region', VERCEL_REGION);
}

faasEnv.set('name', 'vercel');
return faasEnv;
}

if (isAWSFaaS && !(isAzureFaaS || isGCPFaaS || isVercelFaaS)) {
durran marked this conversation as resolved.
Show resolved Hide resolved
if (AWS_REGION.length > 0) {
faasEnv.set('region', AWS_REGION);
}

if (
AWS_LAMBDA_FUNCTION_MEMORY_SIZE.length > 0 &&
Number.isInteger(+AWS_LAMBDA_FUNCTION_MEMORY_SIZE)
) {
faasEnv.set('memory_mb', new Int32(AWS_LAMBDA_FUNCTION_MEMORY_SIZE));
}

faasEnv.set('name', 'aws.lambda');
return faasEnv;
}

if (isAzureFaaS && !(isGCPFaaS || isAWSFaaS || isVercelFaaS)) {
faasEnv.set('name', 'azure.func');
return faasEnv;
}

if (isGCPFaaS && !(isAzureFaaS || isAWSFaaS || isVercelFaaS)) {
if (FUNCTION_REGION.length > 0) {
faasEnv.set('region', FUNCTION_REGION);
}

if (FUNCTION_MEMORY_MB.length > 0 && Number.isInteger(+FUNCTION_MEMORY_MB)) {
faasEnv.set('memory_mb', new Int32(FUNCTION_MEMORY_MB));
}

if (FUNCTION_TIMEOUT_SEC.length > 0 && Number.isInteger(+FUNCTION_TIMEOUT_SEC)) {
faasEnv.set('timeout_sec', new Int32(FUNCTION_TIMEOUT_SEC));
}

faasEnv.set('name', 'gcp.func');
return faasEnv;
}

return null;
nbbeeken marked this conversation as resolved.
Show resolved Hide resolved
}
2 changes: 1 addition & 1 deletion src/connection_string.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { URLSearchParams } from 'url';
import type { Document } from './bson';
import { MongoCredentials } from './cmap/auth/mongo_credentials';
import { AUTH_MECHS_AUTH_SRC_EXTERNAL, AuthMechanism } from './cmap/auth/providers';
import { makeClientMetadata } from './cmap/handshake/client_metadata';
import { Compressor, CompressorName } from './cmap/wire_protocol/compression';
import { Encrypter } from './encrypter';
import {
Expand All @@ -32,7 +33,6 @@ import {
emitWarningOnce,
HostAddress,
isRecord,
makeClientMetadata,
parseInteger,
setDifference
} from './utils';
Expand Down
3 changes: 1 addition & 2 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ export type {
WaitQueueMember,
WithConnectionCallback
} from './cmap/connection_pool';
export type { ClientMetadata, ClientMetadataOptions } from './cmap/handshake/client_metadata';
export type {
MessageStream,
MessageStreamOptions,
Expand Down Expand Up @@ -463,8 +464,6 @@ export type { Transaction, TransactionOptions, TxnState } from './transactions';
export type {
BufferPool,
Callback,
ClientMetadata,
ClientMetadataOptions,
EventEmitterWithState,
HostAddress,
List,
Expand Down
22 changes: 20 additions & 2 deletions src/mongo_client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import type { AuthMechanismProperties, MongoCredentials } from './cmap/auth/mong
import type { AuthMechanism } from './cmap/auth/providers';
import type { LEGAL_TCP_SOCKET_OPTIONS, LEGAL_TLS_SOCKET_OPTIONS } from './cmap/connect';
import type { Connection } from './cmap/connection';
import type { ClientMetadata } from './cmap/handshake/client_metadata';
import type { CompressorName } from './cmap/wire_protocol/compression';
import { parseOptions, resolveSRVRecord } from './connection_string';
import { MONGO_CLIENT_EVENTS } from './constants';
Expand All @@ -24,7 +25,7 @@ import { readPreferenceServerSelector } from './sdam/server_selection';
import type { SrvPoller } from './sdam/srv_polling';
import { Topology, TopologyEvents } from './sdam/topology';
import { ClientSession, ClientSessionOptions, ServerSessionPool } from './sessions';
import { ClientMetadata, HostAddress, MongoDBNamespace, ns, resolveOptions } from './utils';
import { HostAddress, MongoDBNamespace, ns, resolveOptions } from './utils';
import type { W, WriteConcern, WriteConcernSettings } from './write_concern';

/** @public */
Expand Down Expand Up @@ -363,6 +364,7 @@ export class MongoClient extends TypedEventEmitter<MongoClientEvents> {
};
}

/** @see MongoOptions */
get options(): Readonly<MongoOptions> {
return Object.freeze({ ...this[kOptions] });
}
Expand Down Expand Up @@ -660,7 +662,22 @@ export class MongoClient extends TypedEventEmitter<MongoClientEvents> {
}

/**
* Mongo Client Options
* Parsed Mongo Client Options.
*
* User supplied options are documented by `MongoClientOptions`.
*
* **NOTE:** The client's options parsing is subject to change to support new features.
* This type is provided to aid with inspection of options after parsing, it should not be relied upon programmatically.
*
* Options are sourced from:
* - connection string
* - options object passed to the MongoClient constructor
* - file system (ex. tls settings)
* - environment variables
* - DNS SRV records and TXT records
*
* Not all options may be present after client construction as some are obtained from asynchronous operations.
*
* @public
*/
export interface MongoOptions
Expand Down Expand Up @@ -717,6 +734,7 @@ export interface MongoOptions
proxyPort?: number;
proxyUsername?: string;
proxyPassword?: string;

/** @internal */
connectionType?: typeof Connection;

Expand Down
4 changes: 1 addition & 3 deletions src/sdam/topology.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import type { BSONSerializeOptions, Document } from '../bson';
import type { MongoCredentials } from '../cmap/auth/mongo_credentials';
import type { ConnectionEvents, DestroyOptions } from '../cmap/connection';
import type { CloseOptions, ConnectionPoolEvents } from '../cmap/connection_pool';
import type { ClientMetadata } from '../cmap/handshake/client_metadata';
import { DEFAULT_OPTIONS, FEATURE_FLAGS } from '../connection_string';
import {
CLOSE,
Expand Down Expand Up @@ -37,7 +38,6 @@ import type { ClientSession } from '../sessions';
import type { Transaction } from '../transactions';
import {
Callback,
ClientMetadata,
EventEmitterWithState,
HostAddress,
List,
Expand Down Expand Up @@ -138,15 +138,13 @@ export interface TopologyOptions extends BSONSerializeOptions, ServerOptions {
/** The name of the replica set to connect to */
replicaSet?: string;
srvHost?: string;
/** @internal */
srvPoller?: SrvPoller;
/** Indicates that a client should directly connect to a node without attempting to discover its topology type */
directConnection: boolean;
loadBalanced: boolean;
metadata: ClientMetadata;
/** MongoDB server API version */
serverApi?: ServerApi;
/** @internal */
[featureFlag: symbol]: any;
}

Expand Down
Loading