Migrate to ES Module structure/runtime - Node 20, Vite, Vitest, Storybook 8

[twelch]

Dev

Key Requirements/Decisions

• Switch from Jest to Vitest, supports ESM out of the box and is mostly Jest compatible API
• Switch from webpack to esbuild, supports ESM out of the box and more easily configurable.
• Switch from ts-node to tsx. Better node 20 and esm support out of the box.
• upgrade flatgeobuf, and upgrade the docker-gp-workspace to the latest GDAL
• request fix to geoblaze to work in ESM environment, v2.6.1 - https://github.com/GeoTIFF/geoblaze/commits/master/

ToDo

• Migrate packages/geoprocessing
• Swap __dirname and __filename to use node:url
• Run test suite
• Update top-level tsconfig.templates.json
• Migrate other packages
• Experimental publish
• Test with Are the types right? https://arethetypeswrong.github.io/?p=%40seasketch%2Fgeoprocessing%406.1.3-experimental-esm.5
• Test geoprocessing init
• clean up import of CJS dependencies that don't have a true default import
• migrate use of require, enough to get init working
• flatgeobuf
  • figure out how to produce a flatgeobuf that js lib will read from disk. Why does countries.fgb work? But not world_new.fgb? Installing latest QGIS and regenerating flatgeobuf seems to have fixed. Also fixing node-fetch polyfill seems to have fixed.
  • Add tests to flatgeobuf.test.e2e.ts
• VectorDatasource not fetching properly. Fixing node-fetch polyfill seems to have fixed it
• Upgrade to node 20.11.1 and use new import.meta.dirname
• esbuild
  • Test out esbuild for packaging gp functions - https://www.chrisarmstrong.dev/posts/package-aws-lambda-nodejs-functions-individually-with-esbuild-for-faster-cold-start
  • Switch to dynamic import approach to loading bundled report clients
• Add support for deploying Node 20 with ESModule code to AWS using ESM
  • Example project - data import, precalc, build, deploy, publish
• Translation
  • Fix translation:install for when running init in a monorepo (can't find i18n directory because is at top-level of monorepo). build a more robust script instead of an inline command in package.json
• Vite storybook
  • https://github.com/vbenjs/vite-plugin-html?tab=readme-ov-file
  • https://vitejs.dev/guide/api-javascript
  • Use of vite config and babel config - Babel configuration storybookjs/builder-vite#286 (comment)
  • Get running in project directly (build and dev server)
  • Then switch to gp space with Vite config in GP space.
  • Then switch to running programmatically from GP space if needed/possible
  • Add back storybook to build-docs
• Vite build
  • replace esbuild with Vite. This must allow build to be done from GP_ROOT with entry point over in PROJECT_ROOT. esbuild doesn't currently allow this, specifically the html plugin
• Update styled-components to 6.x and use {styled} named import to be able to access class properties.
• Upgrade slugify
• Upgrade turf to 7.0 prerelease (wasn't importing types properly for 6.5)
• Switch to dynamic import approach for loading translations in client
• Update start:client
• Upgrade zx
• Upgrade to latest Typescript
• Update dependencies that are non-breaking
• Make sure territorial waters image loads, currently still has require but esbuild seems to replace

Testing

• Add flatgeobuf.ts test using local file
• Verify devcontainer is working with unstable release of docker-gp-workspace

Release

• Init a new project from scratch and go through to deploy - create sketch and collection, import data, test, deploy, publish data.
• Migrate an existing gp project
• Write migration doc
  • Add "type": "module" to package.json
  • Update tsconfig.json
  • Restart vscode for it to start recognizing use of esm mode
  • update package.json scripts, dependences, devDependencies. Can copy and overwrite directly from [insert link to release
  • install:scripts, then translation:install to upgrade
    file] and add back any that you added just for the project, if any.
• Update docs
  • ANALYZE and MINIFY vars for build command
  • Note use of nvm alias default v20.12.1 and nvm use default in vscode terminal if your vscode terminal keeps not using the right version of node
  • devcontainer upgrade, to get access to node20, latest GDAL, etc.
  • writeup how CLI commands work. They run in gp space using gp dependencies, but run against the project space. Build commands additionally bundle the report and gp/pp functions first in project space?
  • Use of unstable devcontainer and how to pull the latest docker pull seasketch/geoprocessing-workspace:unstable
  • named export required for report component in order for example-reports to work
  • Document static asset handling, and importing images - https://vitejs.dev/guide/assets.html#static-asset-handling

Follow-on tickets:

• Basic smoke test fails for ecoregions, FAIL src/functions/simpleFunctionSmoke.test.ts > Basic smoke tests > simpleFunction

  • tests run against all examples
  • FetchError: request to https://gp-global-datasources-datasets.s3.us-west-1.amazonaws.com/meow-ecos.fgb failed, reason: socket hang up
  • error happens with FSM region, but not Azores, something with the data there? Or does it result in many or very large polygons?

• fix snapshot update via test:scripts:update, need to migrate away from jest-based solution

• Do sideEffects need to be added back for fetchPolyfill?

• Vitest - leftovers from Switch from Jest to Vitest #273

  • Migrate to new aws-cdk/assertion library, assert lib has reached EOL, and also how we use it relies on Jest.
  • GeoprocessingHandler.test.ts has old jest mock
  • Setup vitest with at_testing-library/jest-dom -https://github.com/testing-library/jest-dom?tab=readme-ov-file#installation via Export Vitest extend-expect testing-library/jest-dom#439 (comment)
  • browser tests (fix mocking)
  • e2e tests (fix mocking)
  • Fix all skipped tests
  • Fix vitest globals so explicit import not needed
  • Switch to Vitest workspace - https://vitest.dev/guide/workspace.html

• Once migrate to aws-sdk v3, bring back the dynamodb tests - https://daaru.medium.com/testing-lambda-functions-with-vitest-fc713caa975d

• Once migrate to aws-sdk v3, filter it and aws-cdk out of the function builds using externals

• upgrade Typescript and switch from assert type JSON to with

• Update base translations

• Merge client-ui and client-core

• Reduce project dependencies by bringing CLI commands into GP space (via cd GP_PATH) so that GP dependencies are used. translation:extract is one example where babel and plugin should be able to be moved.

• Try to use top-level turf library

• Consolidate src and scripts into one tsc build step (assuming pure esm)

• Enable eslint - with warnings

• Add monorepolint like turf uses for template packages

• Convert "G extends Geometry" to "G extends Geometry | null" to support null geometry case?

• Update React

• Update i18next

• Migrate from polygon-clipping back to turf clip if possible
  v7 release:

• Publish new base EN translations and translate them to all languages

Old - will not do

• Webpack
  • Migrate client-loader.js to esm export, and to generate ESM report loader code
  • Migrate webpack.client.config.js to ESM - see https://github.com/firebase007/webpack-demo/blob/master/webpack.config.js and article https://blog.logrocket.com/transpile-es-modules-with-webpack-node-js/.
  • Migrate webpack.functions.config.js to ESM
  • Upgrade to webpack 5 and stop the use of NODE_OPTIONS=--openssl-legacy-provider
  • Convert App.tsx require to dynamic import, see (https://stackoverflow.com/questions/50976936/how-to-get-dynamic-imports-to-work-in-webpack-4)
  • https://stackoverflow.com/questions/64796952/webpack-ts-loader-import-with-js-extension-not-resolving
  • https://stackoverflow.com/questions/51356071/how-to-pass-parameters-from-web-pack-to-code

Questions

• What does the top-level tsconfig.json do? Should it be updated or deleted?
  • Doesn't seem to be used so deleted it. See if it breaks anything.
• Should I use "module": "node16" like sindresorhus suggests instead of "module": "nodenext" like Typescript suggests?
  • Switched to nodenext, to get json assert support, which new project init was throwing an error on for import of all json files, even though node16. 🤷
  • node16 should work best with everything in toolchain, and it will be stable/fixed. nodenext can/will change when Typescript is upgraded.
• Is "target": "es2022" for both src and scripts correct?
  • Yes, assuming webpack can handle it at the project build stage. This should ensure that the most modern esm code is generated by the typescript transpiler.
• Should I have esModuleInterop and allowSyntheticDefaultImports enabled?
  • Let's leave them enabled if they already are for now.
• What to replace require.resolve with in CJS files?
  • require.resolve returns a file path to a code files. There is an esm equivalent - https://stackoverflow.com/questions/54977743/do-require-resolve-for-es-modules
  • https://dev.to/stephencweiss/what-is-require-resolve-and-how-does-it-work-1ho4
  • area.test.ts turf area output changed by about .25%. Why?
    • @turf/area Fix earth radius to use WGS84 median earth radius (#2166)

Notes on steps

• nvm install 20.11.1
• npx lern init
• Add packages/geoprocessing
• Copy old package.json and modify for esm and type import
  • Add type: "module" tell node that .ts and .js files are ESM.
  • Add type import to exports - https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-7.html
• Copy over and update tsconfig.json
  • https://github.com/tsconfig/bases/blob/main/bases/node20.json
  • node16 moduleResolution provides support for ES Modules
• Update Typescript
  • You must use a .js extension in relative imports even though you're importing .ts files.
• ts-node fails with node v20 to run .ts files
  • TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for ...
  • ERR_UNKNOWN_FILE_EXTENSION on Node v20.0.0 TypeStrong/ts-node#1997
  • Switch to using tsx, works out of box
• Jest - not working properly, cant find tests...
  • Possibly has to do with .js file extension - ES import of typescript with .js extension fails swc-project/jest#64
  • Trying out vitest as alternative
  • https://blog.hao.dev/setting-up-vitest-within-sveltekit-repo
  • https://vitest.dev/guide/#configuring-vitest
  • Testing out using Node test runner for geoblaze tests which works. Potential to run node tests separately, see How to run the Node built-in testrunner for TypeScript files inside a specific directory? nodejs/help#3902, and using glob command to run all source files through tsx to node test runner "test:node": "glob -c \"node --import tsx --test\" \"./src/**/*.node.test.ts\"",
  • Summary - Jest, not finding tests and likely issues with mocking.
• Webpack
  • Had to add use of NODE_OPTIONS=--openssl-legacy-provider. Webpack 4 said to not fix this.
  • Webpack CLI may not support esm? https://stackoverflow.com/questions/64944796/webpack-5-and-esm
  • https://github.com/firebase007/webpack-demo/blob/master/webpack.config.js
  • https://stackoverflow.com/questions/50976936/how-to-get-dynamic-imports-to-work-in-webpack-4
• ESBuild
  • esm web app part 1 - https://how-to.dev/how-to-start-building-a-project-with-esbuild, part 2 https://how-to.dev/how-to-set-up-a-dev-server-with-esbuild
  • How to dynamically import reports from App.tsx? https://how-to.dev/how-to-lazy-load-with-esbuild
  • esbuild + ESM + NodeJS + Typescript troubles evanw/esbuild#3637 - esbuild + ESM + NodeJS + Typescript
  • Just discards the TS type annotations, still need to run tsc -noEmit in parallel with esbuild to check types
  • Add target of last 2 version or similar
  • Address dynamic import of language locale
  • Address Node Worker loading, no first-class support yet
    • WebWorker support? evanw/esbuild#312
    • threads.js alternatives - Typescript workers don't work with ESM andywer/threads.js#434
  • https://gist.github.com/manzt/689e4937f5ae998c56af72efc9217ef0
• AWS
  • https://aws.amazon.com/blogs/compute/using-node-js-es-modules-and-top-level-await-in-aws-lambda/
  • How to get esm enabled for gp functions? Need to use .mjs extension? Need to switch to nodejs lambda with esbuild, that perhaps looks at package.json for type module and does the right thing?
  • https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda_nodejs-readme.html
  • https://aws.amazon.com/blogs/compute/using-node-js-es-modules-and-top-level-await-in-aws-lambda/
  • Current lambda construct, via use of runtime and code property - https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda-readme.html
  • CDK use of ESM - (aws-lambda-nodejs): .mjs file extension and import.meta not supported aws/aws-cdk#13274
  • Rewrite require() calls of Node built-ins to import statements when emitting ESM for Node evanw/esbuild#2067 (comment)
  • Rewrite require() calls of Node built-ins to import statements when emitting ESM for Node evanw/esbuild#2067
  • https://stackoverflow.com/questions/67361096/how-can-i-zip-node-lambda-dependencies-in-aws-cdk-stack
  • Solutions required
    • build a mini package for each bundled lambda handler function with a package.json (type: module, with main entry point to the handler function). This will enable ESM and ensure handler function is found.
    • Add some esbuild workaround for use of require (banner js). And include entire aws-sdk in the build because v2 is not included already in nodejs lambda. only v3. Need to filter out this dependency after upgrade to v3.
  • Storybook
    • https://storybook.js.org/docs/api/main-config-indexers
    • [RFC] Deprecate StoriesOf storybookjs/storybook#23177

[twelch]

Upgrade WSL Ubunto to 22.04 Jammy

• https://askubuntu.com/questions/1428423/upgrade-ubuntu-in-wsl2-from-20-04-to-22-04
• upgrade postgres packages from 12.x to 14.x then run the cluster upgrade
• upgrade GDAL