Cant run in-process server in rust as per axum example

[danielblignaut]

Bug Report

I am running a Rust axum server and trying to launch an in-process perspective server in Rust (as per https://github.com/finos/perspective/blob/master/examples/rust-axum/src/main.rs )

The perspective crate in rust is the perspective-js crate, not the rust/perspective crate ( https://crates.io/crates/perspective ) so we have to point to the git version of the rust library, therefore we do the following below:

Steps to Reproduce:

1. have an existing rust workspace project
2. git clone https://github.com/finos/perspective
3. cd perspective
4. git checkout tkp/fixes
5. brew install cmake maturin
6. python -m venv perspective_venv
7. source perspective_venv/bin/activate
8. pip install jupyterlab
9. pnpm install
10. pnpm run build
11. in the root rust workspace project, edit Cargo.toml: perspective = { path = "./../perspective/rust/perspective" }
12. run cargo build

Expected Result:

Expect to be able to use the perspective rust library to start an in-process perspective server from Rust

Actual Result:

Various Failures:

CMake Error at /Users/danielblignaut/Documents/Development/trading/perspective/rust/perspective-server/cmake/modules/FindInstallDependency.cmake:42 (configure_file):
Not a directory
Call Stack (most recent call first):
CMakeLists.txt:401 (psp_build_dep)

CMake Error at /Users/danielblignaut/Documents/Development/trading/perspective/rust/perspective-server/cmake/modules/FindInstallDependency.cmake:43 (add_subdirectory):
add_subdirectory given source
"/Users/danielblignaut/Documents/Development/trading/rust/target/debug/build/perspective-server-2350c75ef25453d6/out/build/arrow-src/cpp/"
which is not an existing directory.
Call Stack (most recent call first):
CMakeLists.txt:401 (psp_build_dep)

CMake Error at /Users/danielblignaut/Documents/Development/trading/perspective/rust/perspective-server/cmake/modules/FindInstallDependency.cmake:82 (target_compile_options):
Cannot specify compile options for target "arrow" which is not built by
this project.
Call Stack (most recent call first):
CMakeLists.txt:401 (psp_build_dep)

[danielblignaut]

I’m not experienced with make but it seems like the CMAKE_BIN_DIR may change when running cargo build from my project vs. pnpm run build in the perspective project root. When I run pnpm run build, the perspective project compiles successfully, when I run cargo build in my project, I get the above error

[texodus]

@danielblignaut appreciate your interest in the (new!) Rust library. You should be able to start the example rust server like so (it will compile the library itself):

pnpm run start rust-axum

We have not officially released the crate yet, so it should be considered incomplete [though truthfully, the only PRs open in this milestone are for sibling packages). There is an older patch available on crates.io @ 3.0.0-alpha.1 but it is very different from master branch. Developer instructions for installing this haven't been written yet either, so if you want to build from source your best bet is to follow the steps in our CI script if you have failing dependencies or other mystery errors. We will publish the release crates soon, but I can try to cut an 3.0.0-alpha.2 release before then.

Some other suggestions to get you started:

4. git checkout tkp/fixes

Perspective uses the master branch for development, and you are building a branch on top of that which does not compile on CI. If you are trying to test Perspective's rust library, you are better off on a master itself at least it has passed CI.

5. brew install cmake maturin

maturin is part of the perspective-python requirements.txt, which you should use to install this dep.

When I run pnpm run build ...

pnpm run build is the command for building a sub-project, but there is no build step yet for the Rust library in the build tool. Instead, running the example will compile the perspective crate downstream as part of its own dependencies for now, which you can invoke as above with pnpm run start rust-axum.

The old docs describe the similar workflow from the previous release, though there was no Rust library in the last release and you'll have to mentally replace yarn with pnpm run until we release.

[danielblignaut]

hey @texodus,

Thanks for the feedback. I've got the rust websocket server running locally now and can establish websocket connections to it however the first message the browser tries to send not a buffer, it tries to send a JSON payload:

{"id":-1,"cmd":"init"}

which the rust server doesn't like:

https://github.com/finos/perspective/blob/master/examples/rust-axum/src/main.rs#L86

This seems to be the case even when running the index.html file in the rust example - is there something I am missing or prospectively a bug that needs to be handled in the rust server to manage this message type in the above line reference?

[danielblignaut]

it's as if it's passing the JSON web worker messages over the websocket

[danielblignaut]

it seems like there's a change in the websocket messaging protocol between the latest npm packages and master branch, I'm rebuilding master and going to try use those locally to see if that resolves the issue

[texodus]

You are using JavaScript 2.10.1 against Rust master (e.g. ~3.0-alpha.2). It sounds like you've somehow installed these dependencies from npm, likely by running some variant of npm install in the examples directory itself, instead of from the project root.

These packages will be linked correctly if you've successfully built the project from the source, but you probably have errant node_modules directories in your tree somewhere (where ever you ran the command). Remove them and make sure you've built the entire project locally with pnpm install && pnpm run build at the project root only.

[danielblignaut]

yes, I stupidly thought for the npm packages I could use the last published build instead of going down yarn / pnpm workspace fun. Should've picked up on it earlier.

[danielblignaut]

FYI, I'll open an issue but there's a bug in the latest webpack plugin package on master. https://github.com/finos/perspective/blame/fae909221e09a88ca347ff8bb935f3d8fa90692e/packages/perspective-webpack-plugin/index.js#L57 throws an error as the preceding lines adding the first rule to the array was removed in a previous commit:

rules.push({
            test: /perspective\.worker\.js$/,
            type: "javascript/auto",
            include: this.options.workerPath,
            use: {
                loader: require.resolve("worker-loader"),
                options: {
                    filename: this.options.workerName,
                },
            },
        });