Port Scripting Engine to JavaScript

[freezy]

VP's scripting language of choice is VBScript. I think at some point, IE supported VBScript natively, but this was decades ago. So we need to find a way to run the table scripts in the browser without having to manually rewrite it.

Since browsers today all support JavaScript, the most obvious way would be to transcribe all VBScripts to JavaScript. In order to do that, we need to:

1. Write a grammar
2. Tokenize and parse the VBSes
3. Produce JSes
4. Implement (or stub) the linked libraries

A VBS grammar can be found here in the BNF format. Depending on which parser, the grammar needs to be ported. There are a few parsers in JavaScript:

• nearley.js
• Jison
• PEG.js

Once we have an AST of the script, JavaScript code needs to be emitted. astring is a library doing that, but the AST needs to be ESTree compliant.

Finally, many scripts load Windows DLLs and access their API directly via VBS. These need to be stubbed and implemented where functionality is needed (e.g. VPinMAME). Also, the VBScript's standard library probably needs to be at least partially implemented.

[freezy]

See also kastner/vbscript_in_js, which uses Jison, for inspiration.

[jsm174]

Finally spent some time looking at this.

Initially, I started with node-ebnf, using the VBS grammar from above. I recreated some of the character sets using information from here.

Unfortunately, node-ebnf doesn't seem to support left recursion and I couldn't figure out how to create nullable rules.

I switched to nearley, and have a sample project at:

https://github.com/jsm174/vbscript-parser

I think I have the grammar converted to nearley syntax. It needed to be shuffled around because according to here:

By default, nearley attempts to parse the first nonterminal defined in the grammar.

Currently the sample code is stuck in an infinite loop on a simple Dim statement.

I'll keep working on it.

[freezy]

Fantastic! I had a look at Nearley recently as well. What I've found out is that the grammar from rosettacode seems to be missing white spaces. A small test that adds them looks like that:

@builtin "whitespace.ne"

OptionExplicit      -> "Option" __ "Explicit" NL

NL                  -> NewLine NL
                     | NewLine

NewLine             -> CR LF
                     | CR
                     | LF
                     | ":"

LF                  -> [\x0A]

CR                  -> [\x0D]

The VBS to parse:

Option Explicit

This correctly results in the following results:

[ [ 'Option', null, 'Explicit', [ [ [ '\r' ], [ '\n' ] ] ] ],
  [ 'Option', null, 'Explicit', [ [ [ '\r' ] ], [ [ [ '\n' ] ] ] ] ] ]

You might be able to continue from there and add the remaining grammar incrementally. I would probably directly start adding the correct post processors that result in an ESTree (if that's even possible, haven't tested that).

In any case, thanks for taking a look at this!

[jsm174]

Ok. That helps out tremendously.

The rosettacode grammar does have whitespace rules, but none explicitly referenced in the statements.

The good thing is, what I had was not too far off from the builtin's:

// wschar -> [ \t\n\v\f] {% id %} 

vs:

// WS -> [\x09] | [\x0B] | [\x0C] | [\x20] | [\xA0]

Will need to research how goldparser was automatically working with spaces too.

Whitespace     -> WS:+
               | "_" WS:* CR:? LF:?

[freezy]

Yeah I had the same observation and didn't understand it either. I just remembered reading somewhere that in Nearley, whitespaces must be declared explicitly (which I find kinda obvious), but maybe other parsers like Goldparser do it implicitly. Or Whitespace is some kind of keyword that gets applied to the grammar automatically between terms.

[freezy]

In case you're wondering about the AST, I've found this pretty cool online tool. If we can get our AST into that, we're basically golden.

[freezy]

It's the latter (Whitespace is a keyword):

From the parser's point of view (in particular the Deterministic Finite Automata that it uses) these whitespace characters are recognized as a special terminal which can be discarded. In GOLD, this terminal is simply called the Whitespace terminal and can be defined to whatever is needed. If the Whitespace Terminal is not defined explicitly in the grammar, it will be implicitly declared as one or more of the characters in the pre-defined Whitespace set: {Whitespace}+.

[freezy]

Okay check this out. Given the grammar:

@builtin "whitespace.ne"

@{%
function program(data) {
	return {
	  type: "Program",
	  body: data,
	}
}

function varDecl(data) {
	return {
		type: 'VariableDeclaration',
		kind: 'let',
		declarations: [ data[2], ...data[3] ].map(variableDeclarator)
	}
}

function variableDeclarator(name) {
	return {
		type: "VariableDeclarator",
		id: { type: "Identifier", name },
	}
}
%}

Program              -> NLOpt GlobalStmt:*                           {% data => program(data[1]) %}

GlobalStmt           -> OptionExplicit
                      | BlockStmt                                    {% data => data[0] %}


OptionExplicit       -> "Option" __ "Explicit" NL

BlockStmt            -> VarDecl                                      {% data => data[0] %}

VarDecl              -> "Dim" __ VarName OtherVarsOpt:* NL           {% varDecl %}

VarName              -> ExtendedID ("(" ArrayRankList ")"):?         {% data => data[0] %}

OtherVarsOpt         -> "," __ VarName                               {% data => data[2] %}

ExtendedID           -> SafeKeywordID
                      | ID                                           {% data => data[0] %}

SafeKeywordID        -> "Default"
                      | "Erase"
                      | "Error"
                      | "Explicit"
                      | "Property"
                      | "Step"

ID                   -> Letter IDTail                                {% data => data[0] + data[1] %}
                      | "[" IDNameChar:* "]"

ArrayRankList        -> IntLiteral "," ArrayRankList
                      | IntLiteral

NLOpt                -> NL:*

NL                   -> NewLine NL
                      | NewLine

NewLine              -> CR LF
                      | CR
                      | LF
                      | ":"

IntLiteral           -> DecDigit:+
                      | HexLiteral
                      | OctLiteral

HexLiteral           -> "&H" HexDigit:+ "&":?
OctLiteral           -> "&" OctDigit:+ "&":?

DecDigit             -> [0-9]
HexDigit             -> [0-9A-Fa-f]
OctDigit             -> [0-7]

IDNameChar           -> [\x20-\x5A\x5C\x5E-\x7E\xA0]

Letter               -> [a-zA-Z]

LF                   -> [\n]

CR                   -> [\r]

IDTail               -> [a-zA-Z0-9_]:*                               {% data => data[0].join('') %}

and the script:

Dim test1, test2, test3

Produces:

{
   "type": "Program",
   "body": [
      {
         "type": "VariableDeclaration",
         "kind": "let",
         "declarations": [
            {
               "type": "VariableDeclarator",
               "id": {
                  "type": "Identifier",
                  "name": "test1"
               }
            },
            {
               "type": "VariableDeclarator",
               "id": {
                  "type": "Identifier",
                  "name": "test2"
               }
            },
            {
               "type": "VariableDeclarator",
               "id": {
                  "type": "Identifier",
                  "name": "test3"
               }
            }
         ]
      }
   ]
}

which should compile into JavaScript I hope!

EDIT: After a fix, it does!

console.log(astring.generate(parser.results[0]));

let test1, test2, test3;

[jsm174]

Wow. That's awesome!

Before seeing this, I was able to figure out why Nearley was locking up in the example I put up.

If you look at:

https://github.com/jsm174/vbscript-parser/blob/master/src/vbscript.bnf#L725-L734

You'll see in the rosettacode they were using IntLiteral as a Goldparser variable and a terminal.

I was able to get a bit more parsing working. Figuring out the best place to put the whitespace will be interesting.

Seeing how far you got, is there anyway I can help out?

[freezy]

If you want I can integrate a first working part into the main repo, then you can commit there directly.

I just took a break from porting the physics engine, which is pretty far advanced now, so I would go back to that and let you work on the scripting. That doesn't mean I won't help there, but it would avoid both of us solving the same problems.

[freezy]

I pushed a scripting branch!

https://github.com/vpdb/vpx-js/tree/feature/scripting

To compile: npm run compile. This compiles the grammar as well as the helper functions as they are written in TS and I don't know how to include them in the grammar otherwise.

What's cool in Typescript is that there are type definitions for ESTree! That should help a lot when post-processing. I've also added a unit test to give you an idea how easy it is to test this.

You should have write access, so feel free to push your changes directly to that branch!

Cheers :)

[freezy]

A good read (someone who did the same thing for C#). Gonna need this as well.

[jsm174]

Fantastic! Thank you.

I figured I would work on seeing if I could get a simple Const working.

	it('should transpile a Const declaration', () => {
		const vbs = `Const pi = 3.14`;
		const js = vbsToJs(vbs);

		console.log('Javascript: ' + js);

		//expect(js).to.equal('var pi = 3.14;\n');
	});

Trying to figure out a good dev workflow. TBH, I'm new to mocha and green with Typescript.

To make things faster, I updated mocha.opts:

--require ts-node/register
--require esm
--sourcemap
--timeout 10000
#--file test/setup.ts
#test/**/*.spec.ts
#lib/**/*.spec.ts
lib/scripting/*.spec.ts

Then the flow is:

npm run compile
npm run test

Is that roughly how you work?

[freezy]

Yeah, though in my IDE (IntelliJ IDEA) I can just click on tests and run them individually. Otherwise in the test you can replace the it and describe function by it.only and describe.only respectively and it'll single-out the test(s) in question. Easier than fiddling with mocha.opts.

I hope you're not too much disencouraged by the explanations in the Dan Roberts repo. I suspected that VBScript was a mess, but didn't know it was that bad. Let's start small and iterate from there :)

[freezy]

About your test:

const vbs = 'Const pi = 3.14';

I think you're missing the trailing '\n', since AFAIK the grammar for the const declaration ends with a newline.

[jsm174]

Disencouraged? :) nahh. Everything MS did back then was a mess.

My thoughts (hopes) are most of the table scripts are not using the more "complicated" parts of the language.

For some reason I'm stuck in my ways with Eclipse and CodeMix. I have to force myself switch to IntelliJ or VS Code. :)

[freezy]

My thoughts (hopes) are most of the table scripts are not using the more "complicated" parts of the language.

Mine too! Error handling worries me a little but that's also because I have never written any VBS before. But it seems quite horrible to manage, let alone translate it to another language.

[freezy]

Yesterday I've started implementing Visual Pinball's C API in JavaScript. So far the kicker is done (it also has good coverage). This is going to be the link between the scripting engine and Visual Pinball.

The advantage of the kicker is that's immediately usable. Create a kicker on an empty table, name it BallRelease and this VBScript in Visual Pinball will work:

BallRelease.CreateBall
BallRelease.Kick 0, -2

@jsm174 if you have some spare time to get simple method calls working, that would be awesome. I need to do some more wiring, but that would allow us to get a first table with a simple working script running!

[jsm174]

Sounds good. I have been working (I promise) on the simple Const, and I'm trying to wrap my head around the post processors for estree. I hope to have something shortly.

[freezy]

Cool! Let me know if you need help with post processors.

[jsm174]

So I committed vpdb/vpx-js@b51b03a.

It works, but I know it could be much better (and more correct). :)

I'm thinking if I stare at the post processors long enough, it will start to click - flattening from arrays, etc.

Should we move this conversation over to vpx-js?

[freezy]

Thanks, looks great! :)

I have a huge PR with all the physics that I'll merge today or tomorrow, then you can rebase your branch on that. Feel free to open a PR once rebased, then we can continue the discussion there.

However I propose to merge your scripting changes regularly and open new PRs as you progress, that allows the main branch to already integrate with what the scripting translator is able to do.

The pieces of the puzzle are coming together! :)

[jsm174]

Yup. They sure are! I just commited vpdb/vpx-js@f472624. It gives support for BallRelease.CreateBall.

I'm still trying to figure out which rules actually get used for BallRelease.Kick 0, -2

I think we are in

<SubCallStmt>          ::= <QualifiedID> <SubSafeExprOpt> <CommaExprList>
                         | <QualifiedID> <SubSafeExprOpt>
                         | <QualifiedID> '(' <Expr> ')' <CommaExprList>
                         | <QualifiedID> '(' <Expr> ')'
                         | <QualifiedID> '(' ')'
                         | <QualifiedID> <IndexOrParamsList> '.' <LeftExprTail> <SubSafeExprOpt> <CommaExprList>
                         | <QualifiedID> <IndexOrParamsListDot> <LeftExprTail> <SubSafeExprOpt> <CommaExprList>
                         | <QualifiedID> <IndexOrParamsList> '.' <LeftExprTail> <SubSafeExprOpt>
                         | <QualifiedID> <IndexOrParamsListDot> <LeftExprTail> <SubSafeExprOpt>

If it's SubSafeExprOpt, yikes. :)

I'm going to test in GoldParser, just to make sure I'm in the right place.

[freezy]

I think what you're looking for is:

<SubCallStmt>          ::= <QualifiedID> <CommaExprList>

which is covered by <QualifiedID> <SubSafeExprOpt> <CommaExprList>, because <SubSafeExprOpt> is optional (<SubSafeExpr> |, meaning SubSafeExpr or nothing).

In nearley this would just become:

SubCallStmt          -> QualifiedID __ SubSafeExpr:? __ CommaExprList

[jsm174]

Yeh, I really wanted to see what it was doing for the arguments portion 0, -2.

I have some good stuff to go on now:

[freezy]

Oh, this is pretty cool! I didn't know there was a GOLD parser tool. What does the tree tab say?

[freezy]

Ok got it. So "all" you gotta do is convert this kinda tree:

into this one:

[jsm174]

The AST from above is already in the feature/scripting branch except for the arguments portion.

So BallRelease.CreateBall will generate as BallRelease.CreateBall().
(At least you can test that part).

I'll work on getting arguments working tonight. :)

[freezy]

PR at vpdb/vpx-js#32

[freezy]

Closing this in favor of vpdb/vpx-js#111 and future other follow-ups.