E2 Compiler Rewrite

[Vurv78]

Fixes #2544
Fixes #2049
Fixes #2494
Fixes #2696
Fixes #2713 (by removing optimizer)
Fixes #2667

Overview of Changes

• Rewrite of the parser
• Rewrite of the compiler
• Rewrite of the extension preprocessor
• Tokenizer and Preprocessor now return multiple errors to the editor
• Opcosts generally decreased
• Removed wire_expression2_delta functionality
• Removed the optimizer step (to be reimplemented later)

This is all to achieve a faster and simpler runtime.
For example, for (and foreach) loops run ~100x faster now.

Users

This shouldn't break any of your E2s.
Any broken E2s are caused by bugs and will be fixed.

The only user-facing change is that using j, i, k literals on their own is not accepted.
If you don't know what that is, you don't need to worry about it. Otherwise, see below.

Fixes

A long time issue with switch case has been fixed, which was related to how the old compiler worked. It used to evaluate the expression at the top (ie. a function call) for every single case. This no longer happens on the new compiler.

Complex and quaternion literals are no longer ambiguous. Before, i, j, and k were valid expressions that would equate to complex and quaternion literals. This is no longer allowed (you must prefix them with a 1 now, so 1j). This resolves an issue introduced by the tokenizer rewrite.

Extension Developers

registerFunction and registerOperator

These functions will remain the same by default, but will likely be deprecated in the future, since there is a new form of registering functions and operators with the runtime that is much faster.

If you want to use this new format, either use the e2function syntax, or set the legacy attribute (in the attributes parameter for the function) to false.

The differences between the two formats is that the old format would pass instructions not yet evaluated to the function/operator, while the new format passes the raw values. (If you don't know what this means, stick to e2function syntax).

Specific operator changes

is / operator_is

This no longer supports the legacy format as stated above. If you are using e2function syntax, you should be fine.
Otherwise, you should switch to using e2function syntax or look into the source code for examples.

and / or / operator& / operator|

These operators are no longer used. Instead the compiler uses the is operator for the type.

ass

This operator was the "assignment" operator. It has been removed, and assignment is now implemented in the compiler for all types automatically.

inc / dec

These operators are no longer being used.
Now incrementing and decrementing is syntax sugar for A = A + 1, so any type with + or - operators can be incremented/decremented.

fea

The fea (foreach) operator has been removed and replaced with iter. This expects the operation to return a lua iterator when given the value to "turn into an iterator".

This was done because the fea operator is not used by any extension, and requires the extension author to invoke the E2 chip's runtime itself, which should not be up to the extension author.

Example:

local function iterc(str, i)
	i = i + 1
	if i < #str then
		return i, string_sub(str, i, i)
	end
end

registerOperator("iter", "ns=s", "", function(state, str)
	state.prf = state.prf + #str
	return function()
		return iterc, str, 0
	end
end)

idx

This operator has been split into indexget and indexset respectively.

Example for angle:

registerOperator("indexget", "an", "n", function(state, this, index) -- getter
	return this[floor(math.Clamp(index, 1, 3) + 0.5)]
end)

registerOperator("indexset", "ann", "", function(state, this, index, value)
	this[floor(math.Clamp(index, 1, 3) + 0.5)] = value
	state.GlobalScope.vclk[this] = true
end)

Todos

• String calls to variadic functions.
• Incrementing (++) and decrementing (--) non-numeric values
• A lot of testing.

[Vurv78]

Now user functions highlight and autocomplete, even across included files:

[thegrb93]

Is this ready?

[Vurv78]

Yeah, I haven't had to patch many bugs at all within the past few months, and it's worked fine for the two servers that I know that use it, (E2BI and TAS). I wanted to get a larger server like CFC to try it, but that didn't end up happening.

I was afraid of some giant internal compiler bug that I wouldn't be able to patch quickly happening, but I don't think that will happen.

Also having this unmerged has largely stalled progress on E2..

[Vurv78]

Gonna merge later today if there's no objections, since I'll have the weekend to patch any potential bugs that show up

[dvdvideo1234]

Why is this deprecated ?

[Denneisk]

Luajit can't optimize Lua varargs. E2 already passes args as a table, so doing Lua varargs means you're unpacking a table and then repacking it for no gain.