Serialization of Expressions

[ghost]

NOTE: Sorry for the long post, I just kept finding interesting stuff so I'm posting them here.

Constructing an expression from pure data sounds really useful to me. Currently, I'm working on a game where I end up with an indefinite number of expressions at the development phase. I'm storing these as string along with any arguments etc. and using that data to instantiate an Expression object.

The problem with this approach is that the loading time of my game varies greatly based on the number of expressions. Even though the expression has been tested for its syntax at the development phase, it still needs to go through syntax checking at the final product. This is what may make loading take a long of time.

De-serializing data into an expression and avoiding the syntax check would greatly improve workflows where you know beforehand that the expression is correct. You could even send a great number of Expressions over a network and all of them would have no visible performance costs.

EDIT 1: I noticed there is an internal constructor that can actually create an Expression when given the appropriate arguments. I will try to make a public constructor that takes a List of Tokens and constructs the rest of the arguments through that. Any pointers or a reason on why not to do that would be appreciated.

EDIT 2: Another approach would be to make all the expressions go through an ISerializer where the generic solution would be to convert from - to bytes. Each serializer could also implement a more explicit approach, i.e from - to json for a JsonSerializer. I have also seen that there are some interfaces for user extension, but these are language-bound. That could be solved by providing user-data in the serializer. I believe this approach is the better one.

EDIT 3: I've added this constructor for my use case. (working in C#)

public Expression(string expressionString, List<Token> tokens, 
    params PrimitiveElement[] elements) : base(Expression.TYPE_ID)
{
    expressionInit();
    this.expressionString = expressionString;
    addDefinitions(elements);
    initialTokens = tokens;
    setSyntaxStatus(true, "");
    internalClone = true;
}

Tokens are easily serializable as they are written, so I had no need to add anything for their serialization inside the library. Do note the code below for fast free Argument instantiation and a comparison between the two ways.

//This is really fast
health = new Argument("health",15);
empower = new Argument("empower", 22);
//This is really slow
health = new Argument("health = 15");
empower = new Argument("empower = 22");

//Release Console app was launched in VS, if it was a raw .exe perhaps it would be faster

//Fast arguments no tokens  ~ 25-40ms for each new expression 
expression = new Expression("(health + 2) * empower",  health, empower);
var result = expression.calculate();
//Fast arguments passing serialized tokens ~ 0-8ms, takes longer 
//depending on what .NET has loaded when you call calculate. If everything is loaded
//then it takes 0-1ms
//Not bound by an expression instance
expression = new Expression("(health + 2) * empower", tokens, health, empower);
var result = expression.calculate();

Bear in mind that this is just a simple work-around for me to get a HUGE performance boost for my use-case. It offers pseudo de-serialization of Arguments and Expressions (i.e the above code does not work for dependent Arguments). Instantiating an expression is fast but it gets really slower if you want to instantiate 10000 expressions and more (who would want that, though).

True de-serialization would always be fast, period. I suppose Tokens are the main part to deal with.. but frankly, this is a complex framework and would take some time for me to understand what data needs to be stored and I don't have that time. If the developer ever sees this, I'd be happy to discuss and potentially help if I learn what needs to be serialized.

[mariuszgromada]

In the next release I will try to help with that :-)

[mariuszgromada]

Binary Serialization Utils for MathParser.org-mXparser for JAVA and .NET

• SerializationUtils.enableBinarySerializationIamAwareOfSecurityRisks();
• SerializationUtils.disableBinarySerialization();
• SerializationUtils.isBinarySerializationEnabled();
• SerializationUtils.checkLastOperationWasSuccessful();
• SerializationUtils.getLastOperationMessage();
• SerializationUtils.serializeToBytes(...);
• SerializationUtils.serializeToString(...);
• SerializationUtils.serializeToFile(...);
• SerializationUtils.deserializeFromBytes(...)
• SerializationUtils.deserializeFromString(...)
• SerializationUtils.deserializeFromFile(...)

SECURITY WARNING: Deserializing data from an untrusted source can introduce security vulnerabilities to your application. Depending on the settings used during deserialization, untrusted data may be able to execute arbitrary code or cause a denial of service attack. Untrusted data can come from over the network from an untrusted source (e.g. any network client), or it can be manipulated/tampered by an intermediary while in transit over an unauthenticated connection, or from local storage where it may have been compromised/tampered, or from many other sources. MathParser.org-mXparser does not provide any means to authenticate data or secure it from tampering. Use an appropriate data authentication method before deserializing. Be very mindful of these attack scenarios; many projects and companies and users of serialization libraries in general have been bitten by untrusted deserialization of user data in the past.

https://learn.microsoft.com/en-us/dotnet/core/compatibility/core-libraries/7.0/serializationformat-binary