[RFC] The with statement

[plusvic]

YARA users sometimes create complex rules that use the same expression multiple times in the rule. When this happens the rule's condition becomes very verbose and repetitive. Let's see an example:

import "pe"

rule list_of_hashes {
  condition:
       pe.sections[0] .name == ".text" and
       pe.sections[0].characteristics == 0xC0000000 and
       pe.sections[0].raw_data_size == 0x2000 and
       pe.sections[0].raw_data_offset == 0x1000
       pe.sections[pe.number_of_sections - 1] .name == ".tls" and
       pe.sections[pe.number_of_sections - 1].characteristics == 0xC0000000 and
       pe.sections[pe.number_of_sections - 1].raw_data_size == 0x1000 and
       pe.sections[pe.number_of_sections - 1].raw_data_offset == 0x4000
}

The condition above uses the expressions pe.sections[0] and pe.sections[pe.number_of_sections - 1] multiple times for referring to the first and last sections of the PE respectively. This could be expressed an a more compact and legible way if those expressions could be replaced by identifiers holding the values of pe.sections[0] and pe.sections[pe.number_of_sections - 1].

This RFC proposes the introduction of a new with statement aimed to solve this problem, and that would look like this:

import "pe"

rule list_of_hashes {
  condition:
      with 
         fs = pe.sections[0], 
         ls = pe.sections[pe.number_of_sections - 1] : 
      (
         fs.name == ".text" and
         fs.name.characteristics == 0xC0000000 and
         fs.name.raw_data_size == 0x2000 and
         fs.name.raw_data_offset == 0x1000
         ls.name == ".tls" and
         ls.characteristics == 0xC0000000 and
         ls.raw_data_size == 0x1000 and
         ls.raw_data_offset == 0x4000
      )
}

The syntax for this new statement would be:

with 
    <identifier> = <expression> [,<identifier> = <expression>]* : 
    (
        <boolean expression>
    )
}

The with statement can define one or more identifiers and assign them the values of arbitrary expressions. Of course, this expressions can use any identifier that is visible in the scope where the with statement resides. Each expression is evaluated once, and the resulting value is reused every time the corresponding identifier is used within the boolean expression.

This statement also helps reducing the number of expensive computations, for example:

with file_hash = hashes.sha256(0, filesize) :
(
   file_hash == "479d392....."  or
   file_hash == "50a3b31....."  or
   file_hash == "a2b2024....."  or
)

In the example above the expensive function hashes.sha256 is called once, but its result is re-used three times in the boolean expression.

Note: The hashes.sha256 has an internal cache that saves the results of previous calls and avoids re-computing the same hashes over and over again. This cache was introduced precisely because YARA lacked a way for storing and re-using the result of a function call.

This statement is also useful within loops, for example:

for all offset in (10,20,30) : (
   with val = uint64(offset) | uint64(offset + 4) | uint64(offset + 8) : (
      val == 0x10000 or 
      val == 0x20000 or 
      val == 0x40000
   )
)

Without the with statement the condition should be:

for all offset in (10,20,30) : (
   uint64(offset) | uint64(offset + 4) | uint64(offset + 8) ==  0x10000 or 
   uint64(offset) | uint64(offset + 4) | uint64(offset + 8) ==  0x20000 or 
   uint64(offset) | uint64(offset + 4) | uint64(offset + 8) ==  0x40000 or
)

[wxsBSD]

I like this, it will make writing more complex rules less verbose. I assume this is meant to be orthogonal to the idea of globals as discussed elsewhere as the implementation of the two are not likely to be similar?

[plusvic]

With "globals" do you mean global identifiers that are visible across all rules? Or identifiers that are defined within the scope of a specific rule like show below?

variables:
    foo = <expr>
condition:
    foo

The with statement should be enough if you are using rule variables only for simplifying your condition, however, if we want to use rule variables as a way for exposing their values to the outside world the with statement won't be enough. In that regard I lean to complementing the with statement with a new section in the rule names outputs, like in:

outputs:
    foo = <expr>
    bar = <expr>
condition:
   <boolean expr>

[wxsBSD]

With "globals" do you mean global identifiers that are visible across all rules? Or identifiers that are defined within the scope of a specific rule like show below?

I mean global identifiers visible across all rules. Think of things like first_dword = uint32(0) or is_pdf = uint32be(0) == 0x25504446.

The with statement should be enough if you are using rule variables only for simplifying your condition, however, if we want to use rule variables as a way for exposing their values to the outside world the with statement won't be enough. In that regard I lean to complementing the with statement with a new section in the rule names outputs, like in

I think this RFC is great for rule variables, and exposing them to the outside world is not something I had considered, and I am unable to come up with a nice use-case right now. As such, I think with as you have described it in this RFC is great but we may want to consider global identifiers to further simplify expressions. Not necessarily in this RFC, but at least want to get it out there as they are somewhat related in idea.