Macro annotation (part 1)

[nicolasstucki]

Add basic support for macro annotations

• Introduce experimental scala.annotations.MacroAnnotation
• Macro annotations can analyze or modify definitions
• Macro annotation can add definition around the annotated definition
  • Added members are not visible while typing
  • Added members are not visible to other macro annotations
  • Added definition must have the same owner
• Implement macro annotation expansion
  • Implemented at Inlining phase
    • Can use macro annotations in staged expressions (expanded when at stage 0)
    • Can use staged expression to implement macro annotations
    • Can insert calls to inline methods in macro annotations
  • Current limitations (to be loosened in following PRs)
    • Can only be used on def, val, lazy val and var
    • Can only add def, val, lazy val and var definitions

Example

class memoize extends MacroAnnotation:
  def transform(using Quotes)(tree: quotes.reflect.Definition): List[quotes.reflect.Definition] =
    import quotes.reflect._
    tree match
      case DefDef(name, TermParamClause(param :: Nil) :: Nil, tpt, Some(rhsTree)) =>
        (Ref(param.symbol).asExpr, rhsTree.asExpr) match
          case ('{ $paramRefExpr: t }, '{ $rhsExpr: u }) =>
            val cacheTpe = TypeRepr.of[Map[t, u]]
            val cacheSymbol = 
              Symbol.newVal(tree.symbol.owner, name + "Cache", cacheTpe, Flags.Private, Symbol.noSymbol)
            val cacheRhs = '{ Map.empty[t, u] }.asTerm
            val cacheVal = ValDef(cacheSymbol, Some(cacheRhs))
            val cacheRefExpr = Ref(cacheSymbol).asExprOf[Map[t, u]]
            val newRhs = '{ $cacheRefExpr.getOrElseUpdate($paramRefExpr, $rhsExpr) }.asTerm
            val newTree = DefDef.copy(tree)(name, TermParamClause(param :: Nil) :: Nil, tpt, Some(newRhs))
            List(cacheVal, newTree)
      case _ =>
        report.error("Annotation only supported on `def` with a single argument are supported")
        List(tree)

with this macro annotation a user can write

@memoize
def fib(n: Int): Int =
  println(s"compute fib of $n")
  if n <= 1 then n else fib(n - 1) + fib(n - 2)

and the macro will modify the definition to create

val fibCache = mutable.Map.empty[Int, Int]
def fib(n: Int): Int =
  fibCache.getOrElseUpdate(
    n,
    {
      println(s"compute fib of $n")
      if n <= 1 then n else fib(n - 1) + fib(n - 2)
    }
  )

Based on

• Basic Design of Macro Annotations #15626
• https://infoscience.epfl.ch/record/294615?ln=en

Followed by

• Macro annotations class modifications (part 2)  #16454

[jilen]

Why macro annotations cannot be used for ClassDef ? In scala2 macro annotations are mostly used for typeclass derivation, so most are used for ClassDef.

It will be great for migration if macro annotations can be used with ClassDef

[nicolasstucki]

Why macro annotations cannot be used for ClassDef ?

That will be added in the next PR.

[aherlihy]

Looks good, I played around with some macro annotations and they behave how I expected. I put some clarification questions in the comments but none of them are blockers.

[aherlihy]

what is a reflected representation?

[nicolasstucki]

Seems it should just be

 * This method takes as argument the annotated definition.

[aherlihy]

Background knowledge question - other than SpliceInterpreter in Splicer.scala, prior to this PR, was this class used anywhere?

[nicolasstucki]

The splice interpreter was the only use of this class. This class was factored out in a separate PR to make this PR simpler to review.

[aherlihy]

Would you mind explaining why was this check removed (or actually, what was it doing before and why is it not needed anymore?)

[nicolasstucki]

This check was there to make sure we only called interpretNew on a top-level class. I noticed that our current implementation of interpretNew also supports classes nested in objects.

[aherlihy]

Is this to handle the names generated by newUniqueMethod/Val?

[nicolasstucki]

No. It is to handle classes that are nested withing objects.

For example if we have the class B.A defined in object B (which has a companion class B$), we would see the name B$.A at this point in the compiler. But we lift those classes and therfore we will get a class B$A in the bytecode. This transforms B$.A into B$A.

[smarter]

This should probably use mangledString and not toString to account for name encoding (e.g., try definining and using class +). But also, what about inner classes? If I have class Outer { class Inner }; val o = new Outer; new o.Inner, then the binary class name is also Outer$Inner . I have logic in https://github.com/lampepfl/dotty/blob/e8428102f8372e9c4227a3a13970e15bfc5d94c1/compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala#L147-L171 to do this which could be extracted in its own method.

[nicolasstucki]

I will use mangleString.

We do not need to handle inner classes for now. We do not allow macro annotations to be defined as inner classes (restriction). We wont encounter new in top-level splices neither.

[aherlihy]

Is this because you may end up in a situation where the class defined by the macro annotation does not yet have a definition?

[nicolasstucki]

Yes, does not have a definition in bytecode.

This can happen when we compile the macro definition and use at the same time. In that case we have the definition as a symbol in the compiler but the bytecode for the macro definition has not been generated yet.

[aherlihy]

I'm not sure I understand the distinction between MacroAnnotations and TastyAnnotations, other than one gets checked on a tree the other gets checked on a symbol. what's the difference, conceptually?

[nicolasstucki]

This is a mistake on my part. It should be TastyAnnotations. I will update that.

We started with the name Macro Annotation following the Scala 2 name of the feature. But then decided we should use a different name to make it clear that they are the same. This will help users when they google documentation.

[nicolasstucki]

Changed back to MacroAnnotation after #16392 (comment)

[aherlihy]

Why move the annotation expansion into a separate phase now?

[nicolasstucki]

I pushed a commit I should not. The macro annotation expansion will be done in Inlining.

I was experimenting to see if it would be possible to extract it in the future to support some extra features.

[odersky]

Otherwise LGTM

[odersky]

Suggested change

	if annot.tree.symbol.denot != NoDenotation && annot.tree.symbol.owner.derivesFrom(defn.TastyAnnotationClass) then
	if annot.tree.symbol.maybeOwner.derivesFrom(defn.TastyAnnotationClass) then

[nicolasstucki]

Updated. Also updated the same pattern in TastyAnnotations.isTastyAnnotation

[odersky]

I think TastyAnnotation is too unclear. I would never think of a macro when seeing that name. Instead I would think it's something to do with the internals of Tasty files.

I think the best thing to do is to go back to MacroAnnotation. We will have to explain how they are different from Scala 2 macro annotations, but it's better to do that directly. Otherwise, when someone googles for Scala 3 macro annotation, they will only find Scala 2 macro annotation stuff. I don't think we can establish TastyAnnotation as an obvious term that people will search for.

[nicolasstucki]

Changed back to MacroAnnotation

[simlei]

Yearning for that content to google for ;)
The thesis linked above is a good intro; I assume the test cases (chapter 5) which were only outlined there, not provided, are the ones in these PRs? In any case, documentation or a further reading list would be appreciated in due time :) Excited for this feature!

[odersky]

Style: We usually prefer an explicit self name like thisPhase over a qualified Inlining.this.

[nicolasstucki]

Updated

[nicolasstucki]

Rebased and fixed conflicts in the Interpreter.

[mrdziuban]

@nicolasstucki thank you for all your work on this! Apologies if this isn't the best place to ask, but I was testing out the @memoize example and wondered if it's possible to reference the generated val fibCache from external code? e.g. in the SBT console, I tried

scala> @memoize def fib(n: Int): Int = {
     |   println(s"compute fib of $n")
     |   if n <= 1 then n else fib(n - 1) + fib(n - 2)
     | }
def fib(n: Int): Int

scala> fibCache
-- [E006] Not Found Error: -----------------------------------------------------
1 |fibCache
  |^^^^^^^^
  |Not found: fibCache

[nicolasstucki]

It is not possible to refer to the generated code from external code. This is by design.

[smarter]

@mrdziuban see also the discussion in https://contributors.scala-lang.org/t/scala-3-macro-annotations-and-code-generation/6035