Instruments vs Xcode

[oscbyspro]

As noticed in (#109), the measurements of Instruments' time profiler and Xcode's xctestplan are completely different. It's frustrating, because I cannot use the former to see why the latter takes so long. Ideally, I would just measure some real-world use case, but I haven't gotten around to replacing my arbitrary precision solution yet. As an example, the following takes 8.18 seconds with Instruments' time profiler but 54.5 seconds with Xcode's xctestplan:

func testDividingFullWidth() {
    var lhs = _blackHoleIdentity((UInt256.max))
    var rhs = _blackHoleIdentity((UInt256.max, UInt256.max))
    
    for _ in 0 ..< 100_000_000 {
        _blackHole(lhs.dividingFullWidth(rhs))
        _blackHoleInoutIdentity(&lhs)
        _blackHoleInoutIdentity(&rhs)
    }
}

---

Edit: I'm aware that this is an overflowing example. It mirrors Swift's standard semantics:

UInt.max.dividingFullWidth((UInt.max, UInt.max)) // truncating semantics, no crash

[oscbyspro]

I'm hoping Instruments is correct on this one, not so much because it's faster, but because the xctestplan measurements make pointers seem unusable. Like, single digit addition (pointer-based) appears somewhat faster than full-width addition with Instruments' time profiler, but 2x slower with Xcode's xctestplan.

[oscbyspro]

So, after seven eternities, I found the xctestplan option that matters:

Does anyone know what it does? Leaving it switched on ruins performance.

[oscbyspro]

I found this comment from  at developer.apple.com:

Note: Code coverage data collection incurs a performance penalty. Whether the penalty is significant or not, it should affect execution of the code in a linear fashion so performance results remain comparable from test run to test run when it is enabled. However, you should consider whether to have code coverage enabled when you are critically evaluating the performance of routines in your tests.

[oscbyspro]

Hm, the mystery is solved. I find it strange, however, that it was enabled by default. I've not had that problem before.