MYST_RETAIN_SYMBOLS env var to retain symbols

Normally, when run under a debugger, Mystikos writes out each loaded shared library
to /tmp/myst<abcdef>/ so that the debugger can load symbols from these libraries.
At the end of the execution these files are deleted.

[rr debugger](https://rr-project.org/) allows `record and replay` debugging.
You record a failure once, then debug the recording, deterministically, as
many times as you want. The same execution is replayed every time.
rr also provides efficient reverse execution under gdb. Set breakpoints and
data watchpoints and quickly reverse-execute to where they were hit.

At a very high-level, rr works by capuring all kernel interactions,
including thread scheduling, and then re-applying all the interactions during playback.
The playback is not a real-execution, and does not produce most side effects
(e.g writing files, opening sockets etc).

Thus, during playback, the shared libraries necessay for the debugger are not written out.
To make rr work for Mystikos, the symbol files need to be retained during execution.
That is the purpose of the MYST_RETAIN_SYMBOLS flag.

------------------------------------------------------------------------------------------

Prerequisites:

- Install rr from https://github.com/rr-debugger/rr/releases
- Read up https://rr-project.org/
- Try out rr on a simple hellworld program outside mystikos
  $ rr record helloworld
  $ rr replay
  rr will suggest the necessary configuration settings that need to be made.
  Do as suggested.

Usage:

For a test, doing `make TARGET=linux` would print out the actual Mystikos command.
Execute `MYST_RETAIN_SYMBOLS=1 rr record <mystikos command>` to create a recording.

E.g:
   $ MYST_RETAIN_SYMBOLS=1 rr record myst exec-linux rootfs /bin/hello red green blue
   rr: Saving execution to trace directory `/home/user/.local/share/rr/myst-35'.
   Hello world!
   I received: argv[0]={/bin/hello}, argv[1]={red}, argv[2]={green}, argv[3]={blue}
   === passed test (/bin/hello)

To replay and debug, do `rr replay -d /path/to/myst-gdb`
E.g:
  $ rr replay -d ../../../build/bin/myst-gdb
  GNU gdb (Ubuntu 11.1-0ubuntu2) 11.1
  0x00007fe413b720d0 in _start () from /lib64/ld-linux-x86-64.so.2
  (rr)

Most GDB commands are supported; but keep in mind that this is a replay, not an actual
execution. It is like watching a video where you can easily skip forward and backward,
but not actually waching something actually happen.

Put a breakpoint:
   (rr) b main
   Breakpoint 2 at 0x10000024e32: file host.c, line 527.
   (rr) c
   Continuing.

   Breakpoint 2, main (argc=7, argv=0x7ffda41c9688, envp=0x7ffda41c96c8) at host.c:527
   527	    int ec = _main(argc, argv, envp);
   (rr) c
   Continuing.
   oegdb: Loaded enclave module /home/anand/msft/mystikos/build/lib/libmystkernel.so
   oegdb: analyzing symbols for module /home/anand/msft/mystikos/build/lib/libmystkernel.so
   oegdb: Loaded enclave module ./.mystX68Tlv/libmystcrt
   oegdb: analyzing symbols for module ./.mystX68Tlv/libmystcrt
   oegdb: Loaded enclave module ./.mystX68Tlv/hello
   oegdb: analyzing symbols for module ./.mystX68Tlv/hello

   Breakpoint 2, main (argc=4, argv=0x7fe412cf2010) at hello.c:10
   10	    assert(argc == 4);
   (rr)

Step over, debug as before.
   (rr) n
   11	    assert(strcmp(argv[0], "/bin/hello") == 0);
   (rr) n
   12	    assert(strcmp(argv[1], "red") == 0);
   (rr) n
   13	    assert(strcmp(argv[2], "green") == 0);
   (rr) n
   14	    assert(strcmp(argv[3], "blue") == 0);

Reverse execution works, but seems flaky. Needs more investigation.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
Signed-off-by: Vikas Tikoo <vikasamar@gmail.com>

tools/myst/host/host.c

@@ -191,6 +191,23 @@ long init_symbol_file_tmpdir(char* tmpdir)
     return ret;
 }
 
+static bool _retain_symbols()
+{
+    static int r = -1;
+    // Initialization of r happens when myst_tcall_add_symbol_file is called
+    // for the main application, before other app threads are launched.
+    // Therefore, locks are not necessary.
+    if (r == -1)
+    {
+        const char* env;
+        if ((env = getenv("MYST_RETAIN_SYMBOLS")) && strcmp(env, "1") == 0)
+            r = 1;
+        else
+            r = 0;
+    }
+    return r;
+}
+
 long myst_tcall_add_symbol_file(
     const void* file_data,
     size_t file_size,
@@ -213,6 +230,12 @@ long myst_tcall_add_symbol_file(
 
     if (!tmpdir_init)
     {
+        /* If symbols need to be retained, use a folder in the current directory
+         * so that the symbols survive a machine reboot */
+        if (_retain_symbols())
+        {
+            sprintf(tmpdir, "./.mystXXXXXX");
+        }
         ECHECK(init_symbol_file_tmpdir(tmpdir));
         tmpdir_init = 1;
     }
@@ -368,8 +391,12 @@ long myst_tcall_unload_symbols(void)
 {
     long ret = 0;
 
-    myst_spin_lock(&_modules_lock);
+    // If symbols need to be retained, don't delete the emitted shared
+    // libraries.
+    if (_retain_symbols())
+        return ret;
 
+    myst_spin_lock(&_modules_lock);
     for (debug_module_t* p = _debug_modules; p;)
     {
         debug_module_t* next = p->next;
@@ -384,7 +411,6 @@ long myst_tcall_unload_symbols(void)
 
         p = next;
     }
-
     myst_spin_unlock(&_modules_lock);
     return ret;
 }