No feedback from debugger when program overflows stack

[heybdj]

I had an error in my program that caused the stack to overflow. When I tried stepping through my code in the RAD debugger, I would get strange behaviour, like it would start stepping through assembly instructions indefinitely, or it would stop stepping altogether, and the call stack panel would go blank. Sometimes I wouldn't be able to kill my program's process, so I'd have to quit and reopen the debugger. When I first noticed the issue, it seemed like the debugger was getting confused about where my WinMain entry point was.

This is in a non-CRT build, and I was mistakenly not setting a (large enough) /STACK:, size in the linker parameters.

I've attached a minimal example of what I was doing.

I realize this is a niche and specific concern, and it's ultimately my fault for not setting the stack size. But if it's possible to offer any kind of error messaging that my program overflowed its stack, that would be awesome.

Thank you for great tool!
windows_main_bug.zip

[ryanfleury]

I haven't yet confirmed if we are able to detect the stack overflow in this case (VS also doesn't seem to, for example), but I did fix a serious bug that this program exposed, which may be related to some of the weird circumstances you were experiencing. I get similar results to other debuggers now, as long as I specify the Custom Entry Point for the target explicitly as wWinMainCRTStartup. I'll work on getting the exact entry point searching rules right, so that this extra step is not necessary.

[Husamdababneh]

I think I'm running into the same issue here, Here is my case

• No C Runtime
• Stack size reserve and committed is /STACK:0X200000,0X200000 (2MB)
• Custom entry point RemoveCRTTest
• (If needed I can provide all the flags )

#include <windows.h>

struct structure {
    int a;
};

void tt()
{
    constexpr auto BUFFER_SIZE = 1024 * 1024 * 8;
    char bigArray[BUFFER_SIZE] = {};

    // This is handled by `ExceptionHandler`
    // structure* struc = 0;
    // struc->a = 3;
}

int RemoveCRTTest()
{
    AddVectoredExceptionHandler(1, ExceptionHandler);
    tt();
    return 0;
}

this is the generated code

void tt(void) PROC                               ; tt
$LN3:
        push    rdi
        sub     rsp, 8388624                          ; 00800010H ;; 00800000H for the array itself and 0x10H for BUFFER_SIZE variable
        mov     DWORD PTR BUFFER_SIZE$[rsp], 8388608      ; 00800000H
        lea     rax, QWORD PTR bigArray$[rsp]
        mov     rdi, rax
        xor     eax, eax
        mov     ecx, 8388608                          ; 00800000H
        rep stosb
        add     rsp, 8388624                          ; 00800010H
        pop     rdi
        ret     0
void tt(void) ENDP  

The debugger hangs on the SUB instruction that reserves stack space for the procedure, the weird thing is that it reports an access violation 0xc0000005 which I already can handle using the custom ExceptionHandler routine

Note: I have to set the entry point name in the debugger myself to step into the program
Note: I hope I'm helping and not adding complications here

[ryanfleury]

I don't know if there is any way that I can detect that this exception is caused by a stack overflow specifically (which seems like the same boat that Visual Studio is in), but as of 7496f3b, the debugger properly responds to this situation. This change is on dev and will be in the next release. Let me know if you continue having issues with this.