Catching "Stack Overflow" exceptions in recursive C++ functions

It's not an exception per se, but if you just want to be able to limit your stack usage to a fixed amount, you could do something like this:

#include <stdio.h>

// These will be set at the top of main()
static char * _topOfStack;
static int _maxAllowedStackUsage;

int GetCurrentStackSize()
{
   char localVar;
   int curStackSize = (&localVar)-_topOfStack;
   if (curStackSize < 0) curStackSize = -curStackSize;  // in case the stack is growing down
   return curStackSize;
}

void MyRecursiveFunction()
{
   int curStackSize = GetCurrentStackSize();
   printf("MyRecursiveFunction:  curStackSize=%i\n", curStackSize);

   if (curStackSize < _maxAllowedStackUsage) MyRecursiveFunction();
   else
   {
      printf("    Can't recurse any more, the stack is too big!\n");
   }
}

int main(int, char **)
{
   char topOfStack;
   _topOfStack = &topOfStack;
   _maxAllowedStackUsage = 4096;  // or whatever amount you feel comfortable allowing

   MyRecursiveFunction();
   return 0;
}

There's really no portable way to do it. An out of control recursive function will usually cause an invalid memory access when it tries to allocate a stack frame beyond the stack address space. This will usually just crash your program with a Segmentation Fault/Access Violation depending on the OS. In other words, it won't throw a c++ exception that can be handled in a standard way by the language.

There isn't a portable way. However, there are a few nonportable solutions.

First, as others have mentioned, Windows provides a nonstandard __try and __except framework called Structured Exeption Handling (your specific answer is in the Knowledge Base).

Second, alloca -- if implemented correctly -- can tell you if the stack is about to overflow:

bool probe_stack(size_t needed_stack_frame_size)
{
    return NULL != alloca(needed_stack_frame_size);
};

I like this approach, because at the end of probe_stack, the memory alloca allocated is released and available for your use. Unfortunately only a few operating systems implement alloca correctly. alloca never returns NULL on most operating systems, letting you discover that the stack has overflown with a spectacular crash.

Third, UNIX-like systems often have a header called ucontext.h with functions to set the size of the stack (or, actually, to chain several stacks together). You can keep track of where you are on the stack, and determine if you're about to overflow. Windows comes with similar abilities a la CreateFiber.

As of Windows 8, Windows has a function specifically for this (GetCurrentThreadStackLimits)

Even if you can do this non-portably, as you can in Windows, it's still a very bad idea. The best strategy is to not overflow the stack in the first place. If you need isolation from some code you don't control, run that code in a different process and you can detect when it crashes. But you don't want to do that sort of thing in your own process, because you don't know what sort of nasty corruption of state the offending code is going to do, and that will make you unstable.

There's an interesting, somewhat related blog post by Microsoft's Raymond Chen about why you shouldn't try to check for valid pointers in a user mode application on Windows.