graphviz uses a fuzzy check whether alloca(3) uses malloc(3), which fails in some cases

[neverpanic (Clemens Lang)]

Graphviz uses a check in lib/vmalloc/features/vmalloc that is supposed to find out whether alloca(3) is internally using malloc(3). To do this, this check provides a custom implementation of malloc(3) and assumes that if this implementation is going to be called it was called by alloca(3).

However, this assumption doesn't hold in all cases, e.g. it breaks when having DYLD_INSERT_LIBRARIES is set, as is the case for trace mode, because the Mac OS loader dyld internally calls malloc(3), causing the check to fail.

The check should instead use a global volatile variable to check whether the current call to malloc(3) happened while trying to use alloca(3). The overriden malloc(3) should behave like the standard one in all other cases.

The attached patch fixes this. Additional checks for dlsym(3) and dlfcn.h might be required. Can you take this upstream? I assume the same applies to graphviz-devel, too.

[neverpanic (Clemens Lang)]

Patch against lib/vmalloc/features/vmalloc