I thought I’d blogged about this a year or two ago but when I tried to point a friend to it I found that I had not…
Consider the following C/C++ program (MemoryClobber.exe):
int main(int argc, char** argv)
{
double* d = new double[100];
for (int i = 0; i < 10000; i++)
d[i] = 0;
return 0;
}
It allocates an array of 100 doubles and then initializes 10000 of them. This demonstrates a classic writing over the heap bug.
If I run this is Visual Studio, even with runtime checking on (/RTC1), it does throw an exception but not until i is 1477. So 1376 doubles of memory have been overwritten before it hits something that causes Visual Studio to realize something is up.
Obviously this is bad. What we really want to know is when d[100] is written to. Even worse, this is a contrived example. What something is getting clobbered in a real application with lots of code and data?
Fortunately Microsoft has a debugging utility that can help: Global FLAGs (gflags.exe). It’s part of the Debugging Tools for Windows package. Gflags.exe has many options but what we’re interested in is the pageheap (-p) options.
Assuming you’re in the directory where MemoryClobber.exe is located and that the Windows Debugging tools are in your path the following will set up heap checking for MemoryClobber.exe:
gflags -p /full /protect /unaligned /enable MemoryClobber.exe
Now if you run MemoryClobber in Visual Studio it blows when i is 100 as desired.
To disable heap checking for MemoryClobber.exe run:
gflags -p /disable MemoryClobber.exe
The obvious question is what’s the downside? The downside is in how the heap checking works. It works by putting guard pages before and after the memory allocation to instantly detect memory over/underruns. The good news is this reasonably fast but if your program allocates a lot of memory you can quickly run the process out of pages. The main solutions for this are to figure out how to reproduce the problem with a smaller case or to run it on a 64 bit machine where there’s a lot more addressable memory.
