OutOfMemoryException on declaration of Large Array

Each double is 8 bytes, so you're trying to allocate a single array with just over 5GB. The CLR has a per-object limit of around 2GB IIRC, even for a 64-bit CLR. In other words, it's not the total amount of memory available that's the problem (although obviously you'll have issues if you don't have enough memory), but the per-object size.

I suggest you split it into smaller arrays, perhaps behind a facade of some description. I don't believe there's any way to workaround that limit for a single array.

EDIT: You could go for an array of arrays - aka a jagged array:

double[][] array = new double[12000][];
for (int i = 0; i < array.Length; i++)
{
    array[i] = new double[55000];
}

Would that be acceptable to you?

(You can't use a rectangular array (double[,]) as that would have the same per-object size problem.)

Since you can't create objects larger than 2GB you can try to use MemoryMappedFile to work with chunk of memory of the required size.


var data = MemoryMappedFile.CreateNew("big data", 12000L * 55000L);
var view = data.CreateViewAccessor();
var rnd = new Random();

for (var i = 0L; i < 12000L; ++i)
{
    for (var j = 0L; j < 55000L; ++j)
    {
        var input = rnd.NextDouble();
        view.Write<double>(i * 55000L + j, ref input);
    }
}

Providing that your total memory is sufficient, you can prevent Out of memory exceptions resulting from LOH fragmentation by creating a bunch of smaller arrays, and wrapping them in a single IList<T>, or some other indexed interface.

Here is a link which describes it:

BigArray<T>, getting around the 2GB array size limit

Credits: this post (C# chunked array).

OutOfMemoryException on declaration of Large Array

Related

Recent Posts