Out of memory despite never having much allocated at any one time
Look at this cute, innocent function:
function crash(outerLimit, innerLimit) var arr =  for i = 0 to outerLimit loop arr =  for j = 0 to innerLimit loop arr[j] = 0 repeat repeat return void
(Real code would fill the array with more interesting data and do something with the filled array after each iteration.)
It only ever has one array, maximum size 'innerLimit'. Harmless! However,
crashes with an out of memory error. As do crash(80, 10000) and crash(8000, 100). Interestingly
crash(600, 1000) crash(600, 1000)
(and as many repetitions as I cared to test) is fine, so whatever is happening, it is not a real leak; once the crash() function is done, it seems to be cleaning up properly. Also fine: crash(6000, 100) and crash(60, 10000). The product of the two arguments in each case, crashing or non-crashing, is suspiciously close to the maximum size of a single array you can use.
Workaround: Instead of filling the outer scope array, first fill up a new inner scope array, then assign to the outer scope at the end. This is fine:
function noCrash(outerLimit, innerLimit) var arr =  for i = 0 to outerLimit loop var arr2 =  for j = 0 to innerLimit loop arr2[j] = 0 repeat arr = arr2 repeat return void
I have no clue why that should be better. Maybe it avoids memory fragmentation?
Also fine are variations where you stuff the array into a struct and either do the clearing or filling in a separate function taking that struct by reference.
In other words, or a peace of an explanation: Looks like
arr=allocates new memory in your first example. And it does not clean up the deprecated
arrfrom the previous for loop iteration, before it gets out of the for loop. At its way to 800'000 elements it cannot allocate new memory. Until 600'000 elements it still works.
In your second example
arr2can be cleaned in every iteration, because there is no scope relation outside of the loop. And
arr=arr2does not allocate new memory. Only the last
arr2values are kept.
Anyone with another idea?
...or maybe we should not use var anymore since the last release for arrays, just for fuze specific types like shapes, sprites, files or handle types in general? (Sorry, for the duplicated posts🙄)
OK I will flag for investigation
spikey: 'var' is not the issue. This also crashes:
function crash(outerLimit, innerLimit) int arr for i = 0 to outerLimit loop int arr2[innerLimit] arr = arr2 for j = 0 to innerLimit loop arr[j] = 0 repeat repeat return void
which doesn't even use the autogrow feature of array element assignment in the inner loop. The different scopes seem to be essential, the 'arr = ' assignment in the outer loop before the inner loop, and the inner loop doing assignments.
@Z-Mann cool, thanks for checking.
The product of the two arguments in each case, crashing or non-crashing, is suspiciously close to the maximum size of a single array you can use.
You're falling into a sweet spot (or not-so-sweet spot) where more than 90% of the for loop context's memory is filled, and it tries to "garbage collect" it. If you're under that, it'd work, and if you're over that, it'd be an error. There seem to be at least a couple problems with this GC process on our end. I'll keep you posted. I would also recommend restarting F4NS after running this code as it has the potential to cause other issues.
Ah, sorry, I wasn't clear. When I said 'crash', I always meant: abort with "Stack overflow. Too many functions called, maximum memory for variables exceeded." error. Though I sometimes managed to fully make F4NS itself crash and terminate to the system UI with some other combinations of parameters and repeated runs. That may be the sour spot you describe. Not very reproducible, I'm afraid.
I didn't know F4NS used garbage collection. Could it simply be that for this particular crazy allocation and command pattern, the heuristics that tell it when to run fail and would only run it when it is too late?
Another variation, no assignment to the array elements required, the inner loop can be very short and empty (still needs to be a loop):
function crash(outerLimit, innerLimit) int arr for i = 0 to outerLimit loop int arr2[innerLimit] arr = arr2 for j = 0 to 1 loop repeat repeat return void
This one runs out of memory for me only with crash(880, 1000).
for l = 0 to 1000 loop crash(4, 114681) repeat
, the innerLimit parameter being the highest one where a single run does not lead to an out of memory error here, reliably kills off F4NS properly for me after about four seconds.
for l = 0 to 1000 loop crash(4, 100000) repeat
on the other hand happily runs as long as I'm willing to give it, supporting your sour spot hypothesis.
I didn't know F4NS used garbage collection.
Well, I put it in air quotes for a reason! It's not true garbage collection, but something much simpler in that same realm of cleaning up unused data. I'm unsure at the moment what causes the hard crash, but I have been able to reproduce that on my end. I think we're going to have to run some more tests before we can make a decision on how best to proceed here.
@Willpowered The thing about the GC and the loop context's memory is interresting. I have a bug that look like this
var balls // 150 "Ball" struct containing a sprite for i = 0 to 150 loop createBall(balls[i]) // call createSprite and initialize the sprite repeat loop clear() for i = 0 to 150 loop updateSprite(balls[i].sprite) drawSprite(balls[i].sprite) repeat update() repeat for i = 0 to 150 loop deleteBall(balls[i]) // call removeSprite but give an error if the loop above run for too long (Variable does not exists: balls) repeat
You can reproduce by running my program Q5D63MNDHY. just run it and let the title screen for 5 minutes. The title screen feature 3 layers of "stars", when the smallests stars start flickering / disappearing, or a horizontal grey line suddently appear and disappear, it mean the bug will happen. The code start to be a little big but if you look inside hopefully the error will guide you if you need some sample to reproduce.
@Z-Mann I can finally confirm that a fix for this issue is complete and will be delivered with the next patch.
@romain337 The program you shared runs successfully on my end, and it should work for you in the next patch too.