A Persistent File System

PickleCatStars

OMG I’ve been waiting for this!

Mechanical

This seems to be a useful thing, didn't quite understand what it does, but what I think this will be used to "wallpaper" textures on 3d models? Since we don't have a model editor yet?

pianofire

@Mechanical Specifically this allows you to store multiple files in a virtual file system (as Fuze currently only supports one)

PickleCatStars

i am hoping to use it to store attributes of 3d models I'm using to furnish my landscape. I'd like to add multiple save slots too, so hopefully this'll make things easier.

Gothon

@Mechanical As pianofire said, this is a virtual file system, that can be used to save whatever data you want. Ex. save games, level maps, high scores, etc. I am planning on using it myself for saving "wallpaper" textures on 3D models as you say. That is a separate project of mine at the moment.

I am currently working on an update for this which includes:

A fix for a bug where the UI doesn't test for the max filename length (126 bytes). You might corrupt the filesystem if you make a filename too long.
Improved UI that lets you quickly scroll more files than fit vertically on the screen.
Improved link and unlink which will keep files in directory entrys compact and in a more predictable order for large directories with more than 1 block of filenames.
Various spelling errors and typos in the comments

I hope to resubmit with these changes soon.

spikey

@Gothon I looked at the code, impressiv, is this the implementation analog to the FAT system on a disk? But how do you store "wallpaper" textures: do you store a lot of data points for texture corners or do you store the images used for the textures itself? Did I miss a command that can save an image? or extract binary data out of an image handle? Sorry, for so many questions, but thats very fascinating. I don't expect an essay, just a few words, if you have a moment. Or additional links: I will have a look at your first Game Jam entry, maybe I should start there. ;-)

Gothon

@spikey

is this the implementation analog to the FAT system on a disk?

Yes it works similar to DOS in the way it uses a FAT to track the use of data blocks.

But how do you store "wallpaper" textures: do you store a lot of data points for texture corners or do you store the images used for the textures itself?

I am storing the images themselves encoded as a string representing each pixel as 1 ANSI character used as an index to a 126 color palette. Currently the images are all 256x256, however I have managed to compress the string using run length encoding (RLE) so I can represent less detailed images compactly.

Did I miss a command that can save an image? or extract binary data out of an image handle?

You did not. Fuze currently has no way of reading pixel data from an image buffer. What I have in my other project allows you to draw the images in regular memory as an array of integers. For memory efficiency I pack 8 pixels of color indexes into a single (64bit) integer which seems to work quite well in the program. After an image has been drawn I can then convert this array into a string for more memory efficiency and saving.

spikey

@Gothon "draw the images in regular memory as an array of integers": are you referring to uploadImage()? So, you manually add ~256x256=65'536 / 2 (maybe as an average good encoding rate) = 30'000 characters by typing them into your code? and convert them to vectors (colors) to generate the image? - I think I am still missing a point. Recently I had a quick thought about converting a bitmap (as an easy format) to a vector array: to convert something like this to pass it to uploadImage() .
But that ends up in a lot of typing. I checked your FUZEGJ0103 jump project, but there you use the decent imagebuffer drawQuad() and to draw you use shapes. Ok, I feel like I am pestering you with questions - I will just follow your work with great interest. Congratulations, for that FAT system.

Gothon

@spikey No, I am not using uploadImage(). What I have is a prototype interactive drawing program so there isn't a need to have a big chunck of data in the source code.

This is simplified but it works something like this:

Function CPU_Plot(ImArr, Pos, ColIdx)
    int Idx = 256 * Pos.y + Pos.x
    int ByteIdx = Idx % 8
    Idx = Int(Idx / 8)
    ImArr[Idx] = bitFieldInsert(ImArr[Idx], 8 * ByteIdx, 7, ColIdx)
Return VOID

var Palette = [ white, platinum, lightGrey, silver ]

int ImData[8 * 1024]
var ImHandle =  CreateImage(256, 256, 0, Image_RGBA)

// Initialize with an 8x8 checkerboard:
SetDrawTarget(ImHandle)
For Y = 0 To 8 Loop
    For X = 0 To 8 Loop
        int C = 2 + ((X + Y) % 2)
        CPU_Plot(ImData, {X, Y}, C)
        Plot(X + 1, Y + 1, Palatte[C])
    Repeat
Repeat

Loop
    SetDrawTarget(FrameBuffer)
    Clear(SkyBlue)
    DrawImage(ImHandle, 100, 100, 2)
    Update()
    // Do Input Handling here
Repeat

It keeps the data both in main memory and video memory and keeps them synchronized throughout the program so no explicit uploading is needed. Although I do have a function to copy the whole array to the image buffer. I use it occasionally for debugging.

spikey

@Gothon Thank you for taking your time to share this 👍. I will definitively need some daylight and a tee to absorb this properly.

Gothon

I have re-shared an update. The download code is the same.

The file system structure is also the same, so if you have pre-existing files, the new code should load them. However you must be careful not to overwrite the project with the new one as that would erase the files and instead only copy-paste the source code.

This update contains the previously mentioned improvements and a new file/folder copy/paste feature.

Gothon

@Gothon said in A Persistent File System:

there could be bugs I haven't found. I haven't even tested the File Table when it needs more than 1 block (at around 64kb file system size).

As it turns out, there was a bug with the FAT when the File System reached 64kb size. I have fixed the bug and resubmitted. The fix involved moving the position of each FAT record by 1 position so previously made File Systems will not load correctly with the new code.

I now also have the file system code working properly with my 'Image Editor' which is part of my ongoing textured 3D model editor project. I have decided that the file system is now stable enough and the image editor is complete enough to share publicly as-well. This project has a complete touchscreen enabled interface and reflects my more recent work improving the interface to the file system.

The share code for the "Image File Editor Preview" is: 7VK73CYH9L (live)

PB____

This does a lot of things really well. Clearly a lot of work has gone into this, and the result is impressive! :)

vinicity

@Gothon said in A Persistent File System:

Share Code: N5XQ3RND9L (live)

This is quite amazing. Any chance that you will be adding a text editor as well?

Gothon

@vinicity Sorry, I am not planning on making a text editor at the moment. It would certainly make the demo much better. Although the text entry box produced by the input() command is almost a text editor by itself. If it were to just take a parameter for initial contents of the box, I could load the file contents into it and it would really seem like one. (especially if the noise it now makes in v2.15.0 is also fixed)

Also, I have re-shared the two programs in this thread with fixes for Fuze v2.15.0. Mostly it was just a lot of places where Ref was added to function parameters.

If you have a prior file-system you want to retain, be careful not to overwrite the old project with the new one, and instead only copy the source code from the new project over to the code of the old project. That way you can get the new code without overwriting the persistent file.

tony7059

Thanks for all your hard work! I've not programmed at that level in a long time. I can remember a book I had for C language that was dedicated to coding a GUI system... windows, menu bar, dropdown menus with hotkey options. So, you not only had the code to process these GUI components, you of course had to develop the API to manipulate things as a consumer of those services! Back in the day, memory was so cheap, and I recall there being some kind of technique that someone had used to store a multitude of texts for a relatively large program in its day. I think it had to do with base 64 and you got me thinking about this lower level stuff now, so now I have to go see if I can find information on it : )

Gothon

I have updated these programs today. Most of the changes minor, however there are a couple of bug-fixes, including a fix for a serious bug discovered by @joyrider3774

Missing ref bug

Affects the first program "File System Demo" N5XQ3RND9L not the second one "Image Editor Preview".
Can cause file-system corruption if you make a file or a directory larger than one block (512 bytes) in size. Once corrupted a program may crash on startup while loading the file-system. This can be fixed by formatting the system with the FS_Format() function, however all data will be lost.
Caused by the keyword ref missing from the first argument of the function FS_SetFAT(Ref FS, H, B, Next) and can be easily fixed by manually adding the ref to that function.

Floating point division inaccuracy.

Affects both programs
Potentially cause problems when large values are present such as file sizes as small as about 10MB (but still smaller than the design maximum of ~248MB)
Fixed by replacing inaccurate floating point division with accurate integer division or equivalent bit shift operations throughout the program.

monkeee

@gothon The ID codes listed here don't work anymore. It says project not found when I enter the code

Gothon

@monkeee It seems that my shared programs have switched to 'private' status and changed codes. I will resubmit the programs here, so people can get them after they have been approved again.

The new ID codes are:
"File System Demo" - EPK73MQN7L
"Image File Editor Preview" - D5WT6MND9L
(live)

HeyoMayo

great! now make a operating system (sarcasm), this is really impressive and WOW you made a image editor?!