Coding for Fun: Serialization
-
While it looks like someone else already has a better virtual file system covered, this project has (de)serialization too. Use them together for maximum power.
If you aren't familiar, serialization is when you convert variables to strings, and deserialization is when you convert the strings to variables again.
This supports everything my improvised type function does (arrays (even arrays of arrays of arrays of strings of vectors), vectors, float, strings, int).
f = open() // make read() happy without adding/overwriting content write("") close(f) function read_until(handle, stop_char) var out = "" var res = "" var num = "" while res != stop_char loop out += res res = read(handle, 1) repeat return out function read_char(f) return read(f,1) function index(name, names) // does the name appear in the array of names? int idx = -1 for i=0 to len(names) loop if names[i] == name then idx = i break endif repeat return idx function load_files(names) var f = open() var files = [] int countA = len(names) for i = 0 to countA loop files[i] = "" repeat int countB = 0 //find the file // get the length of the name. var valid = read_char(f) while countA != countB and valid != "" loop // whether or not it is a valid %/!, read the content // to move the seek position var curr_name = read_next(f)[0] var content = read_next(f)[0] if valid == "%" then var idx = index(curr_name, names) if idx >= 0 then files[idx] = content countB+=1 endif endif // skip the content marker read_char(f) // get whether the next file is valid "%" or "!" valid = read_char(f) repeat close(f) return files function write_string(name, content) string s = "%"+len(name)+"|" + len(content)+"|"+content+"|" return s function delete_skip(f, names) // mark files for deletion skip to the end if the file list. var countA = len(names) var countB = 0 var seek_pos = 0 var d var curr_name var n1 var content var n2 var idx var valid = read_char(f) while countA != countB and valid != "" loop d = read_next(f) curr_name = d[0] n1 = d[1] if n1 == 0 then break endif d = read_next(f) content = d[0] n2 = d[1] idx = index(curr_name, names) if valid == "%" and idx >= 0 then seek(f, seek_pos) write(f, "!") countB += 1 endif seek_pos += len(curr_name) + len(content) + 4 + len(str(n1)) + len(str(n2)) seek(f, seek_pos) valid = read_char(f) repeat while valid == "%" or valid == "!" loop d = read_next(f) curr_name = d[0] n1 = d[1] d = read_next(f) content = d[0] n2 = d[1] seek_pos += len(curr_name) + len(content) + 4 + len(str(n1)) + len(str(n2)) valid = read_char(f) repeat seek(f, seek_pos) return countB function delete(names) // mark files for deletion var f = open() var countA = len(names) var countB = 0 var seek_pos = 0 var d var curr_name var n1 var content var n2 var idx var valid = read_char(f) while countA != countB and valid != "" loop d = read_next(f) curr_name = d[0] n1 = d[1] if n1 == 0 then break endif d = read_next(f) content = d[0] n2 = d[1] idx = index(curr_name, names) if valid == "%" and idx >= 0 then seek(f, seek_pos) write(f, "!") countB += 1 endif seek_pos += len(curr_name) + len(content) + 4 + len(str(n1)) + len(str(n2)) seek(f, seek_pos) valid = read_char(f) repeat close(f) return countB function undelete(names) // mark files for deletion var f = open() var countA = len(names) var countB = 0 var seek_pos = 0 var d var curr_name var n1 var content var n2 var idx var valid = read_char(f) while countA != countB and valid != "" loop d = read_next(f) curr_name = d[0] n1 = d[1] if n1 == 0 then break endif d = read_next(f) content = d[0] n2 = d[1] idx = index(curr_name, names) if valid == "!" and idx >= 0 then seek(f, seek_pos) write(f, "%") countB += 1 endif seek_pos += len(curr_name) + len(content) + 4 + len(str(n1)) + len(str(n2)) seek(f, seek_pos) valid = read_char(f) repeat close(f) return countB function clear_file() // abuse the behavior of open() to clear the file var f = open() var f2 = open() close(f) close(f2) f = open() write(f, "") close(f) return void function cleanup() // permanently delete files var files = "" var f = open() var v = read_char(f) while v != "" loop n = read_next(f)[0] c = read_next(f)[0] if v == "%" then // keep files that aren't marked for deletion files += write_string(n,c) endif read_char(f) v = read_char(f) repeat close(f) clear_file() f = open() // rewrite the good files write(f, files) close(f) return void function write_files(names, contents) var f = open() delete_skip(f, names) for i=0 to len(names) loop var s = write_string(names[i], contents[i]) write(f,s) repeat close(f) return void function ls() var f = open() var files = [] var countB = 0 var valid = read_char(f) while valid != "" loop var curr_name = read_next(f)[0] read_next(f)[0] // skip over content files[countB] = valid+curr_name countB += 1 read_char(f) valid = read_char(f) repeat close(f) return files function type(content) string s = "" // everything (I can distinguish) can be safely string'ed // crashes on handles, sprites, etc string st = str(content) // check for an empty string if st == "" then s = "string" else // check for a vector or array based on the brackets // make sure it isn't a string with brackets // (i.e., "{fakeout}") by seeing if the len() is 0 (a property of vectors) if st[0] == "{" and len(content) == 0 then s = "vector" else // if the bracket suggests it is an array, // verify that the len() for the string version // is different than the content : // given arr = [0], len("[ 0 ]") != len(arr) if st[0] == "[" and len(st) != len(content) then s = "array" else // we established that it isn't an empty string, // so see if it has any length because ints and floats do not. if len(content) > 0 then s = "string" else // the valuable decimal place will always appear for floats if strContains(st, ".") then s = "float" else if int(content) == content then s = "integer" endif endif endif endif endif endif return s string str_sep = "|" function serialize(content) string s = "" var t = type(content) if t == "array" then s += "arr[" var l = len(content) for i=0 to l loop s += serialize(content[i]) if i < l-1 then s+="," endif repeat s+="]" else if t == "vector" then s+="vec{" for i=0 to 4 loop s+=str(content[i]) if i < 3 then s+="," endif repeat s+="}" else // add the type and content s += t[0]+t[1]+t[2]+":" if t == "string" then s+=str_sep+str(content)+str_sep else s+= str(content) endif endif endif return s function substring(text, start, count) string s = "" for i=0 to count loop s+=text[start+i] repeat return s function substr(text, start, end) string s = "" for i=start to end loop s+=text[i] repeat return s function find_closing(text, item_idx) // strings cause [] and {} to be invalid until they are closed var openers = [str_sep, "[", "{"] var closers = ["}","]"] int locationA = item_idx int locationB = -1 int quotes = false int layers = 0 l = len(text) for i=locationA to l loop if quotes then if text[i] == str_sep then quotes = false layers -= 1 endif else if index(text[i], openers) >= 0 then if text[i] == str_sep then quotes = true endif layers += 1 else if index(text[i], closers) >= 0 then layers -= 1 endif endif endif if layers == 0 then locationB = i break endif repeat arr = [locationA, locationB] return arr function deserialize(text) var value // arrays are recursive // get the current type string t = substring(text, 0,3) int l = len(text) if t == "arr" then value = [] // cut off "arr[" string stext = substr(text, 4,l) int counter = 0 // make sure there are contents while len(stext) > 1 and stext[0] != "]" loop var tmp = deserialize(stext) var cut = len(serialize(tmp)) value[counter] = tmp counter += 1 if stext[cut] == "," then cut += 1 endif stext = substr(stext, cut, len(stext)) repeat else if t == "vec" then // get 4 comma separated floats value = {} // skip "vec{" int idx = 4 for i=0 to 4 loop find = "," if i == 3 then find = "}" endif // find the end of the number var count = strFind(substr(text, idx, l), find) // save the float number value[i] = float(substring(text, idx, count)) // move the index beyond the number. idx += count+1 repeat else // find the comma or end point to convert if t == "str" then // skip str:| int idx = 5 // notice that there are two different substring functions here! // one uses the ending index (substr) and one uses a count (substring) int count = strFind(substr(text, idx, l), str_sep) value = substring(text, idx, count) else int idx = 4 if t == "int" or t == "flo" then int countC = len(text)-idx2 int countA = strFind(substr(text, idx, l), ",") if countA == -1 then countA = countC endif int countB = strFind(substr(text, idx, l), "]") if countB == -1 then countB = countC endif int stop = min(countC, min(countB, countA)) if t == "int" then value = int(substring(text, idx, stop)) else value = float(substring(text, idx, stop)) endif endif endif endif endif return value vector v v.z = 2.3 v.x=2 write_files(["Test_Data"], [serialize([451, [5, "arr[", 5445, "yay"], "5", v])]) cleanup() loop clear() ls() print(serialize(v), "\n") print(deserialize(serialize(v)), "\n") print(deserialize(load_files("Test_Data")[0]), "\n") update() repeatThe project is at XFXE2MNDN5, but it doesn't match the transcribed version here: it hasn't established the str_sep variable yet, and it uses a single quote as the separator, which I decided wasn't the best idea because apostrophes are used in regular language.
Thanks for reading,
Jason -
Thanks for this. It looks super useful!
-
@chronos So if I want to store an array full of vectors would it be something like this:
write_files( ["Block_Data"] , [serialize(positionInfo)] )And then to retrieve it would it be:
positionInfo = deserialize( load_files(["Block_Data"])[0] )Also what does cleanup() do? I tried to include that but it keeps crashing my program.
This is very cool project..😎
-
So pianofire’s ’persistent data’ functions are also using serialisation/deserialisation to get the job done, I guess. I wonder what the differences are..
-
@monkeee The cleanup function removes "deleted" files from the virtual filesystem. Files are marked for deletion if you call delete() or if you overwrite a file.
If you have an error message from cleanup(), that may help me understand why it isn't working on your end.
-
@toxibunny From what I can tell at a glance, pianofire's persistent data functions don't support arrays, vectors, or arrays of vectors.
-
It does support vectors. Not arrays though.
-
Looks like something broke in 3.0. It can't empty the real file anymore with clear_file() (no idea how it is intended to be emptied or deleted).
Probably unrelated to 3.0, but if "i" is globally defined elsewhere, then all of the functions that use i will need to have explicit local definitions (int i) for the "for" loops to work correctly.
-
I updated the file system project to use what I used to save the Space Swarm data. (fixed the int i problems, and fixed the "hanging" cleanup() problem that I assume @monkee saw)
-
As kind of an extension of this, I made a system for serializing special sprites. They are just sprites, but with just a little extra handling.
They can't also have sprites as attributes though.
ID NX2D62FDN5
