Navigation

    Fuze Arena Logo
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Popular
    • Users
    • Groups
    • Help
    • Discord

    Coding for Fun: Serialization

    Functions
    4
    10
    819
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Chronos
      Chronos last edited by Chronos

      While it looks like someone else already has a better virtual file system covered, this project has (de)serialization too. Use them together for maximum power.

      If you aren't familiar, serialization is when you convert variables to strings, and deserialization is when you convert the strings to variables again.

      This supports everything my improvised type function does (arrays (even arrays of arrays of arrays of strings of vectors), vectors, float, strings, int).

      f = open()
      // make read() happy without adding/overwriting content
      write("")
      close(f)
      
      function read_until(handle, stop_char)
          var out = ""
          var res = ""
          var num = ""
          while res != stop_char loop
              out += res
              res = read(handle, 1)
          repeat
          return out
      
      function read_char(f)
          return read(f,1)
      
      function index(name, names)
          // does the name appear in the array of names?
          int idx = -1
          for i=0 to len(names) loop
              if names[i] == name then
                  idx = i
                  break
              endif
          repeat
          return idx
      
      function load_files(names)
          var f = open()
          var files = []
          int countA = len(names)
          for i = 0 to countA loop
              files[i] = ""
          repeat
          int countB = 0
          //find the file
          // get the length of the name.
          var valid = read_char(f)
          while countA != countB and valid != "" loop
              // whether or not it is a valid %/!, read the content
              // to move the seek position
              var curr_name = read_next(f)[0]
              var content = read_next(f)[0]
      
              if valid == "%" then
                  var idx = index(curr_name, names)
                  if idx >= 0 then
                      files[idx] = content
                      countB+=1
                  endif
              endif
              // skip the content marker
              read_char(f)
              // get whether the next file is valid "%" or "!"
              valid = read_char(f)
          repeat
          close(f)
          return files
      
      function write_string(name, content)
          string s = "%"+len(name)+"|" + len(content)+"|"+content+"|"
          return s
      
      function delete_skip(f, names)
          // mark files for deletion skip to the end if the file list.
          var countA = len(names)
          var countB = 0
          var seek_pos = 0
          var d
          var curr_name
          var n1
          var content
          var n2
          var idx
          var valid = read_char(f)
          while countA != countB and valid != "" loop
              d = read_next(f)
              curr_name = d[0]
              n1 = d[1]
              if n1 == 0 then
                  break
              endif
              d = read_next(f)
              content = d[0]
              n2 = d[1]
              idx = index(curr_name, names)
              if valid == "%" and idx >= 0 then
                  seek(f, seek_pos)
                  write(f, "!")
                  countB += 1
              endif
      
              seek_pos += len(curr_name) + len(content) + 4 + len(str(n1)) + len(str(n2))
              seek(f, seek_pos)
              valid = read_char(f)
          repeat
          while valid == "%" or valid == "!" loop
              d = read_next(f)
              curr_name = d[0]
              n1 = d[1]
              d = read_next(f)
              content = d[0]
              n2 = d[1]
      
              seek_pos += len(curr_name) + len(content) + 4 + len(str(n1)) + len(str(n2))
              valid = read_char(f)
          repeat
          seek(f, seek_pos)
          return countB
      
      function delete(names)
          // mark files for deletion
          var f = open()
          var countA = len(names)
          var countB = 0
          var seek_pos = 0
          var d
          var curr_name
          var n1
          var content
          var n2
          var idx
          var valid = read_char(f)
          while countA != countB and valid != "" loop
              d = read_next(f)
              curr_name = d[0]
              n1 = d[1]
              if n1 == 0 then
                  break
              endif
              d = read_next(f)
              content = d[0]
              n2 = d[1]
              idx = index(curr_name, names)
              if valid == "%" and idx >= 0 then
                  seek(f, seek_pos)
                  write(f, "!")
                  countB += 1
              endif
      
              seek_pos += len(curr_name) + len(content) + 4 + len(str(n1)) + len(str(n2))
              seek(f, seek_pos)
              valid = read_char(f)
          repeat
          close(f)
          return countB
      
      function undelete(names)
          // mark files for deletion
          var f = open()
          var countA = len(names)
          var countB = 0
          var seek_pos = 0
          var d
          var curr_name
          var n1
          var content
          var n2
          var idx
          var valid = read_char(f)
          while countA != countB and valid != "" loop
              d = read_next(f)
              curr_name = d[0]
              n1 = d[1]
              if n1 == 0 then
                  break
              endif
              d = read_next(f)
              content = d[0]
              n2 = d[1]
              idx = index(curr_name, names)
              if valid == "!" and idx >= 0 then
                  seek(f, seek_pos)
                  write(f, "%")
                  countB += 1
              endif
      
              seek_pos += len(curr_name) + len(content) + 4 + len(str(n1)) + len(str(n2))
              seek(f, seek_pos)
              valid = read_char(f)
          repeat
          close(f)
          return countB
      
      function clear_file()
          // abuse the behavior of open() to clear the file
          var f = open()
          var f2 = open()
          close(f)
          close(f2)
          f = open()
          write(f, "")
          close(f)
          return void
      
      function cleanup()
          // permanently delete files
          var files = ""
          var f = open()
          var v = read_char(f)
          while v != "" loop
              n = read_next(f)[0]
              c = read_next(f)[0]
              if v == "%" then
                  // keep files that aren't marked for deletion
                  files += write_string(n,c)
              endif
              read_char(f)
              v = read_char(f)
          repeat
          close(f)
          clear_file()
          f = open()
          // rewrite the good files
          write(f, files)
          close(f)
          return void
      
      function write_files(names, contents)
          var f = open()
          delete_skip(f, names)
      
          for i=0 to len(names) loop
              var s = write_string(names[i], contents[i])
              write(f,s)
          repeat
          close(f)
          return void
      
      function ls()
          var f = open()
          var files = []
          var countB = 0
          var valid = read_char(f)
          while valid != "" loop
              var curr_name = read_next(f)[0]
              read_next(f)[0] // skip over content
              files[countB] = valid+curr_name
              countB += 1
              read_char(f)
              valid = read_char(f)
          repeat
          close(f)
          return files
      
      function type(content)
          string s = ""
          // everything (I can distinguish) can be safely string'ed
          // crashes on handles, sprites, etc
          string st = str(content)
          // check for an empty string
          if st == "" then
              s = "string"
          else
              // check for a vector or array based on the brackets
              //  make sure it isn't a string with brackets 
              //  (i.e., "{fakeout}") by seeing if the len() is 0 (a property of vectors)
              if st[0] == "{" and len(content) == 0 then
                  s = "vector"
              else
                  // if the bracket suggests it is an array,
                  //  verify that the len() for the string version 
                  //  is different than the content :
                  //  given arr = [0], len("[ 0 ]") != len(arr)
                  if st[0] == "[" and len(st) != len(content) then
                      s = "array"
                  else
                      // we established that it isn't an empty string, 
                      //  so see if it has any length because ints and floats do not.
                      if len(content) > 0 then
                          s = "string"
                      else
                          // the valuable decimal place will always appear for floats
                          if strContains(st, ".") then
                              s = "float"
                          else
                              if int(content) == content then
                                  s = "integer"
                              endif
                          endif
                      endif
                  endif
              endif
          endif
          return s
      string str_sep = "|"
      function serialize(content)
          string s = ""
          var t = type(content)
          if t == "array" then
              s += "arr["
              var l = len(content)
              for i=0 to l loop
                  s += serialize(content[i])
                  if i < l-1 then
                      s+=","
                  endif
              repeat
              s+="]"
          else
              if t == "vector" then
                  s+="vec{"
                  for i=0 to 4 loop
                      s+=str(content[i])
                      if i < 3 then
                          s+=","
                      endif
                  repeat
                  s+="}"
              else
                  // add the type and content
                  s += t[0]+t[1]+t[2]+":"
                  if t == "string" then
                      s+=str_sep+str(content)+str_sep
                  else
                      s+= str(content)
                  endif
              endif
          endif
          return s
      
      function substring(text, start, count)
          string s = ""
          for i=0 to count loop
              s+=text[start+i]
          repeat
          return s
      
      function substr(text, start, end)
          string s = ""
          for i=start to end loop
              s+=text[i]
          repeat
          return s
      
      function find_closing(text, item_idx)
          // strings cause [] and {} to be invalid until they are closed
          var openers = [str_sep, "[", "{"]
          var closers = ["}","]"]
          int locationA = item_idx
          int locationB = -1
          int quotes = false
          int layers = 0
          l = len(text)
          for i=locationA to l loop
              if quotes then
                  if text[i] == str_sep then
                      quotes = false
                      layers -= 1
                  endif
              else
                  if index(text[i], openers) >= 0 then
                      if text[i] == str_sep then
                          quotes = true
                      endif
                      layers += 1
                  else
                      if index(text[i], closers) >= 0 then
                          layers -= 1
                      endif
                  endif
              endif
              if layers == 0 then
                  locationB = i
                  break
              endif
          repeat
          arr = [locationA, locationB]
          return arr
      
      function deserialize(text)
          var value
          // arrays are recursive
      
          // get the current type
          string t = substring(text, 0,3)
          int l = len(text)
          if t == "arr" then
              value = []
              // cut off "arr["
              string stext = substr(text, 4,l)
              int counter = 0
              // make sure there are contents
              while len(stext) > 1 and stext[0] != "]" loop
                  var tmp = deserialize(stext)
                  var cut = len(serialize(tmp))
                  value[counter] = tmp
                  counter += 1
                  if stext[cut] == "," then
                      cut += 1
                  endif
                  stext = substr(stext, cut, len(stext))
              repeat
          else
              if t == "vec" then
                  // get 4 comma separated floats
                  value = {}
                  // skip "vec{"
                  int idx = 4
                  for i=0 to 4 loop
                      find = ","
                      if i == 3 then
                          find = "}"
                      endif
                      // find the end of the number
                      var count = strFind(substr(text, idx, l), find)
                      // save the float number
                      value[i] = float(substring(text, idx, count))
                      // move the index beyond the number.
                      idx += count+1
                  repeat
              else
                  // find the comma or end point to convert
                  if t == "str" then
                      // skip str:|
                      int idx = 5
                      // notice that there are two different substring functions here!
                      // one uses the ending index (substr) and one uses a count (substring)
                      int count = strFind(substr(text, idx, l), str_sep)
                      value = substring(text, idx, count)
                  else
                      int idx = 4
                      if t == "int" or t == "flo" then
                          int countC = len(text)-idx2
                          int countA = strFind(substr(text, idx, l), ",")
                          if countA == -1 then
                              countA = countC
                          endif
                          int countB = strFind(substr(text, idx, l), "]")
                          if countB == -1 then
                              countB = countC
                          endif
                          int stop = min(countC, min(countB, countA))
                          if t == "int" then
                              value = int(substring(text, idx, stop))
                          else
                              value = float(substring(text, idx, stop))
                          endif
                      endif
                  endif
              endif
          endif
          return value
      
      vector v
      v.z = 2.3
      v.x=2
      
      write_files(["Test_Data"], [serialize([451, [5, "arr[", 5445, "yay"], "5", v])])
      cleanup()
      
      loop
          clear()
          ls()
          print(serialize(v), "\n")
          print(deserialize(serialize(v)), "\n")
          print(deserialize(load_files("Test_Data")[0]), "\n")
          update()
      repeat
      

      The project is at XFXE2MNDN5, but it doesn't match the transcribed version here: it hasn't established the str_sep variable yet, and it uses a single quote as the separator, which I decided wasn't the best idea because apostrophes are used in regular language.

      Thanks for reading,
      Jason

      monkeee 1 Reply Last reply Reply Quote 4
      • vinicity
        vinicity F last edited by

        Thanks for this. It looks super useful!

        1 Reply Last reply Reply Quote 0
        • monkeee
          monkeee F @Chronos last edited by

          @chronos So if I want to store an array full of vectors would it be something like this:

          write_files( ["Block_Data"] , [serialize(positionInfo)] )
          

          And then to retrieve it would it be:

          positionInfo = deserialize(  load_files(["Block_Data"])[0]  )
          

          Also what does cleanup() do? I tried to include that but it keeps crashing my program.

          This is very cool project..😎

          Chronos 1 Reply Last reply Reply Quote 0
          • PickleCatStars
            PickleCatStars F last edited by

            So pianofire’s ’persistent data’ functions are also using serialisation/deserialisation to get the job done, I guess. I wonder what the differences are..

            Chronos 1 Reply Last reply Reply Quote 0
            • Chronos
              Chronos @monkeee last edited by

              @monkeee The cleanup function removes "deleted" files from the virtual filesystem. Files are marked for deletion if you call delete() or if you overwrite a file.

              If you have an error message from cleanup(), that may help me understand why it isn't working on your end.

              1 Reply Last reply Reply Quote 0
              • Chronos
                Chronos @PickleCatStars last edited by

                @toxibunny From what I can tell at a glance, pianofire's persistent data functions don't support arrays, vectors, or arrays of vectors.

                1 Reply Last reply Reply Quote 1
                • PickleCatStars
                  PickleCatStars F last edited by

                  It does support vectors. Not arrays though.

                  1 Reply Last reply Reply Quote 1
                  • Chronos
                    Chronos last edited by

                    Looks like something broke in 3.0. It can't empty the real file anymore with clear_file() (no idea how it is intended to be emptied or deleted).

                    Probably unrelated to 3.0, but if "i" is globally defined elsewhere, then all of the functions that use i will need to have explicit local definitions (int i) for the "for" loops to work correctly.

                    1 Reply Last reply Reply Quote 2
                    • Chronos
                      Chronos last edited by

                      I updated the file system project to use what I used to save the Space Swarm data. (fixed the int i problems, and fixed the "hanging" cleanup() problem that I assume @monkee saw)

                      1 Reply Last reply Reply Quote 1
                      • Chronos
                        Chronos last edited by

                        As kind of an extension of this, I made a system for serializing special sprites. They are just sprites, but with just a little extra handling.
                        They can't also have sprites as attributes though.
                        ID NX2D62FDN5

                        1 Reply Last reply Reply Quote 1
                        • First post
                          Last post