TextureMind Framework – Insights #5 – Serialization in TBF

TBF stands for TextureMind Binary Format and it's the format used to serialize binary data in the TextureMind Framework, as alternative to xml and json. TextureMind Framework has an entire serialization system with an abstraction layer that can be specialized with different kind of implementations, in human-readable text format or binary format. The built-in formats are:

  • xml
  • json
  • raw
  • tbf

All these formats are wrapped by the same abstract interface inside the framework. You can serialize a large set of types:

  • Primitive types: bool_t, uint8_t, int8_t, uint16_t, int16_t, uint32_t, int32_t, uint64_t, int64_t, float, double
  • Strings of any size
  • Data structures of any kind
  • TextureMind Objects
  • Containers of objects

You can serialize the content of single variables or multiple variables with a single call. You can save data in human readable text formats like xml or json, or in binary format to save CPU and space. All the formats implement principles of backward / forwards compatibility, except for raw data. With raw, you need to make sure that the writer and the reader share the same version number. Raw is very fast and keeps the serialization of complex data structures, like the objects or container of objects, without the burden of being compact or compatible between different versions. It has been programmed to guarantee a way for high performance client / server transmission of very large amount of data. All the formats are byte order agnostic and the byte order of reference used in binary formats is little endian (on big endian machines, values are converted into little endian, even if there is still a way to bypass the convertion for performance reasons).

TextureMind Binary Format (TBF) has been designed to guarantee a compact and fast binary format with strong principles of backward and forward compatibility, like Google Protobuf. It has been used for client / server transmission of messages and all the native formats in the TextureMind Framework for saving images, sounds, gui pages, 2D and 3D models. It can be used to serialize a large amount of data, virtually with the size limit of a 64 bit value, so you can save 3D models with billion of polygons and vertices. It's optimized for being fast and compact: integer values are stored in base128 and the fields are not tagged for every recurrence inside the data structure: they are deserialized thanks to a special system based on reading keys. You don't need to keep numeric tags like in protobuf, but just the variable names. Unlike protobuf, you don't need to build source code for every data structure in your project. You can also compress data with ZIP or LZ4 inside a serialized structure, or transport data files in native format without changing the original content (for example, you can transport an image in jpeg or a music in protracker format).

Leave a Reply