Currently I'm working at the development of the TMD (TextureMind Desktop) remoting protocol. I want to reach a point where it will be possible to connect to an host machine as soon as possible. I took the chance to improve the framework architecture implementing an entire plugin system. Now it's possible to add functionalities through external modules. I added YUV conversion with I420, YV12, NV12 and NV21 pixel formats. I improved the image format so now it's possible to serialize multi-planes YUV images. I made all the rgba pixel formats endianness agnostic. I wrote an abstraction layer for video compression and I'm writing a plugin module with ffmpeg libraries to support h.264 and av1 compression. I already implemented the transport layer with support for TCP where it's possible to send messages between processes through the network with my serialization system. Now I have to implement the communication channels for the remote session, in particular display and input channels. I designed the entire architecture, where I think I made lot of improvements compared to classic remote protocols. There will be an abstraction layer to capture or control not only the desktop but also applications. Imagine to write an application for a virtual museum and you want to have multiple users connected into it: you can deliver the application in the form of SaaS. Then a channel may have two modes of communication: per client or broadcast. Just to make a valid example, the display channel is typically per client and the input channel is broadcast, but you can also extend the display channel for streaming in broadcast. Finally, I designed a smart adaptive frame rate control to avoid jittering. The first version of TMD will support only software video encoding / decoding, but future versions will be optimized for supporting the GPU, in particular NVIDIA with NvEnc and AMD with Amf.

Post link