Teeworlds 0.7
home overview fundamentals packet_layout traffic map_download snap_items system_messages game_messages ctrl_messages

Fundamentals

There are a few basic concepts that you need to understand to fully grasp the whole network protocol from scratch.

Tokens

Every game related packet (excluding server info for now) has a packet header with a token field. From network.h: 

                
    packet header: 7 bytes (9 bytes for connless)
        unsigned char flags_ack;    // 6bit flags, 2bit ack
        unsigned char ack;          // 8bit ack
        unsigned char numchunks;    // 8bit chunks
        unsigned char token[4];     // 32bit token
        // ffffffaa
        // aaaaaaaa
        // NNNNNNNN
        // TTTTTTTT
        // TTTTTTTT
        // TTTTTTTT
        // TTTTTTTT

    packet header (CONNLESS):
        unsigned char flag_version;				// 6bit flags, 2bits version
        unsigned char token[4];					// 32bit token
        unsigned char responsetoken[4];			// 32bit response token

        // ffffffvv
        // TTTTTTTT
        // TTTTTTTT
        // TTTTTTTT
        // TTTTTTTT
        // RRRRRRRR
        // RRRRRRRR
        // RRRRRRRR
        // RRRRRRRR

    if the token isn't explicitly set by any means, it must be set to
    0xffffffff
The token field contains a token sent by the receiving party; when the client sends a packet to the server, the token field contains the token previously sent by the server and vice versa.
The client and server each generate a random token and then exchange it.

If the token is not known yet, the token field will be FF FF FF FF indicating it being empty. This happens only in the very first packet the client sends to initiate the connection.

If the client or server gets a packet with an unexpected token it will simply ignore it without any error message in the logs. If you are trying to reimplement the protocol, this is one of the first things you need to get right before you can properly send valid packets.

Quick debug tip if you suspect token bugs in your implementation: comment out or add a dbg_msg in this line where packets with wrong tokens are silently dropped.

Sequence numbers

Every chunk header can contain a vital flag (MSGFLAG_VITAL). Note that one Teeworlds packet can contain multiple chunks. If said vital flag is set to true, the receiving party has to increment a counter. This counter is called the sequence or acknowledge number. The client and server will send the amount of vital chunks they received as the sequence number in the packet header.

The CNetConnection class keeps track of these sequence numbers. 

                
    unsigned short m_Sequence;
    unsigned short m_Ack;
    unsigned short m_PeerAck;
Using these three variables, the client and server keep track of: Beware that both client and server silently drop all packets with wrong sequence numbers set in the packet header.
Quick debug tip if you suspect sequence number bugs in your implementation: comment out or add dbg_msgs in these lines.

Message queueing, chunks and flushing

Control messages are sent immediately. All control messages go through the SendControlMsg() method which calls SendPacket() directly. In SendPacket() the Teeworlds packet header gets added and the packet is sent over the network. Note that control messages always have 0 chunks because of the Construct.m_NumChunks = 0; line in SendControlMsg(): 
            
    void CNetBase::SendControlMsg(const NETADDR *pAddr, TOKEN Token, int Ack, int ControlMsg, const void *pExtra, int ExtraSize)
    {
        CNetPacketConstruct Construct;
        Construct.m_Token = Token;
        Construct.m_Flags = NET_PACKETFLAG_CONTROL;
        Construct.m_Ack = Ack;
        Construct.m_NumChunks = 0;
        Construct.m_DataSize = 1+ExtraSize;
        Construct.m_aChunkData[0] = ControlMsg;
        if(ExtraSize > 0)
            mem_copy(&Construct.m_aChunkData[1], pExtra, ExtraSize);

        // send the control message
        SendPacket(pAddr, &Construct);
    }
However, game messages and system messages usually go through SendMsg() -> m_NetClient.Send(&Packet); where they end up here: 
            
    if(pChunk->m_Flags&NETSENDFLAG_CONNLESS)
    {
        // [..]
    }
    else
    {
        // [..]
        m_Connection.QueueChunk(Flags, pChunk->m_DataSize, pChunk->m_pData);
    }
As the name suggests, m_Connection.QueueChunk() only puts these messages into a queue of so-called chunks; no packet header is created and no data is being sent over the network yet. For now the messages only get collected in a queue. This queue is then sent as soon as it is full or a message with the flag MSGFLAG_FLUSH is sent. When MSGFLAG_FLUSH is sent, all messages in the queue get their own chunk header and in the very beginning of the UDP packet, the packet header includes one field stating the amount of chunks in the packet.

This is probably done for performance reasons: to avoid wasting bandwidth on Teeworlds/UDP/IP headers when non time critical messages are being sent that are followed by other messages anyways; messages are bundled up and sent together.

If you are calling SendMsg() and care about the time delay, make sure to use the MSGFLAG_FLUSH flag otherwise it might be slightly delayed. The delay is not very dramatic since flushes happen quite frequently so it should only be used for super time sensitive things such as user inputs or tee positions.

Let's illustrate chunks and flushing via a real world example. When a client connects to a server, the first packet received with multiple chunks is Motd, Server Settings, Ready. This happens when OnClientConnected() is called, which is the function where the MOTD and server settings are sent. 
            
    void CGameContext::OnClientConnected(int ClientID, bool Dummy, bool AsSpec)
    {
        // [..]

        // send motd
        SendMotd(ClientID);

        // send settings
        SendSettings(ClientID);
    }
            
        
            
    void CGameContext::SendMotd(int ClientID)
    {
        CNetMsg_Sv_Motd Msg;
        Msg.m_pMessage = Config()->m_SvMotd;
        Server()->SendPackMsg(&Msg, MSGFLAG_VITAL, ClientID);
    }

    void CGameContext::SendSettings(int ClientID)
    {
        CNetMsg_Sv_ServerSettings Msg;
        Msg.m_KickVote = Config()->m_SvVoteKick;
        Msg.m_KickMin = Config()->m_SvVoteKickMin;
        Msg.m_SpecVote = Config()->m_SvVoteSpectate;
        Msg.m_TeamLock = m_LockTeams != 0;
        Msg.m_TeamBalance = Config()->m_SvTeambalanceTime != 0;
        Msg.m_PlayerSlots = Config()->m_SvPlayerSlots;
        Server()->SendPackMsg(&Msg, MSGFLAG_VITAL, ClientID);
    }
Note that the call to SendPackMsg() contains the flag MSGFLAG_VITAL, not MSGFLAG_FLUSH so they are just added to the queue rather than being sent right away. Directly after OnClientConnected() is called, SendConnectionReady() gets called. 
            
    GameServer()->OnClientConnected(ClientID, ConnectAsSpec);
    SendConnectionReady(ClientID);
            
        
            
    void CServer::SendConnectionReady(int ClientID)
    {
        CMsgPacker Msg(NETMSG_CON_READY, true);
        SendMsg(&Msg, MSGFLAG_VITAL|MSGFLAG_FLUSH, ClientID);
    }
This time the flag MSGFLAG_FLUSH is used which causes the whole queue (currently MOTD, Settings) to be sent, including the newly added CON_READY. Basically we're queueing 3 messages and sending them together, but this is all happening in the same tick so it feels instant.

So these three pseudocode method calls: 
            
    Send(CNetMsg_Sv_Motd, MSGFLAG_VITAL);                   // nothing is sent
    Send(CNetMsg_Sv_ServerSettings, MSGFLAG_VITAL);         // nothing is sent
    Send(NETMSG_CON_READY, MSGFLAG_VITAL|MSGFLAG_FLUSH);    // all 3 are sent
cause one UDP packet to be sent that looks like this: 
            
    +-------------------------+---------------+------+---------------+-----------------+---------------+-----------+
    | teeworlds packet header | chunk header  | motd | chunk header  | server settings | chunk header  | CON_READY |
    |        chunks = 3       |     vital     |      |     vital     |                 | vital & flush |           |
    +-------------------------+---------------+------+---------------+-----------------+---------------+-----------+

Int packing

The libtw2 docs also has documentation on ints.
There is also a more technical overview with examples about int packing.

All integer numbers sent through the network are packed using a custom Teeworlds-specific packer.
Even though it's a custom packer, the numbers 0-63 are pretty much the standard binary representation;
binary 00000001 is decimal 1,
binary 00000010 is decimal 2,
and so on. If we exceed 63 it starts to differ.

The int packer code can be found here, together with this annotation comment: 
                
    Format: ESDDDDDD EDDDDDDD EDD... Extended, Data, Sig
This means that the second bit of the first byte of the integer is its sign bit: whether or not the number is signed (that is, negative). If this byte is 1, the number is negative. If this byte is 0, the number is positive.
The first bit of every byte is the extension bit. If it is set to 1, another byte is following. If it is set to 0, the current byte is the last byte. This extension bit allows you to send small numbers (-62 to +63) as a single byte over the network. It also means that technically the protocol allows numbers of any size; however, this is advised against as they are read into a 4-byte-long C++ int.

Note that the byte order is little endian. The smallest possible two byte number, 64, is represented as the bits 10000000 00000001.
This number 64 can be dissected as so: 
                
    ESDDDDDD EDDDDDDD
    10000000 00000001
    ^^^    ^ ^^    ^
    ||\   /  | \   /
    || \ / not  \ /
    ||  \ extended
    ||   \      /
    ||    \    /
    ||     \  /
    ||      \/
    ||      /\
    ||     /  \
    ||    /    \
    || 0000001 000000
    ||       |
    ||       v
    ||       64
    ||
    |positive
    extended

There is a Rust crate named teeint which solely focuses on int packing.

String packing

When strings are sent via the network, they are sent as plain C strings. They do not contain a length, and they have to be null terminated.

The receiver can then use different types of sanitizers when unpacking the string:

Raw packing

If you want to send raw bytes over the network, the Teeworlds protocol got you covered.
There is void CPacker::AddRaw(const void *pData, int Size) and its counterpart const unsigned char *CUnpacker::GetRaw(int Size)

They will just send and receive byte by byte the raw data you give it. Note that while reading bytes you have to know the size in advance. The raw data does not include a size field or is terminated in any way.
Raw data can only be used in contexts where the length is known to the recipient. This usually happens by a leading integer field holding the size.

Huffman compression

Packets can be compressed using Huffman compression. The Teeworlds packet header is never compressed, only its payload. The packet header sets the COMPRESSION flag if the payload is compressed. Usually corresponding packets are compressed, but technically every packet could be compressed or not.

The Huffman compression algorithm is not specific to Teeworlds. To understand its fundamentals you can simply check out any non-Teeworlds related Huffman documentation like this YouTube video. Huffman compression depends on a weight tree, so if you reimplement Huffman yourself make sure it uses the same weight tree as the official Teeworlds implementation does. The official Teeworlds frequency table can be found here.
But chances are you do not have to reimplement the Huffman algorithm even if you want to build a server or client in a new programming language. There are already community implementations for Python, JavaScript, Ruby and Rust.