While looking into US-CERT TA-05-291A. This is what I found.

While snort does review the traffic on port 31337, it will also look
for any UDP traffic that is using Back Orifice’s magic cookie.

* spp_bo.c comments
*
* Purpose: Detects Back Orifice traffic by brute forcing the weak encryption
* of the program’s network protocol and detects the magic cookie
* that it’s servers and clients require to communicate with each
* other.
*
* Back Orifice magic cookie is “*!*QWTY?”, which is located in the first
* eight bytes of the packet. But it is encrypted using an XOR.

When exploiting this we want this function of the preprocessor to kick
off. Which is why you will have to create a UDP packet that is not
using port 31337.

Below is where the fun happens.

//snippet from spp_bo.c
static int BoGetDirection(Packet *p, char *pkt_data)
{
u_int32_t len = 0;
u_int32_t id = 0;
u_int32_t l, i;
char type;
char buf1[1024]; #Interesting ??? A static array? Is this checked? hehe
char buf2[1024]; #Interesting ??? A static array? Is this checked? hehe
char *buf_ptr;
char plaintext;
//snippet from spp_bo.c
 
I don't see any checks.
 
//snippet from spp_bo.c
/* Only examine data if this a ping request or response */
if ( type == BO_TYPE_PING )
{
i = 0;
buf_ptr = buf1;
*buf1 = 0;
*buf2 = 0;
/* Decrypt data */
while ( i < len )
{
plaintext = (char) (*pkt_data ^ (BoRand()%256));
*buf_ptr = plaintext;
i++;
pkt_data++;
buf_ptr++;
if ( plaintext == 0 )
buf_ptr = buf2;
}
 
/* null-terminate string */
*buf_ptr = 0;
 
DEBUG_WRAP(DebugMessage(DEBUG_PLUGIN, "buf1 = %s\n", buf1););
 
if ( *buf2 != 0 )
{
DEBUG_WRAP(DebugMessage(DEBUG_PLUGIN, "buf2 = %s\n",buf2););
}
 
DEBUG_WRAP(DebugMessage(DEBUG_PLUGIN, "crc = 0x%x\n", (char)
(*pkt_data ^ (BoRand()%256))););
 
if ( len > 4 && !strncasecmp((buf1+3), "PONG", 4) )
{
return BO_FROM_SERVER;
}
else
{
return BO_FROM_CLIENT;
}
}
//snippet from spp_bo.c

To validate it a bit more I ran the code through flawfinder. This is
the output.

Examining spp_bo.c
spp_bo.c:430: [2] (buffer) char:
Statically-sized arrays can be overflowed. Perform bounds checking,
use functions that limit length, or ensure that the size is larger
than
the maximum possible length.
spp_bo.c:431: [2] (buffer) char:
Statically-sized arrays can be overflowed. Perform bounds checking,
use functions that limit length, or ensure that the size is larger
than
the maximum possible length.

Just my findings.