Reifying Your Commands – Interprocess Communications by Example

In the first part of this series, I introduced readers to the work reify, which means to make something real. So far, we have seen how the ActionScript application converts a logging request into a command object, serialises it into a byte array, and sends it over a TCP socket connection into the waiting arms of a server. The server for its part must deserialise the byte array back into a command object and execute it.

This last part in the series explains how this is done.

The Story So Far

We have seen how the Puppeteer receives a message. In the receive callback method is a call to deserialise the message into an object.

byte[] message = new byte[messageLength];
Buffer.BlockCopy(buffer, 4, message, 0, messageLength);

ICommand command = Util.Deserialize(message);

The Deserialize utility method receives only the portion of the message that constitutes the actual data. The first 32 bits are discarded as they are not relevant to the deserialisation process. The Deserialize method is extremely simple.

public static ICommand Deserialize(byte[] message)
{
    Dictionary<byte, type=""> instructionClassMap = new Dictionary<byte, type="">() { { 0x02, typeof(Trace) } };

    Type commandType = null;
    ICommand command = null;
    if (instructionClassMap.TryGetValue(message[0], out commandType))
    {
        command = (ICommand)Activator.CreateInstance(commandType, new object[] { message });
    }

    return command;
}

It reads the first byte from the message which contains the instruction to be executed. The instruction is then used as key to look for the type in the instruction map. The Activator.CreateInstance() API is used to instantiate the type into a variable. The instance is then returned from the function.

The receive callback then dispatches a CommandReceived event. The application implements the plumbing from that point onward to handle the event notification and act upon it.

At this point, we need to take a step back and observe the command object instantiation in detail. Each command type has its own implementation detail which interprets and utilises the message. The Trace class, for example, reads the level, category and message values from the message. Its constructor is listed below.

public Trace(byte[] message)
{
    int unixTimeStamp = message[1] << 24 | message[2] << 16 | message[3] << 8 | message[4];
    TimeStamp = Util.UnixTimeStampToDateTime((double)unixTimeStamp);
    int paramCount = message[5] << 24 | message[6] << 16 | message[7] << 8 | message[8];
    parameters = new string[paramCount];
    int index = 9;
    
    for (int i = 0; i << paramCount; i++)
    {
        int length = message[index] << 8 | message[index + 1];
        parameters[i] = Encoding.UTF8.GetString(message, index + 2, length);

        index += (2 + length);
    }

    Level = parameters[0];
    Category = parameters[1];
    Parameters = parameters;
}

The first byte contains the instruction. This is ignored since we already know that the instruction is Trace (0x02).

The next four bytes contain the timestamp of the message as a 32-bit integer. The value is converted into DateTime object through a utility method.

The next four bytes contain the number of parameters that are passed into the Trace command. The command uses this number to determine the number of string objects to retrieve from the message. Remember that each string object is prefixed by a 16-bit integer that contains the number of characters that make up the string. That’s where the index + 2 comes from, which offsets the current position in the array by another 2 bytes. Once the parameters are loaded into an array, they are assigned to public accessors of the Trace class.

The application uses the public members to display the Trace command on screen and store them into a database for persistence.

Socket Talk – Interprocess Communications by Example

In the previous article in this series, I explained the logging framework that the Flex SDK provides, limitations of the Flash Player at providing effective run-time logging, and how they can be circumvented by using the Socket API. I also described a custom message data structure that encodes the information to be sent out of the Flash Player. This part of the article explores the implementation of the socket server that receives the message, and how it decodes it into meaningful instructions and information.

Introducing…Puppeteer!

Puppeteer is a desktop application that runs on Windows which can be used to manipulate one or more client applications. The application works by opening up a TCP port for client applications to connect to, then exchanging messages over the connection to send or receive instructions back and forth. Messages are binary-encoded and follow a simple format which was described in the previous article. Instructions are 8-bit integers, and therefore, have an upper limit of 255 possible values. The message may contain an additional payload along with the instruction. There is no limit to the length of the message.

Client applications require to integrate the ability to dispatch, receive and parse messages to Puppeteer. It cannot connect to any application on its own without the application explicitly requesting the connection, and having the ability to dispatch or react to messages from the server. Client applications can be written on any platform that supports TCP sockets.

On startup, Puppeteer begins listening for incoming connections from client applications on a well-known port number. It also accommodates certain idiosyncrasies specific to the Flash Player. Applications written in other languages can ignore these aspects, as they are most likely not relevant to their own ability to function correctly.

“Hello!”

One of the restrictions applied on the Flash Player is that every TCP socket connection must be explicitly authorised by the owner of that server. To quote Peleus Uhley from the Adobe website –

One of these features is the ability to create TCP sockets in order to exchange data with servers. From a network administrator’s point of view, the idea that content from the Internet could make socket connections to internal hosts is scary. This is why Flash Player requires permission from the target host before it will allow content to make the network connection.

The policy file is requested automatically by the Flash Player the first time the content playing in makes the connect() API call. The process followed is described below.

  1. The content (.swf file playing the the Flash Player) requests a connection to the server through the Socket API.
  2. The Flash Player checks its whitelist if the server has already allowed access to the content.
  3. If it does not already have a policy file for the server in memory, it pauses the connection request from the content and instead tries to connect on port number 843. If a connection is established on this port number, it sends a message containing the string “”.
  4. If the server does not respond on port number 843 unti timeout, the Flash Player attempts to connect again on the same port number that is requested by the content.
  5. The server must respond by sending back the contents of the policy file as soon as the connection is accepted. Any other response from the server will be considered invalid and the Flash Player will disconnect from the server.
  6. Once the policy file is served, the Flash Player parses it for correctness and checks if it authorises the content to connect to the server. If the policy file allows the connection, the content is notified about it and is now able to communicate freely with the server.

So the first requirement for Puppeteer is to be able to listen for a client request for a socket policy file and serve it up. The application does not listen for incoming policy file requests on port 843. Instead, it responds with the policy file when the client attempts to connect on the public incoming request port number (1337 by default). The response is an XML document encoded as a null-terminated string. The null at the end is necessary. The XML response will be treated invalid without it (many hours were wasted in learning this seemingly insignificant detail).

A Mirrored Standard for Commands

Puppeteer works in tandem with the command pattern implemented by its corresponding client library. Each message is serialised into a byte array, which contains the instruction number of the command being invoked, and the payload that accompanies the instruction. It must, therefore, be able to parse and interpret the message, which is done by mirroring the ICommand interface and its implementation classes in .NET.

public interface ICommand
{
    event ExecutionCompletedHandler ExecutionCompleted;

    DateTime TimeStamp
    {
        get;
    }

    void Execute();
}

The ICommand interface declares the Execute() method, same as the one contained within the ActionScript code. It also declares a public property called TimeStamp and an ExecutionComplete event. A CommandBase class implements this interface and provides the basic common set of methods which are required to fulfill this contract.

public abstract class CommandBase : ICommand
{
    …
}

Finally, separate classes are written for each command that Puppeteer must be able to interpret and understand.

public class Trace : CommandBase
{
    …
}

Insert Plug Here

The previous article talked about sockets very briefly. So let me explain that topic before proceeding.

Sockets are a software construct that operating systems use to communicate over the network. They are handles to resources over a network interface, same as files are handles to resources on the hard disk. They come in various different types, of which datagram and stream sockets are most commonly heard of. They are often referred to by the protocol they typically use – UDP for datagram sockets, and TCP for stream sockets. Puppeteer is built on stream sockets. The .NET framework ships with a managed implementation of the socket API in the System.Net.Sockets namespace. The Socket class is the primary actor in this namespace.

On startup, the application begins with instantiating the Socket class, binding it to a local end point and setting it to listen for incoming connection requests.

listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
endPoint = new IPEndPoint(0x0100007F, 1337);
listener.Bind(endPoint);
listener.Listen(0);
listener.BeginAccept(BUFFER_SIZE, new AsyncCallback(AcceptConnection), listener);

In this example, a stream socket is initialised to use the IPv4 addressing scheme and TCP protocol. In order to begin listening, the socket must be bound to an end point, which is a combination of an IP address and a port number. Every TCP connection can be uniquely identified by its two end points – one on the host, and the other on the client. The IPEndPoint class represents an end point on one end of the connection. The IP address of the end point is represented as a hexadecimal-encoded, big-endian 64-bit integer.

Note: The IP address of the computer is hard-coded to 127.0.0.1 in this example. There are more sophisticated techniques to identify the IP address of the computer that the application runs on at run-time which are excluded here for brevity.

Listen is a non-blocking API call. The thread continues to run while the socket is listening for incoming connections. For single-threaded applications, the programmer must put the application into a loop while calling BeginAccept() periodically, in order to prevent it from exiting. Since Puppeteer is a Windows Forms application, this is not necessary.

The buffer size parameter of the BeginAccept API specifies the number of bytes that the server has to read from the message sent by the client. The AsyncCallback is a reference to a callback that is fired when an incoming connection request is received. The last parameter is a state object which can be used to pass around the state of the connection. In this case, the Socket instance itself is used as the state object.

The callback contains code to handover the connection from the listener to another Socket instance dedicated to communicating with the client. This is required so that the listener can be freed to continue to listen for incoming requests on from other clients. The EndAccept() API automatically creates a new Socket instance to communicate with the client.

private void AcceptConnection(IAsyncResult result)
{
    Socket listener = (Socket)result.AsyncState;
    Socket clientSocket = listener.EndAccept(result);

    byte[] response = Encoding.ASCII.GetBytes(SocketPolicy);
    clientSocket.BeginSend(response, 0, response.Length, SocketFlags.None, new AsyncCallback(SendData), clientSocket);
    listener.BeginAccept(new AsyncCallback(AcceptConnection), listener);
}

Once the connection has been established on the new socket, the server must first publish the string containing the socket policy file required by the Flash Player. It does this with the BeginSend method on the new socket. Finally, the original socket instance which is bound to a well-known port number is set back to accept incoming connection requests.

The .NET framework triggers an AsyncCallback when the data has been sent over the new socket instance. This callback signals the socket to end the sending operation, clears the incoming buffer and sets the socket to begin receiving data when the client sends it.

private void SendData(IAsyncResult result)
{
    Socket clientSocket = (Socket)result.AsyncState;
    clientSocket.EndSend(result);
    Array.Clear(buffer, 0, BUFFER_SIZE);
    clientSocket.BeginReceive(buffer, 0, BUFFER_SIZE, SocketFlags.None, new AsyncCallback(ReceiveData), clientSocket);
}

The last stage in this cycle is to receive and process the data, and either respond to the data if required, or set the client socket back into the receiving state as shown here. This is where the meat of the operation occurs. The basic structure of the ReceiveData() method is shown below.

private void ReceiveData(IAsyncResult result)
{
    Socket clientSocket = (Socket)result.AsyncState;

    if (clientSocket.Connected)
    {
        Array.Clear(buffer, 0, BUFFER_SIZE);
        clientSocket.BeginReceive(buffer, 0, BUFFER_SIZE, SocketFlags.None, new AsyncCallback(ReceiveData), clientSocket);
    }
}

Some basic processing is also performed in this method. The application is aware of the message structure that was described in the previous article in this series. The ReceiveData method receives the entire message, but only processes the first four bytes which contain the total length of the message. The rest of the bytes from the message are read from the buffer into a byte array and passed on to a deserialisation utility class, which converts it into an ICommand instance.

private void ReceiveData(IAsyncResult result)
{
    Socket clientSocket = (Socket)result.AsyncState;

    // Extract the length of the message from the first 4 bytes
    int messageLength = buffer[0] << 24 | buffer[1] << 16 | buffer[2] << 8 | buffer[3];

    // Extract the bytes containing the message and deserialize it into an ICommand object
    byte[] message = new byte[messageLength];
    Buffer.BlockCopy(buffer, 4, message, 0, messageLength);

    ICommand command = Util.Deserialize(message);

    if (clientSocket.Connected)
    {
        Array.Clear(buffer, 0, BUFFER_SIZE);
        clientSocket.BeginReceive(buffer, 0, BUFFER_SIZE, SocketFlags.None, new AsyncCallback(ReceiveData), clientSocket);
    }
}

Finally, the method triggers a CommandReceived event which other classes in the application subscribe to in order to act upon the command.

private void ReceiveData(IAsyncResult result)
{
    Socket clientSocket = (Socket)result.AsyncState;

    // Extract the length of the message from the first 4 bytes
    int messageLength = buffer[0] << 24 | buffer[1] << 16 | buffer[2] << 8 | buffer[3];

    // Extract the bytes containing the message and deserialize it into an ICommand object
    byte[] message = new byte[messageLength];
    Buffer.BlockCopy(buffer, 4, message, 0, messageLength);

    ICommand command = Util.Deserialize(message);

    if (null == CommandReceived)
    {
        return;
    }

    CommandReceivedEventArgs e = new CommandReceivedEventArgs(command);
    CommandReceived(this, e);

    if (clientSocket.Connected)
    {
        Array.Clear(buffer, 0, BUFFER_SIZE);
        clientSocket.BeginReceive(buffer, 0, BUFFER_SIZE, SocketFlags.None, new AsyncCallback(ReceiveData), clientSocket);
    }
}

In the next part in this series, I will explain how the command is deserialised and executed by the Puppeteer.

A Message in a Socket – Interprocess Communications by Example

Introduction

I couldn’t help but feel like a luddite as I sat down to write down this article. Who works on Flash applications in 2016 any more? Aren’t we all done with it now?

The Flash Player might have its own set of problems, which is a topic for another discussion. The ActionScript language, however, is fun to work with. It is syntactically similar to Java, has a class-based inheritance model, a choice between static or dynamic typing, or both, within the same code base, namespaces, and a host of easy to use and powerful APIs for 2D and 3D graphics, network operations, animation, audio processing, XML and text processing, and more. Frameworks like the open source Flex SDK make it possible to do even more interesting things on top of this.

But, more importantly, we are heavily invested into the Flash ecosystem for the VMX platform at Vertika. We have 9 years of effort and 40,000 lines of code written in ActionScript and MXML, which would be very expensive to port over to a new platform without a very compelling business reason. So here we are today.

Unfortunately, the Flash Platform also comes with its idiosyncrasies, one of which is addressed in this article.

For those losing interest at the mention of Flash, the good thing about this client architecture is that it can easily be ported over to literally any other platform with little change and still work. And the server-side stuff is written in C#. Collectively, this solution covers a plethora of different concepts such as socket programming, data structure design, a custom-built lightweight ORM framework and design patterns.

Raison D’être

The Great Catsby is a rather upset feline. He is fond of company and loves nothing more than to kick back with a saucerful of heavy cream milk and a stack of Calvin & Hobbes comics with his human pals. But they just won’t let him out of his cage. Catsby, you see, is a 300 kilogram Bengal Tiger at the zoo. And no matter how much he pleads, they are not going to let him out and explore the city.

It’s a rather bleak state of affairs for the poor cat.

The guys at Adobe (then Macromedia) had the same ideas when they designed the Flash Player, which was designed primarily as a browser plug-in to play specially authored content served off websites. And as every competent security-minded person knows, it can be a very bad idea to allow strange websites to gain unrestricted access to your computer. Even if every ActionScript programmer pinky-swore not abuse their users’ trust, the guys at Macromedia were not going allow them free rein over the computers running the Flash Player. Hence, Flash content runs in a sandbox with several restrictions, one of them being no direct access the file system.

This is a tough situation for many programmers who need any kind of runtime application logging. No file system access means there is no persistent data storage on the client.

It is not that there’s no logging at all. The Flash Player Debugger exposes a very limited, plain-text log file access that is disabled by default, and requires a configuration flag in a specific plain-text file in order to be activated. The state of this flag cannot be queried or controlled from within the Flash Player. Extracting the file is a task and a half because it’s buried deep within the file system hierarchy. The log file is shared among all Flash Player applications which are executed on that computer. While there is no possibility of data conflicts, filtering and extraction of the data that is relevant to your application becomes a challenge. Finally, since this is a plain-text file, there is no structured information storage facility available here other than what can be achieved with plain text files (such as comma-separated values).

Did I mention that even this facility is only available on a special Debugger edition of the Flash Player that most people don’t use?

In effect, it’s impossible to have any kind of reliable, persistent runtime logging on a production computer. A third-party intervention is essential. And this is the challenge that is addressed in this article – a logging framework that allows Flash content to delegate the task of persistent storage to another process, which runs at a higher privilege level and has access to the file system. The persistence process can be running locally on the same computer, or on another computer. The mechanism works identically as long as the two computers are able to establish a connection with each other over standard sockets.

Hi! It’s me, your brother!

Parents are like the operating system of the family. They spawn child processes, manage resources and schedule time for tasks. A well-organised family is like Unix. Every child gets their own space, and nobody gets into each other’s way. When they have to communicate, they send messages to pre-designated points. These could be a cell phone, a sticky note on the fridge or a whiteboard in the hallway. They could also speak across the dinner table, but nobody does that in 2016. So that’s out.

Speaking over the cell phone is more versatile because it works whether the person is at home or away at work, out running an errand, or at a party. A message on a sticky note is only delivered when the person returns back home and checks the fridge.

Operating systems also offer similar means to communicate between processes. You could send a message to the recipient over a socket, or write to a file that the recipient checks periodically. Since we have already discussed earlier that writing to files is out, that leaves us with socket communications.

Fortunately, the Flash Player has an API to communicate over TCP sockets. It works identically when communicating with a different device or another process on the same computer where the Flash application is being run. The Flash Player Socket API can only make client-side calls. It cannot listen for connections from another process. Therefore, the external application must play the role of a server and begin listening for connection requests from the Flash Player on a known port number. Once the connection is established, the two processes can send messages back and forth.

This mechanism can be utilised to create an external logging tool that listens for messages from a Flash Player client, and logs them to a persistent storage such as hard disk. Since the socket API works locally as well as remotely, the logging server can be situated at an offsite location, and as long as the two computers can communicate with each other, log messages will be successfully delivered and logged.

Flex Logging API

A lot of work that is needed to establish a robust logging API has already been implemented as part of the Flex SDK in form of the Logging API, which is loosely modelled on its Java framework equivalent. This API provides a convenient way to create structured log messages with severity levels and named categories. A log message is triggered by simple API calls, and is printed to a destination by a target object. Targets can be programmed to listen for messages at a certain severity level (and above), as well as certain categories only. For example, low-latency logging such as an in-memory temporary store can be used for diagnostic messages which are triggered more often, while infrequent but critical errors can be dispatched to remote persistent storage.

The SDK ships with two in-built logging targets – LineFormattedTarget and TraceTarget – which provide basic logging facilities to the application. TraceTarget prints the message to the Flash Builder or fdb console and extends LineFormattedTarget.

The Flex Logging API is documented elsewhere in sufficient detail. However, I’ll cover it briefly in this article to provide context to the reader for how it is used.

The API is centred around four types.

  1. mx.logging.Log
  2. mx.logging.ILogger
  3. mx.logging.LogEvent
  4. mx.logging.ILoggingTarget

The SDK ships with a mx.logging.AbstractTarget class that implements the ILoggingTarget interface. Developers can extend this class to build their own custom logging target implementations.

How the Flex Logging API Works

The ILogger interface provides methods to send messages to one or more targets. The developer doesn’t create the logger instance directly. Rather, they call upon the getLogger static method of the Log class to retrieve an ILogger implementation instance. The getLogger method takes a string parameter called category, which can be used to filter log messages down to a particular sub-system within an application.

The category is conventionally set to the fully-qualified name of the class that calls the ILogger API.

The ILogger interface also exposes methods to perform log operations at five different severity levels, and a generic log method that takes the severity level as a parameter.

The severity levels in increasing order are DEBUG, INFO, WARN, ERROR and FATAL.

var logger:ILogger = Log.getLogger("com.notadesigner.ExpletiveGenerator");
logger.info("Sonofagun!");
logger.log(LogEventLevel.FATAL, "Crumbs!");

A logging target receives the log message and prints it to the destination medium. The logging target has a property called level, which determines the severity of messages that it will receive.

var target:ILoggingTarget = new TraceTarget();
target.level = LogEventLevel.INFO;

The target receives all messages of its assigned severity level and below. For example, the target instance in the example above would be able to receive DEBUG and INFO messages.

A logging target instance also has a filters property of type array that contains the categories which this instance should listen for.

target.filters = [ "com.notadesigner.ExpletiveGenerator" ]; // Category name must match

This snippet enables the target to receive all log messages whose category is set to “com.notadesigner.Example”. The filters array can contain more than one category.

target.filters.push("com.notadesigner.GreetingsGenerator");

The target is actually of greatest interest in this article, because it is this class that performs the message printing. In order to send the message to an external process, we need to build a custom logging target that is able to dispatch messages over a socket.

com.notadesigner.sockPuppet.SocketTarget

The SocketTarget class extends AbstractTarget rather than LineFormattedTarget, because the latter merges all the message fields into a single string, which we don’t want. By keeping them as separate fields of their native types, we get maximum control over the structure of the message, as well as the least amount of memory usage.

We begin by instantiating the class.

var target:SocketTarget = new SocketTarget("localhost", 1337);

The constructor takes two parameters for the host name and the port number on which the socket server is already ready and running. We will gloss over the details of performing the connection and maintaining it.

When a message is received, the logging framework triggers the logEvent method of the SocketTarget instance. This is defined as an empty method in the AbstractTarget class that takes a parameter of type LogEvent. By overriding this method, the SocketTarget instance can then dispatch the message to the server.

Making It Real

Before Daryl got into the programming business, he used to wait at the local deli. His job was to take orders from diners and pass them on to the kitchen, and serve the completed order after it was prepared by the kitchen staff. Daryl’s task was simple. He would jot down a list of orders from the table and pass it on to the chef. The chef would then prepare whatever items came up in the queue, and ring up the waiting staff once it was ready. The staff picked the completed order and served it fresh and hot to their patrons.

You want a burger and fries? Sure! Footlong with everything? On its way, sir. Need coffee with extra cream? Here it comes.

The food was good and footfalls were high. The entire team loved their job and did their best to stick to the process because it was so efficient. It abstracted away the underlying complexity of preparing a meal with all its customisations into a simple, mechanical task, while still retaining all necessary details that the kitchen staff would need to build an order.

In the years that followed, Daryl was on his way to become an accomplished software developer of some fame. It was during this period that he learned about design patterns in software engineering and came across the command pattern. In his mind, he could relate immediately.

A command is a reified method call. – Bob Nystrom, Game Programming Patterns

I find Bob‘s explanation to be much more relatable than the longer definition defined in the GOF book. There’s a beauty in its terseness. To understand it, of course you need to understand what reified means. To borrow another quote from Bob‘s book

“Reify” comes from the Latin “res”, for “thing”, with the English suffix “–fy”. So it basically means “thingify”, which, honestly, would be a more fun word to use.

So there we are. The command pattern converts an abstract item like a deli order into a real, fresh, piping hot meal. I‘m sure even reading about this makes you hungry. Maybe you would like to repeat the experience right about now, and issue a command to your local delivery joint for a large pizza with Coke. Go ahead. I‘ll wait.

Are you back? Let‘s continue.

Returning to the task at hand, we are now at a stage in the logging operation where the message is to be dispatched to the socket server process. The Flex framework already provides all the necessary structured information in an event handler – the logEvent method – that is overridden by the SocketTarget class. It is a matter of sending this information to the external process, so that it can be logged to disk. Reify the log, if you please. We need some infrastructure in order to implement this mechanism.

A Standard for Commands

The invention of the assembly line provided a massive boost to human progress during the industrial revolution. It was a break from the previous practice of having few master-craftsmen assembling a product from start to finish. By delegating each part of the assembly to a single individual, the human skill required was greatly diminished and productivity soared. Rather than having one exceptionally good worker build a single sword in a day, factories were able to assemble dozens by assigning several average-skilled workers, each handling only one aspect of the assembly process. All workers just followed one instruction – assemble the product. But what each did in response to this instruction was different. Somewhat akin to what we do with the ICommand interface.

This interface unifies all commands so that they can be guaranteed to have a single method for execution. The ActionScript implementation looks like this.

public interface ICommand
{
    function execute():ByteArray;
}

The obvious follow up is why have a command interface at all when the only operation we need is to trace messages. However, it’s a small leap from where we are now (sending plain-text messages for printing) into communicating more elaborate commands such as enabling and disabling logging, force-flushing the log to disk, clearing the log or sending other structured diagnostic information such as internal data structures. It becomes easier to swap out one command for another, much like the assembly line, if they all must adhere to a common interface. All these commands can be triggered from the Flash application by instantiating an appropriate command class instance, serialising it, and dispatching the result over the connection.

The details of the execution of the command are specific to the command. The common operation is serialisation of the message from a Flash object to something that the server can read. For this, we use a byte-array because it is compact and efficient to send over a socket.

Trace generates a certain kind of layout in the byte array, which contains the string to be traced in the message log. Another possible message might be to toggle the state of the logging operation itself. In this case, the byte array could only require a Boolean value to indicate the toggle state. A more complex diagnostic message might send across complete object instances after serialisation, along with information on how they are to be deserialised again.

While message specifics are different, they all share a common structure called the message header. The header in the current version of this API requires the following elements.

  1. Message length (32-bit integer) – The total length of the message in bytes (including the header and all parameters).
  2. Instruction (byte) – An 8-bit integer value that contains the instruction code. Each command has a unique instruction number associated with it. This makes the message more compact than sending the human-readable command name as a string.
  3. Message timestamp in UTC (32-bit integer) – The date and time that the message was created in the Flash Player, encoded as seconds since the UNIX epoch. The time is calculated relative to UTC and must be converted into local time as an additional step by the receiver of the message, if needed.
  4. Number of parameters (32-bit integer) – A count of the number of parameters attached to the message.

The instruction for the Trace command is assigned the value 0├ù02. The byte array needed to trace the string “Hello” at severity level INFO is shown in the table below. Each cell represents 1 byte.

Message Length (4 bytes)
0×000×000×000×20 
Instruction (1 byte)
0×02 
Timestamp (4 bytes)
0×570×CD0×AC0×31 
Parameter Count (4 bytes)
0×000×000×000×03 
String Length (2 bytes)
0×000×04 
Severity Level (“INFO”, variable-length string)
0×490×4e0×460×4f 
String Length (2 bytes)
0×000×04 
Category (“Main”, variable-length string)
0×4d0×610×690×6e 
String Length (2 bytes)
0×000×05 
Message (“Hello”, variable-length string)
0×480×650×6c0×6c0×6f

The first 4 bytes contain the message length. The next 1 byte contains the instruction code (0×02). Four bytes are allocated to sending the timestamp in seconds (1,473,096,753 seconds since 1 January 1970). The next 4 bytes contain the number of parameters attached to this message (3). The contents of the rest of the message are specific to the Trace command. It is made up of 3 data fragments that encode a string prefixed by its length as a 16-bit integer. A string is an array of characters. C-style strings mark the end of the array by placing a null character in the last position of the array. When a piece of code needs to perform any string operation, it walks the length of the array until it arrives at the null terminating character. Alternative to this are length-prefixed strings, or Pascal strings, which prefix the length of the character array at the beginning. Pascal strings do not require a null terminating character as the length of the string is already known beforehand.

This kind of data structure has an advantage of speed over null-terminated strings. Any operation that requires the length of the string can peek into the beginning of the sequence, making it a constant time operation. Finding the length of null terminated strings, on the other hand, is a linear operation because it requires walking the entire byte array to locate the null terminator. Longer strings take more time, shorter strings take less.

The writeUTF method of the ActionScript ByteArray class uses Pascal strings. Programmers do not have a choice in this matter. Therefore, all commands, present as well as new ones in the future, use this same structure to encode strings.

A Pregnant Pause

Once the message has been serialised into a byte array, the send method of the Socket class is used to dispatch the message over the network.

At this point, the message is out of the realm of the Flash Player and is crossing boundaries into the .NET runtime. If the server at the other end receives the message in full (which is more or less guaranteed due to the TCP stack), it can be parsed, deserialised and acted upon by the server. If the message fails for any reason, the Socket class raises an error event within the Flash Player for the programmer to handle. For brevity, we ignore that aspect and proceed with the assumption that the message has been received successfully and in full. Transparent to the ActionScript programmer, the server dispatches an ACK response to acknowledge receipt of the message. There is nothing more that the Flash Player has to after this point and it can return back to its steady state.

Keep watching this space for part two of this article that explores receiving the message on the server, interpreting it into its component parts, then acting upon the instruction contained in it.

WPF Game of Life

I was introduced to cellular automatons a good decade ago through books written by Peter Small. Back then, we were using Adobe (then Macromedia) Director as our primary development platform, where object-oriented programming was still a bit of a novelty. Small’s writings explained OOP using artificial life forms such as cellular automatons, with Conway’s Game of Life being an oft used concrete example. I was hooked and would spend hours running patterns on a simple implementation of the game that shipped with Windows Entertainment Pack.

But I never got around to implementing the application myself until now. The game has a fairly simple model that does not take much effort to build. This is handy when learning a new platform or language, leaves the programmer free to explore the features of the framework instead of spending cycles trying to nail down the business logic correctly using an unfamiliar language syntax.

Background

There are plenty of resources available online that describe, implement and demonstrate this game. The basic essence of the game is a two-dimensional array of cells and an endless loop that iterates over these cells to update their state depending upon some simple rules.

  1. Any live cell with fewer than two live neighbours dies, as if caused by under-population.
  2. Any live cell with two or three live neighbours lives on to the next generation.
  3. Any live cell with more than three live neighbours dies, as if by overcrowding.
  4. Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

This implementation of the game also consists of the following features for convenience.

  1. Clicking on a cell toggles its state between being alive or dead.
  2. A Randomize button initializes the board by randomly setting the state of all cells to either alive or dead.
  3. A Stop button lets the user stop the cell update iterations in order to observe the arrangement of the board at any given time. This is useful for times when an interesting pattern appears on the screen that seeks further observation.
  4. A Next button complements the Stop button to let the user iterate to progressive generations of the board, one step at a time.

Implementation

This implementation is made up of the following classes.

MainWindow.xaml and MainWindow.xaml.cs

The application window is a XAML class called MainWindow.xaml, which defines the layout of the application. The layout is built around a DockPanel that divides the screen into two – a controller toolbar and the board view.

<DockPanel Name="layout">
    <ToolBar DockPanel.Dock="Top">
        <Button Content="Start" Name="btnStart" Height="23" Width="75" Margin="5" Click="toggleStart_Click" />
        <Button Content="Next" Name="btnNext" Height="23" Width="75" Margin="5" Click="nextIteration_Click" />
        <Button Content="Clear" Name="btnClear" Height="23" Width="75" Margin="5" Click="clear_Click" />
        <Button Content="Randomize" Name="btnRandomize" Height="23" Width="75" Margin="5" Click="randomize_Click" />
    </ToolBar>
    <ScrollViewer HorizontalScrollBarVisibility="Auto" VerticalScrollBarVisibility="Auto">
        <life:BoardView x:Name="view" DockPanel.Dock="Top" Margin="10" Click="view_Click"></life:BoardView>
    </ScrollViewer>
</DockPanel>

The MainWindow class stores a reference to an instance of BoardModel as a private field. Buttons in the toolbar are wired to trigger event handlers in the MainWindow class, which in turn trigger public methods exposed by the model and view instances.

BoardModel.cs

The model class is where the business logic behind the application is implemented. It contains two arrays of bytes of identical length, which store the state of the board in the current iteration and the next iteration. It also initializes a timer that is used to iterate over the board state automatically. In addition, it exposes public methods to control the state of the board – Next(), Start(), Stop(), Clear() and Randomize().

The model instance dispatches an Update event every time the board changes. The window class listens for this event and passes on the contents of the new board cells to the Update() method of the view instance.

BoardView.cs

The view class inherits from System.Windows.FrameworkElement in order to take advantage of its built-in layout functionality. The board is drawn using low-level Visual elements. While it is easier to implement the cell drawing by using Drawings or Shapes, it also adds a lot of overhead for unnecessary features such as styles, data binding, automatic layout and input events. DrawingVisuals provide a much more performant way of drawing large numbers of objects on the screen.

private void DrawCells()
{
    if (null == this.values)
    {
        return;
    }

    using (DrawingContext dc = this.cells.RenderOpen())
    {
        int x = 0;
        int y = 0;
        Rect rect = new Rect(OUTLINE_WIDTH, OUTLINE_WIDTH, Constants.CELL_SIZE - OUTLINE_WIDTH, Constants.CELL_SIZE - OUTLINE_WIDTH);

        for (int i = 0; i &amp;lt; values.Length; i++)
        {
            x = (i % Constants.CELLS_X);
            y = (i / Constants.CELLS_X);
            rect.Location = new Point((x * Constants.CELL_SIZE) + OUTLINE_WIDTH, (y * Constants.CELL_SIZE) + OUTLINE_WIDTH);

            if (1 == values[i])
            {
                dc.DrawRectangle(Brushes.Red, null, rect);
            }
        }
    }
}

One must fetch a reference to a DrawingContext in order to draw content onto a DrawingVisual. This is done through the RenderOpen() method of the DrawingVisual. Since we are only drawing basic rectangles, we can use the in-built DrawRectangle() method of the DrawingContext class to draw this shape for us. Internally, this method still uses a GeometryDrawing and a subclass of the Geometry class, but abstracts away those details for convenience.

The drawing is displayed on the screen by adding the Visual instances to the visual tree of the BoardView class, which is done in its constructor.

public BoardView()
    : base()
{
    this.AddVisualChild(this.grid);
    this.AddLogicalChild(this.grid);

    this.AddVisualChild(this.cells);
    this.AddLogicalChild(this.cells);

    this.visuals = new DrawingVisual[] { this.grid, this.cells };

    this.drawGrid();
    this.drawCells();
}

Additionally, the BoardView class also overrides the VisualChildrenCount, GetVisualChild and MeasureOverride members of the FrameworkElement class.

protected override int VisualChildrenCount
{
    get
    {
        return this.visuals.Length;
    }
}

protected override Visual GetVisualChild(int index)
{
    if (index < 0 || index > this.visuals.Length)
    {
        throw new ArgumentOutOfRangeException("index");
    }

    return this.visuals[index];
}

protected override Size MeasureOverride(Size availableSize)
{
    return new Size(Constants.CELLS_X * Constants.CELL_SIZE, Constants.CELLS_Y * Constants.CELL_SIZE);
}

The BoardView class instance also dispatches an event when the user clicks on a cell on the board. This is in turn passed over to the model instance via the window, causing the model to toggle the state of the cell that was clicked.

ClickEventArgs.cs

An instance of this class is passed as a parameter along with the Click event is dispatched from the BoardView class. It exposes two parameters to identify the X and Y coordinates of the cell which was clicked upon, relative to the board.

The Visual Studio solution for this project can be downloaded from here as a ZIP archive.

Storing Values with Bit Packing and Unpacking

Bit packing and unpacking is a handy way of consolidating multiple data values into a single variable, thus compressing the amount of data being stored or transmitted. The number of values that can be stored depends upon the width of the data to be stored as well as the type of the value that it is packed into. A simple byte can store up to 8 bits of data. Larger types such as ints can store up to 16, 32 or 64 bits. This is an especially efficient technique for storing several small-width values, often smaller than the smallest width supported by a platform (such as byte or boolean flags) into a single large value such as a 32-bit integer.

Bit flags are commonly used for implementing low-level features, such as storing file access-permissions or packing values into a single value before transmitting across a bus. However, they can be applied with equal ease to higher level tasks such as storing user preferences or choosing which widgets to display from an enumerated list. We will see here how to use bit flags to store font formatting preferences, and apply them later to a label.

Bitwise Operators

There are a couple of operators we need to understand before we can move on to the implementation. Bitwise operators, by definition, work on individual bits inside a value. Since they are implemented directly by the processor itself, they are much faster than arithmetic operators such as division and multiplication. We will use bitwise AND (&), bitwise OR (|) and left shifts (<<) in this exercise.

A bitwise AND operation takes the binary representations of two values and performs a logical AND operation on each bit. The result is 1 in every position where both the bits are 1, and 0 if either or both bits are 0.

    001010
AND 011011
    ------
    001010

Bitwise OR on the other hand, compares two bits in corresponding positions, and sets the result to 1 if either of them is 1, or to 0 if both of them are 0.

    001010
OR  011011
    ------
    011011

Bitwise left shift operator moves individual bits within a single value by the number of places specified in the second operand. The value is padded with 0s on the right, and the left-most bits are dropped off.

    001010 << 1 = 010100

Implementation

We set up a simple Windows Forms project and draw three checkboxes and one label on the form. The aim is to have the checkboxes control three font properties of the label – weight, style and underlining. All checkboxes are given appropriate labels and configured to execute the _changeFormatting method of the form every time the CheckStateChanged event is fired. The code for this method is shown below.

private void ChangeFormatting(object sender, EventArgs e)
{
    byte flags = 0;

    flags = (byte)(
        Convert.ToByte(this.chkUnderline.Checked) << 2 |
        Convert.ToByte(this.chkItalic.Checked) << 1 |
        Convert.ToByte(this.chkBold.Checked)
        );

    Font f = new Font(this.label1.Font.FontFamily, this.label1.Font.Size, (FontStyle)(
        (flags & (byte)FontStyle.Underline) |
        (flags & (byte)FontStyle.Italic) |
        (flags & (byte)FontStyle.Bold)
        ));

    this.label1.Font = f;
}

Packing

In the first statement, the flags variable is populated with the values of each checkbox. We want to store the three flags in the last three bits of a single byte.

PositionSetting
7Unused
6Unused
5Unused
4Unused
3Unused
2Underline
1Italic
0Bold

In order to do so, we take the value of each boolean (either true or false), convert it into a byte, then shift it by an appropriate number of positions. The value of the underline flag is to be stored in the 2nd bit (starting from 0). So we left-shift its value by 2. Similarly, the italic flag is stored in the 1st position, so its boolean value is shifted by 1. The value of the bold flag does not need to be shifted at all.

    00000001 << 2 = 00000100 // Underline
    00000001 << 1 = 00000010 // Italic
    00000001                 // Bold (no shifting required)

A consolidated value can be generated by ORing the three values together.

    00000100
 OR 00000010
 OR 00000001
    --------
    00000111 // Decimal value 7
    00000000
 OR 00000010
 OR 00000001
    --------
    00000011 // Decimal value 3
    00000100
 OR 00000000
 OR 00000001
    --------
    00000101 // Decimal value 5

The decimal value can then be stored in a database or other persistent storage system as an integer or byte. This is better than having to store three boolean fields. This information can transmitted across systems too as a packed unit, to be unpacked later only when the preferences have to be applied to a display element.

In our example, we are unpacking and applying the values immediately for brevity. But a more practical situation would probably involve serializing the value somewhere, then deserializing and applying the font properties later at another location.

Unpacking

In order to apply the font styles on a display element, the individual values of each style parameter must be extracted from the packed value and then applied. The .NET framework defines enumerations for each of these style parameters in the System.Drawing.FontStyle enum. The values for each style parameter are listed below.

SettingDecimal ValueBinary Value
Regular000000000
Bold100000001
Italic200000010
Underline400000100

You will notice that each enumeration is double the value of its predecessor, hence moving the digit 1 by one position leftwards with every increase. This is a key feature of bit flags. Each element differs from the others only in the position of the 1 bit. Thus, the value of a given flag can be extracted from the packed value by ANDing the packed value with the value of the enumeration.

     00000111 // Packed value decimal 7
AND  00000100 // Underline enum decimal 4
     --------
     00000100 // Result - show underlines

This operations shows that the value of the underline flag is true. If the packed value was the decimal 3 instead of 7, then the operation would play out as shown below, resulting in the value 0 for the underline flag.

     00000011 // Packed value
AND  00000100 // Underline enum
     --------
     00000010 // Result - hide underlines

All that is needed then is to convert the result byte into a boolean and apply it wherever required. In our example above, the constructor of the Font class requires the values packed together any way as a FontStyle enum. To do this, each bit is ANDed with its corresponding enum, then all of them are combined together again using an OR operation. The resultant byte is cast into a FontStyle before being passed to the constructor.