Debugging without a debugger, profiling without a profiler

Johann Höchtl asked a few interesting questions on the Google Go mailing list regarding my experiences with using Go for a sizable project. Let me answer these questions in turn:

The Go package system

Quite simply, in my experience the Go packaging system is already mature enough. I ran into no troubles with it whatsoever. A few months ago the packaging system was slightly immature in that you could not have packages with names clashing with the Go packages, but this has been fixed for a long time now. And so, the packaging system is good to go for large projects, in my opinion.

What was a headache?

There are a couple of GO features that one needs to understand carefully, otherwise the bugs they produce can be time consuming to find.

The first one: If you implement your own io.Reader (or io.Writer) interface make sure it really truly behaves as the specification of io.Reader requires. In particular, you will notice that a Read() method is never supposed to return a negative byte count, even when an error has occurred. This is an easy mistake to make for those used to POSIX conventions. If you do make this mistake and use your Reader in conjunction with, say, bufio.Reader you will be faced with bizzarre behavior because bufio.Reader assumes proper behavior from your io.Reader and makes no error checking. I gather (from the Go Team) that this is an intended design decision (not to check for correct input/output argument behavior, or correct interface behavior). It makes sense and, of course, it does mean that in the future there should be an automated way of running your program in a “debug” mode where proper behavior is ensured at real time and bugs are caught. Or one might even have a static analysis tool.

The second: This code is flawed. Do you see why? The err initialized inside the blocks is not the same as the return parameter err. As a result, when you use return with no arguments, you are returning the wrong err (the one that is nil). I quickly learned to avoid this by not using argument-less return, however the bugs caused by this kind of code were even harder to catch than the ones given above.

What is missing from the Go language?

There’s nothing missing that was really annoying or disturbing. Once in a while I could have saved a few lines of code if there were generics. But such cases were neither common nor particularly crucial. I would not pressure the Go Team to have generics until they have a very good design in place.

Debugging and profiling without the tools

The main thing that you cannot debug with ease using Printf’s is catching deadlocks, because in a system as complicated as Tonika, they could be anywhere. For this I built a very handy profiler package (given below) that monitors time spend inside each lock and you can get a textual printout at any time and see which locks have not been released or have been held for too long. This package is a real treat, but I have not submitted it to the Go repository because I believe that the long term solution will be different (using a real profiler and debugger). But what I have was more than sufficient. The bottom line with this package is that anytime you use a mutex, you will define your variable as prof.Mutex, rather than sync.Mutex and nothing else changes. See the source code below:

• Statistical functions for keeping running average and variance of time locks held: avgvar.go
• Main profiler type: prof.go
• Convenience type for profiling Mutex’s: mutex.go

Thanks for your questions!

Build 1.0.0.0

Tonika Build 1.0.0.0 is the first public appearance of Tonika. It is part of Tonika’s Alpha Test stage, so you should expect that it would be somewhat rough around the edges. The Alpha Test stage is designed to give us a feel of how the core functions of Tonika work, and thus various convenience features are intentionally missing. This release is mainly targeted at computer savvy people and engineers. We expect to bounce off neat ideas on the Tonika Google group and maybe strike up some open-source contributions relationship with some of you.

There are a few things to bear in mind while using Tonika Build 1.0.0.0:

• Eventually, when we move out of the test stage, the Tonika protocols may change and this release may not be compatible with future versions
• This release cannot break through NATs and firewalls, which means that for a pair of users to be able to establish contact, at least one of them must have a publicly visible Internet address and port where the Tonika client can be accessed directly.
• The Tonika Client will send some anonymous usage information to us. This information is fully-privacy preserving in that it has no relation to you or how you use Tonika. It merely helps us make sure that the routing algorithms are well and healthy. For extra security, these statistical updates travel the Internet to our server in an encrypted manner.

That said, go right ahead and download Tonika from the home page.

How Tonika works, in short

This will be the first in a series of articles informally describing the inner workings of Tonika. As a first, we’ll begin with a higher level picture of what is really going on.

The basics: Sharing with friends

When you run the Tonika software (called the Tonika Client) for the first time you obtain a unique and permanent URL which looks something like http://34jpr5y7kzwb9z.5ttt.org. A user’s first order of business is to invite and add a few friends. When you add a friend, your Tonika Client and that of your friend arrange a mechanism to talk to each other in a secure and authenticated fashion. This is effectively the same as having a direct wire to your friend that no one can eavesdrop on. Your friends become, what we call, your Tonika neighborhood. As you’d expect, you can share files and other content with your friends in a private manner. This is done quite seamlessly. Whenever you place a file, say, mydiary.html in your designated home directory (see the Installation instructions) on your computer, it becomes momentarily accessible to your friends under the URL http://34jpr5y7kzwb9z.5ttt.org/mydiary.html (where the host part of this address is your Tonika URL, whatever it happens to be). Your friends, however, have no way of knowing that you added the file mydiary.html. You would have to prompt them somehow of its existence. We’ve made a simple provision that makes this somewhat convenient. If you create a file named index.html in your home directory, it will be shown every time one of your friends visits your base Tonika URL (i.e. http://34jpr5y7kzwb9z.5ttt.org). You can use this file to put links to the various files that you have placed in your home directory. If you don’t have an index.html file, your base Tonika URL will simply show a blank page.

Beyond the basics: Re-sharing content via links

Imagine now that one of your friends really likes your mydiary.html and wants to share it with another friend of theirs. The inefficient thing to do would be to download mydiary.html from you and then email it to their other friend. Instead, they can take advantage of a key Tonika feature: URLs work globally anywhere in the Tonika network. Your friend can simply email the third party the original link http://34jpr5y7kzwb9z.5ttt.org/mydiary.html (or place it in their index.html).

When the third party tries to access this link, they will get the file mydiary.html. However, this file will not be coming physically from you, rather from their friend. Which brings us to the main Tonika principle:

A user’s computer never performs any communication with parties other than their immediate friends.

This is a good place to stop for now, and let you fiddle with the software. More concepts coming up in future posts.

Running Tonika on Windows

Currently, Tonika has no native binaries for Windows for a simple reason: Google Go does not support Windows yet. The good news is that Go support for Windows is in active development and it will, most likely, be ready in the near future.

In the meantime, you can still run Tonika under Windows, but you will need to take care of a few technical steps. What follows is an outline of how to install Tonika under Windows. Make sure you read the installation instructions before you read this:

1. Install FreeBSD (or Linux) in a VMWare (or other) virtual machine.
2. Install Tonika inside the virtual machine.
3. Pick a Home Directory for yourself on your Windows drive and map it (using VMWare’s tools) to a directory inside FreeBSD. You will instruct Tonika to use that latter directory as its Home Directory, using the --hdir option.
4. Finally, make sure that you tell Tonika to allow access to the Front End proxy from your Windows machine. You will do this using the --fe-allow command-line option.

You may want to review the Tonika command-line options for more details.

Command-line options

There are a few command-line options:

• --addr=[host_and_port], where [host_and_port] can be an IPv4 address and port, or an IPv6 address and port, or a URL of a host and port. E.g. myurl.org:45, or 1.2.3.4:555, or [de:ad:be:ef::ca:fe]:877. This option specifies what address the Tonika client binds to. By default, it binds to localhost and a random available port.
• --id=[filename] specifies the location of your Identity File. This is the file containing all of your private information, like who your friends are and what your private cryptographic keys are.
• --hdir=[dirname] specifies the location of your Home Directory. This is the directory where you put files that you want to publish on the Tonika network.
• --cdir=[dirname] specifies the location of your Cache Directory. This is a directory that Tonika will use internally to cache content while performing downloads and other tasks for you.
• --pdir=[dirname] specifies the location of the Program Directory. This is the directory where you unpacked the Tonika installation ZIP file. The executable tonika (as well as some other subdirectories) should be in this directory.
• --fe=[host_and_port] specifies which host and port the Tonika Front End interface binds to. This is the same host and port that you configure your browser to use as a web proxy. The format of [host_and_port] is similar to that for the --addr option.
• --fe-allow=[comma_separated_hosts] specifies which hosts can access the Tonika Front End interface. By default, only connections from localhost can access the Tonika Front End.

Installation instructions

Tonika Builds numbered 1.*.*.* are part of the Alpha Test stage. At this stage Tonika comes as a downloadable ZIP file (rather than as a self-installing program), which you can obtain here.

To install Tonika and get it running, follow these steps:

1. Unpack the contents of the ZIP file. So e.g. say you are currently in your home directory /Users/petar (in my case). You will run the command:
   

   unzip tonika-osx-1.0.0.0.zip

   This will create a directory /Users/petar/tonika, which we call the Program Directory.

2. Create a directory which will keep all your Tonika-related files. E.g.
   

   mkdir /Users/petar/MyTonika

3. Before you can run Tonika, you have to create two special directories. The first one is your Home Directory. This is where you will be placing files that you want to publish. You can put this directory anywhere and call it anything. So e.g.
   

   mkdir /Users/petar/MyTonika/Home

   The second directory is your Cache Directory. Tonika uses this directory internally. So e.g.

   mkdir /Users/petar/MyTonika/Cache

4. Tonika saves all of your private identity information (who your friends are, your private keys, etc.) in a file called the Identity File. You need to decide what the file should be called and where it resides. E.g. I use
   

   /Users/petar/MyTonika/Identity

   You don’t have to create this file yourself.

5. You are now ready to run Tonika:
   

   /Users/petar/tonika/tonika \
     -pdir="/Users/petar/tonika" \
     -hdir="/Users/petar/MyTonika/Home" \
     -cdir="/Users/petar/MyTonika/Cache" \
     -id="/Users/petar/MyTonika/Identity"

   Obviously, substitute the actual file and directory names that you use.

6. Tonika is now running. To use it, you need to configure your browser to use
   

   localhost:4949

   as a proxy (for all protocols, i.e. HTTP, HTTPS, etc.). On FireFox, your network preferences dialog box should look something like this:

   

7. You are now ready to use Tonika. To get started with adding friends, go to the Tonika Administration page by entering the URL http://a.5ttt.org in your browser.

Will it be open source?

Eventually and gradually, yes. Being a complex system, Tonika involves a handsome amount of code and making it open-source is not a simple undertaking. To make matters trickier, there are multiple protocols involved, which I hope I will be able to lay out in properly written specifications when time presents itself.

On the upside, a good chunk of Tonika is already open source. As a co-author of the Google Go language, I have contributed the entire HTTP machinery of Tonika (which is a big part of it) to the HTTP library of the Google Go language. Some of it is already part of the official releases of Google Go and some is still in the form of “change-lists” waiting for review.

To be even more precise, Tonika consist of multiple cooperating subsystems. I am looking into open-sourcing them one-by-one so that I can invite your contributions. It is just a matter of time as I find the best setup for open-sourcing Tonika.

Some theory

The core routing algorithm behind Tonika is based on a few (some of which yet unpublished) papers of mine. The first one in the sequence is:

• Electric Routing and Concurrent Flow Cutting, Jonathan Kelner, Petar Maymounkov — In proceedings of The 20-th International Symposium on Algorithms and Computation, and to appear in Elsevier Journal of Theoretical Computer Science.

The following papers will be published here as they become available.

Acknowledgements

Tonika comes in two pieces: the science and the system.

The mathematical problem of designing scalable and robust distributed routing algorithms for social networks has lingered in the academic community for over 10 years (essentially, since the advent of peer-to-peer systems), yet no great solutions have emerged. It is a hard problem. Had I not met my current academic adviser Prof. Jonathan Kelner, the algorithm behind Tonika may not have come to light. I am grateful for his guidance.

Regarding the systems behind Tonika, I would like to thank the Google Go Language team for creating the Go language and being so attentive to its first adopters. Tonika is 100% Go. Its internal workings are quite elaborate and very multi-threaded. I have spent almost an year investigating various other roads to implementing Tonika, none of which produced results in a reasonable timeframe under the one-programmer constraint. Go enabled me to implement the first cut of Tonika single-handedly in 4 months and has been a joy to use.