Today, I installed Atom! It’s nice… and certainly better than blogging using vi but I have to say the documentation isn’t the greatest!

I’ll hack around at it and see if I can make it a permanent part of my blogging toolbag

I read a great article summarizing the interrelationship between go routines, channel communications, unix signals and proper termination by Adam Presley. In his article he describes how to coordinate starting a go routine, handling an interrupt, and then communicating through a channel to the go routine that the program wants to terminate.

So what do you do if you have more than one go routine? You need to communicate to all and wait for all when you quit. Here’s a contrived example that demonstrates one way to do it.

One point to note. In this example we don’t distinguish between which routines we wish to quit in any particular order. In fact, as implemented here, there is no deterministic way of knowing the order (How might you implement the code so that you would be able to deterministically know the order of go routine termination?)

Here’s an example that demonstrates how you might handle an arbitrary number of go routines:

Let’s say we start off with a constant number of routines we wish to create:

1
2
3

const (
        maxGoRoutines = 50
)

We will start a go routine maxGoRoutines times and then we will ensure that we wait for the same number of routines to complete by using a waitGroup

1
2

waitGroup := &sync.WaitGroup{}
waitGroup.Add(maxGoRoutines)

Now let’s define a simple go routine. We’ll pass a channel to let us know when to quit, a waitGroup to indicate that we’ve quit once we’ve left the routine, and an identifier to distinguish between go routines to make our demo look cool!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

go func(shutdownChannel chan bool, waitGroup *sync.WaitGroup, id int) {
  log.Println("Starting work goroutine...")
  defer waitGroup.Done()

  for {
      /*
      * Listen on channels for message.
      */
      select {
      case _ = <-shutdownChannel:
          log.Printf("Received shutdown on goroutine %d\n", id)
          return

      default:
      }

      // Do some hard work here!
  }
}(shutdownChannel, waitGroup, i)

Once we’ve launed the routines, we’ll wait for the program to terminate. We’ve established a signal handler to let us know when SIGTERM or SIGQUIT by the following lines:

1
2

quitChannel := make(chan os.Signal)
signal.Notify(quitChannel, syscall.SIGINT, syscall.SIGTERM)

Next, we’ll wait to receive a signal that we’ve quit by blocking on the quitChannel. Once we receive a message indicating that we’ve quit, we’ll send a boolean true to our go routine shutdownChannel. Notice that we have to send as many messages to this channel as we have go routines. Otherwise, we’ll leave go routines hanging around and that will block us from terminating.

And finally, we wait for the waitGroup to complete. After each go routine calls its defered waitGroup.Done() function, we will unblock on the waitGroup.Wait() and can successfully exit!

1
2

        waitGroup.Wait()
        log.Println("Done.")

Here’s the whole thing from soup to nuts!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

package main

import (
        "log"
        "os"
        "os/signal"
        "runtime"
        "sync"
        "syscall"
)

const (
        maxGoRoutines = 50
)

func main() {

        runtime.GOMAXPROCS(int(float64(runtime.NumCPU()) * 1.25))

        log.Println("Starting application...")

        /*
         * When SIGINT or SIGTERM is caught write to the quitChannel
         */
        quitChannel := make(chan os.Signal)
        signal.Notify(quitChannel, syscall.SIGINT, syscall.SIGTERM)

        shutdownChannel := make(chan bool)
        waitGroup := &sync.WaitGroup{}

        waitGroup.Add(maxGoRoutines)

        /*
         * Create a goroutine that does imaginary work
         */
        for i := 0; i < maxGoRoutines; i++ {
                go func(shutdownChannel chan bool, waitGroup *sync.WaitGroup, id int) {
                        log.Println("Starting work goroutine...")
                        defer waitGroup.Done()

                        for {
                                /*
                                 * Listen on channels for message.
                                 */
                                select {
                                case _ = <-shutdownChannel:
                                        log.Printf("Received shutdown on goroutine %d\n", id)
                                        return

                                default:
                                }

                                // Do some hard work here!
                        }
                }(shutdownChannel, waitGroup, i)
        }

        /*
         * Wait until we get the quit message
         */
        <-quitChannel

        log.Println("Received quit. Sending shutdown and waiting on goroutines...")

        for i := 0; i < maxGoRoutines; i++ {
                shutdownChannel <- true
        }

        /*
         * Block until wait group counter gets to zero
         */
        waitGroup.Wait()
        log.Println("Done.")
}

Sometimes you want to automate a workflow process by watching a filesystem directory and treating it like a work queue. As files arrive in the directory, a process watching the directory can be notified of new work items and kick off a process to perform the desired task.

A golang package called fsnotify does most of the nitty-gritty to make the details of implementing a queue manager somewhat trivial. Below is an example of how you would set up a queue to watch a directory using the package:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

   package main

  import (
      "log"

      "github.com/go-fsnotify/fsnotify"
  )

  func main() {

      watcher, err := fsnotify.NewWatcher()
      if err != nil {
          log.Fatal(err)
      }
      defer watcher.Close()

      done := make(chan bool)
      go func() {
          for {
              select {
              case event := <-watcher.Events:
                  log.Println("event:", event)
                  if event.Op&fsnotify.Write == fsnotify.Write {
                      log.Println("modified file:", event.Name)
                  }
              case err := <-watcher.Errors:
                  log.Println("error:", err)
              }
          }
      }()

      err = watcher.Add("/tmp/foo")
      if err != nil {
          log.Fatal(err)
      }
      <-done
  }

In the main of the golang program, we create a new watcher and start a go-routine that selects events from the watcher Events channel. Each event is logged and then if an event of type ‘fsnotiy.Write’ is received by the watcher, an additional log message is printed. Outside of the go-routine, we add the /tmp/foo directory to the watcher and wait on the done channel (which will block indefinitely).

Here’s a fun little git I came across from Antoine Grondin. It’s a cute little logfile color package to make your logfiles really pop!

Tired of those old boring foreground on background logs? Well, try this!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

   package main

  import (
      "log"
      "os"
      "time"

      "github.com/aybabtme/rgbterm/rainbow"
  )

  func main() {
      joyfulOutput := rainbow.New(os.Stderr, 252, 255, 43)
      log.SetOutput(joyfulOutput)

      for i := 0; i < 500; i++ {
          log.Printf("This is surely going to drive you nuts!\n")
          time.Sleep(250 * time.Millisecond)
      }

  }

Okay, that’s sure to get you screamed at if you actually used it. But a more useful package used by the rainbow package is the rgbterm package. This package lets you color any text and display it on stdout.

Here’s an example using this package:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

   package main

  import (
      "fmt"

      "github.com/aybabtme/rgbterm"
  )

  func main() {

      var r, g, b uint8
      // pick a color
      r, g, b = 255, 255, 255
      // choose a word
      word := "=)"
      // colorize it!
      coloredWord := rgbterm.String(word, r, g, b, r^r, g^g, b^b)

      fmt.Println("Oh!", coloredWord, "hello!")
  }

Enjoy!

Just for fun, I thought I’d play around with Alan Smith’s ACSLogo for Mac. This great program is a full-featured Logo interpreter that is perfectly suited for teaching basic programming and math concepts to people of all ages.

I’ve put together a repo on github that is a collection of fun and interesting procedures mostly do perform graphically interesting recusive problems. Feel free to add to the collection!

I strongly suggest start by reading the User Guide and the Command Reference documents to orient yourself to the Logo dialect.

Have Fun!

MongoDB and Go, a pair meant to be together!

I’ve recently been working on a start-up that’s afforded me the need to get better acquainted with Mongodb and Go. Quickly working with these two technologies, one feels that they were meant for each other. Throw in the json package and you’re cooking with Gasoline!

The fun begins when you go wild with the use of map[string]interface{} and []map[string]interface{}. At first glance to the noob, these are daunting looking type definitions. But with a little experimenting to get you legs underneath you, you’ll find that you coudn’t live without these guys! And for good measure, their big brother, interface{}, is pretty handy too when you want to generically throw these guys around and inspect them through Go’s reflect package to see what type you’re dealing with at runtime.

Here’s the driver that I find works nicely with mongodb

More later…

Building web services with Go is pretty straight-forward. See the godocs for http to see how to write both client and server code.

With a little more help, we can go beyond the standard Go packages and use a package web framework that accelerates the development of a web service stack. These frameworks do a few basic things:

1. Routing
2. Parameter Handling
3. JSON marshalling and unmarshalling
4. HTML templates and form processing

Below are two fo the frameworks that are VERY light weight and easy to use:

1. Martini
2. Gin

Basically both of these frameworks are very similar with the biggest difference being that Martini supports Dependency Injection. Gin uses a Context for its parameters.

1. Martini
2. Gin

Implementing and Benchmarking Bloom Filters in Golang:

Here’s a great article on impementing and benchmarking Bloom Filters in Golang.

Three separate types of filters are implements: Standard, Partitioned, and Scalable.

For a detailed look Read is paper.

Here’s a quick sample (credit: astaxie) that demonstrates how to upload a file in golang:

There are two things going on here: First is the handler section that processes the GET and then the section handling the POST (to be more accurate, we should check the r.Method once more to see if it’s exactly the POST method).

In the get, we generate a token to ensure that the file we receive is indeed the one requested from the GET. Although in this code, there isn’t a check to see that that’s indeed the case in the POST (left as an exercise for the reader).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

   http.HandleFunc("/upload", upload)

  // upload logic
  func upload(w http.ResponseWriter, r *http.Request) {
      fmt.Println("method:", r.Method)
      if r.Method == "GET" {
          crutime := time.Now().Unix()
          h := md5.New()
          io.WriteString(h, strconv.FormatInt(crutime, 10))
          token := fmt.Sprintf("%x", h.Sum(nil))

          t, _ := template.ParseFiles("upload.gtpl")
          t.Execute(w, token)
      } else {
          r.ParseMultipartForm(32 << 20)
          file, handler, err := r.FormFile("uploadfile")
          if err != nil {
              fmt.Println(err)
              return
          }
          defer file.Close()
          fmt.Fprintf(w, "%v", handler.Header)
      fmt.Printf("value=%s\n", r.FormValue("token"))
          f, err := os.OpenFile("./test/"+handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)
          if err != nil {
              fmt.Println(err)
              return
          }
          defer f.Close()
          io.Copy(f, file)
      }
  }

And the template above, upload.gtpl looks like:

1
2
3
4
5
6
7
8
9
10
11
12

 <html>
  <head>
      <title>Upload file</title>
  </head>
  <body>
  <form enctype="multipart/form-data" action="http://127.0.0.1:9090/upload" method="post">
          <input type="file" name="uploadfile" />
          <input type="hidden" name="token" value=""/>
          <input type="submit" value="upload" />
  </form>
  </body>
  </html>

Another RESTful golang JSON package is available to quickly support the development of RESTful JSON-based server apps.

git clone https://github.com/ant0ine/go-json-rest.git