29 Jan 2012

SMTP Logger in GO

Go is really intuitive to write and one thing I like most is the interface design.

So I wrote a SMTP log handler that has a Write method, meaning it implements io.Writer, and can be used anywhere that requires io.Writer, e.g. log.new:

func New(out io.Writer, prefix string, flag int) *Logger

So that I can use:

logger := log.New(sw, "SMTPLOG: ", 0)
logger.Print("This is a test sent from SMTP Logger\n\nnemo")

sw is a SMTP log handler:

type SMTPWriter struct {
    auth                   smtp.Auth
    user, pass, host, port string
    sendto                 []string
    subject                string
}

func (w *SMTPWriter) Write(b []byte) (n int, err error) {
    header := fmt.Sprintf(`Subject: %s
From: %s
To: %s
Content-Type: text/plain; charset=ISO-8859-1

`, w.subject, w.user, strings.Join(w.sendto, ";"))
    err = smtp.SendMail(
        w.host+":"+w.port,
        w.auth,
        w.user,
        w.sendto,
        append([]byte(header), b...),
    )
    if err != nil {
        return 0, err
    }
    return len(b), nil
}

func NewSMTPWriter(user, pass, host, port string,
    sendto []string, subject string) *SMTPWriter {
    var auth = smtp.PlainAuth("", user, pass, host)
    return &SMTPWriter{auth, user, pass, host, port, sendto, subject}
}
sw := NewSMTPWriter(
    "test@jlee.co",    // user
    "test@jlee.co",    // pass
    "smtp.gmail.com",  // server
    "587",             // port
    []string{"me@jlee.co", "jackie.space@gmai.com"},  // sendto
    "Testing smtpwriter in a logger",                 // subject
)
And then like what you expect, you can use:

logger.Print("This is a test sent from SMTP Logger\n\nnemo")

But it'll send a email on every call on logger's method. That's too verbose. I want to send a email when there is say 1024 chars.
Go has a bufio package. We can wrap our SMTPWriter using bufio:

func NewBufSMTPWriter(user, pass, host, port string,
    sendto []string, subject string, bufsize int) (*bufio.Writer, error) {
    sw := NewSMTPWriter(user, pass, host, port, sendto, subject)
    w, err := bufio.NewWriterSize(sw, bufsize)
    if err != nil {
         return nil, err
    }
    return w, nil
}


See? you don't have to define a new BufSMTPWriter, just return any struct that has a Write method, here I have a *buifio.Writer return type, you can even use io.Writer return type, because the target method call only cares if you have a Write method.

the code is at https://github.com/jackielii/smtpwriter

9 Jan 2012

errno 530 when building go program

So I was trying to get my pulleb program work on my office machine. I built my program on my virtual machine, 32bit Debian, and copied the binary to the work machine, ran it. It says "errno 530". Go is statically linked, so it can't be the library problem, so it might has something to do with the system compatibility.
So the `uname -a` on the office machine is:
Linux hostname 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686 i686 i386 GNU/Linux

$cat /etc/issue
Red Hat Enterprise Linux WS release 4 (Nahant Update 4)
Kernel \r on an \m
Then I downloaded go source and tried building the compiler, and got the same error:
...
# Building go command from bootstrap script.
# Building Go code.
# net
open cgo_linux.go: errno 530
# os/user
open lookup_unix.go: errno 530
...
WHAT?!!!
Then I filed a issue: http://code.google.com/p/go/issues/detail?id=2640
and turns out this is an issue go deep in the kernel.
RHEL uses a customised kernel and changed libc constants.

So either I can upgrade the kernel, or there is a workaround: 
in $GOROOT/src/pkg/syscall/zerrors_linux_386.go, change:
    O_CLOEXEC = 0x80000
to
    O_CLOEXEC = 0 
And if there are more error, try:
export CGO_ENABLED=0
Reference: 
[1] http://code.google.com/p/go/issues/detail?id=293
[2] http://sourceware.org/bugzilla/show_bug.cgi?id=5227

1 Jan 2012

Golang: read IBM 32-bit floating point in Go, and compare with python

Since I'm learning Go, so rewrote reading IBM 32bit floating point as in last post in GO:

package ibmtools

import "math"

const p24 = 2 << 23
func Ibm32frombits(b [4]byte) float64 {
 ui := uint32(b[3]) | uint32(b[2])<<8 | uint32(b[1])<<16 | uint32(b[0])<<24
 sign := 1 - 2*float64(ui>>31&0x01)
 exponent := math.Pow(16.0, float64(ui>>24&0x7f)-64.0)
 mantissa := float64(ui&0x00ffffff) / float64(p24)
 return math.Copysign(mantissa*exponent, sign)
}

You can go to https://github.com/jackielii/ibmtools to view the full source or use gointall to install it.

I also did some speed tests:

Python:

p24 = pow(2, 24)
def ibm2ieee(ibm):
    """ read an IBM floating point number. """
    sign = ibm >> 31 & 0x01
    exponent = ibm >> 24 & 0x7f
    mantissa = ibm & 0x00ffffff
    mantissa = (mantissa * 1.0) / p24
    ieee = (1 - 2 * sign) * mantissa * pow(16, exponent - 64)
    return ieee

And run it using:

nemo@EliteBook:~/workspace$ python -m timeit "from testibm import ibm2ieee" "ibm2ieee(0b11000010011101101010000000000000)"
1000000 loops, best of 3: 1.62 usec per loop

Test Go using:

package ibmtools

import (
 "testing"
 // "strconv"
)

var (
 bit = "11000010011101101010000000000000"
 by  = [4]byte{194, 118, 160, 0}
)

func BenchmarkIbm2ieee(b *testing.B) {
 for i := 0; i < b.N; i++ {
  Ibm32frombits(by)
 }
}
nemo@EliteBook:~/workspace/ibmtools$ gomake bench
gotest -test.bench=. -test.run="Do not run tests"
rm -f _test/ibmtools.a
6g   -p ibmtools -o _gotest_.6 ebcdic.go ibmfloat.go  ibmtools_test.go
rm -f _test/ibmtools.a
gopack grc _test/ibmtools.a _gotest_.6 
PASS
ibmtools.BenchmarkIbm2ieee 20000000         90.3 ns/op

Result: Python 1.62 us = 1620 ns, and Go 90.3 ns. Go is around 17 times faster.

I know it's not a fair comparison, but it's illustrative.

python: unpack IBM 32-bit float point

IBM floating point 32-bit numbers cannot be parsed directly in python, you need to unpack it by yourself. The format is described here:

   1     7               24              width in bits
  +-+-------+------------------------+
  |S|  Exp  |   Fraction             |
  +-+-------+------------------------+
  31 30   24 23                     0    bit index (0 on right)
     bias +64

So what we need to do is unpack the byte stream to unsigned 32 bit integer, and calculate each part of the above format. We'll use the example in http://en.wikipedia.org/wiki/IBM_Floating_Point_Architecture#An_Example

In the example the nibbles look like this:

   1     7               24              width in bits
  +-+-------+------------------------+
  |S|  Exp  |  Fraction              |
  |1|1000010|011101101010000000000000|
  +-+-------+------------------------+
  31 30   24 23                     0    bit index (0 on right)
     bias +64

Expected output is -118.625

>>> import struct
>>> bit = '11000010011101101010000000000000'
>>> n = int(bit, 2)                               # convert bit string to integer
>>> sign = n >> 31 & 0x01
>>> sign = 1 - sign * 2                           # 1 negative, 0 positive
>>> exponent = n >> 24 & 0x7f
>>> exponent = pow(16, exponent - 64)             # exponents are power of 16 not 2, and bias is 64
>>> mantissa = n & 0x00ffffff                     # bit representation of fraction part in the above graph
>>> mantissa = float(mantissa) / pow(2, 24)       # convert fraction(mantissa) part to its real value
>>> sign * exponent * mantissa
-118.625

So everything in a function:

def ibm2ieee(ibm):
    """
    Converts an IBM floating point number into IEEE format.
    :param: ibm - 32 bit unsigned integer: unpack('>L', f.read(4))
    """

    sign = ibm >> 31 & 0x01
    exponent = ibm >> 24 & 0x7f
    mantissa = ibm & 0x00ffffff
    mantissa = (mantissa * 1.0) / pow(2, 24)
    ieee = (1 - 2 * sign) * mantissa * pow(16, exponent - 64)
    return ieee

I also have a post in stackoverflow.com that implemented a class to unpack IBM 32-bit float: http://stackoverflow.com/questions/7125890/python-unpack-ibm-32-bit-float-point

Reference:

[1]http://mail.scipy.org/pipermail/scipy-user/2009-January/019392.html

[2]http://en.wikipedia.org/wiki/IBM_Floating_Point_Architecture