BusyBox-style bundling of Go programs can save space

[dcf]

Mike suggested that compiling all the Go programs into one executable à la BusyBox could save space because there would be only one static copy of the Go runtime.

[dcf]

Shell script that generates source for a bundled executable.

[dcf]

Here's a script that generates a multi-executable from Go programs. I tested it on obfs4proxy, meek-client, meek-client-torbrowser, and the new tor-fw-helper (#13338). With these four programs, bundling saves this much in the linux build:

• uncompressed: saves 10 MB (16 MB goes to 5.8 MB)
• compressed: saves 0.7 MB (2.0 MB goes to 1.3 MB)

attachment:mkmulti​ is a shell script. Run it like this to create the source for a multi-executable:

export GOPATH=~/go
PROGS="git.torproject.org/pluggable-transports/obfs4.git/obfs4proxy git.torproject.org/pluggable-transports/meek.git/meek-client git.torproject.org/pluggable-transports/meek.git/meek-client-torbrowser github.com/yawning/tor-fw-helper"
go get -d $PROGS
./mkmulti multi $PROGS

The script writes a source file to multi/multi.go. Build it using the same flags we use in tor-browser-bundle.

cd multi
go build -ldflags '-s'

This is the multi-executable:

 5992352 (5.8M) multi

And these are the constituent programs compiled separately:

 4531488 (4.4M) meek-client
 2090368 (2.0M) meek-client-torbrowser
 5074976 (4.9M) obfs4proxy
 4603264 (4.5M) tor-fw-helper
---------------------
16300096  (16M) total

Here are the compressed sizes:

 1263988 (1.3M) multi.tar.xz
 2013728 (2.0M) separate.tar.xz

[dcf]

On linux and mac, we can use symbolic links to the executable. I don't know what is done on Windows. Maybe batch files, or stub programs that change argv[0] and then exec?

[yawning]

Hey, that's really neat.

Replying to dcf:

On linux and mac, we can use symbolic links to the executable. I don't know what is done on Windows. Maybe batch files, or stub programs that change argv[0] and then exec?

NTFS kind of sort of has symlinks but assuming that the target filesystem is NTFS instead of FAT32/exFAT will end in tears.

How about adding code to the begining of main() that:

• Looks at os.Args[1] for --progname=<name> (Do this manually, and not with the flag package.)
  • If present (and name in progName), do something like:

    // Rewrite argv[0], and strip out progname from the arguments, so that using the
    // 'flag' package doesn't break.
    newArgs := make([]string, 0, 1 + len(os.Args) - 2)
    newArgs = append(newArgs, progName)
    if len(os.Args) > 2 {
      newArgs = append(newArgs, os.Args[2:]...)
    }
    os.Args = newArgs
    

Edit the Windows torrc as appropriate. It's a bit kludgey, but it probably beats batchfiles and stub executables, and it's relatively robust, and having that there is relatively harmless across all platforms (assuming that missing argument is treated as a non-fatal error).

[dcf]

Replying to yawning:

Hey, that's really neat.

Replying to dcf:

On linux and mac, we can use symbolic links to the executable. I don't know what is done on Windows. Maybe batch files, or stub programs that change argv[0] and then exec?

NTFS kind of sort of has symlinks but assuming that the target filesystem is NTFS instead of FAT32/exFAT will end in tears.

How about adding code to the begining of main() that:

• Looks at os.Args[1] for --progname=<name> (Do this manually, and not with the flag package.)

Or just always switch on argv[1], and ignore argv[0]. For example,

ClientTransportPlugin obfs4 exec TorBrowser\Tor\PluggableTransports\multi obfs4proxy

and do the same thing on all platforms:

ClientTransportPlugin obfs4 exec ./TorBrowser/Tor/PluggableTransports/multi obfs4proxy

[teor]

Set all open tickets without a severity to "Normal"