Opened 4 years ago

Last modified 17 months ago

#13770 new enhancement

BusyBox-style bundling of Go programs can save space

Reported by: dcf Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords:
Cc: gk Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Mike suggested that compiling all the Go programs into one executable à la BusyBox could save space because there would be only one static copy of the Go runtime.

Child Tickets

Attachments (1)

mkmulti (1.7 KB) - added by dcf 4 years ago.
Shell script that generates source for a bundled executable.

Download all attachments as: .zip

Change History (7)

Changed 4 years ago by dcf

Attachment: mkmulti added

Shell script that generates source for a bundled executable.

comment:1 Changed 4 years ago by dcf

Here's a script that generates a multi-executable from Go programs. I tested it on obfs4proxy, meek-client, meek-client-torbrowser, and the new tor-fw-helper (#13338). With these four programs, bundling saves this much in the linux build:

  • uncompressed: saves 10 MB (16 MB goes to 5.8 MB)
  • compressed: saves 0.7 MB (2.0 MB goes to 1.3 MB)

attachment:mkmulti is a shell script. Run it like this to create the source for a multi-executable:

export GOPATH=~/go
PROGS="git.torproject.org/pluggable-transports/obfs4.git/obfs4proxy git.torproject.org/pluggable-transports/meek.git/meek-client git.torproject.org/pluggable-transports/meek.git/meek-client-torbrowser github.com/yawning/tor-fw-helper"
go get -d $PROGS
./mkmulti multi $PROGS

The script writes a source file to multi/multi.go. Build it using the same flags we use in tor-browser-bundle.

cd multi
go build -ldflags '-s'

This is the multi-executable:

 5992352 (5.8M) multi

And these are the constituent programs compiled separately:

 4531488 (4.4M) meek-client
 2090368 (2.0M) meek-client-torbrowser
 5074976 (4.9M) obfs4proxy
 4603264 (4.5M) tor-fw-helper
---------------------
16300096  (16M) total

Here are the compressed sizes:

 1263988 (1.3M) multi.tar.xz
 2013728 (2.0M) separate.tar.xz

comment:2 Changed 4 years ago by dcf

On linux and mac, we can use symbolic links to the executable. I don't know what is done on Windows. Maybe batch files, or stub programs that change argv[0] and then exec?

comment:3 Changed 4 years ago by gk

Cc: gk added

comment:4 in reply to:  2 ; Changed 4 years ago by yawning

Hey, that's really neat.

Replying to dcf:

On linux and mac, we can use symbolic links to the executable. I don't know what is done on Windows. Maybe batch files, or stub programs that change argv[0] and then exec?

NTFS kind of sort of has symlinks but assuming that the target filesystem is NTFS instead of FAT32/exFAT will end in tears.

How about adding code to the begining of main() that:

  • Looks at os.Args[1] for --progname=<name> (Do this manually, and not with the flag package.)
    • If present (and name in progName), do something like:
      // Rewrite argv[0], and strip out progname from the arguments, so that using the
      // 'flag' package doesn't break.
      newArgs := make([]string, 0, 1 + len(os.Args) - 2)
      newArgs = append(newArgs, progName)
      if len(os.Args) > 2 {
        newArgs = append(newArgs, os.Args[2:]...)
      }
      os.Args = newArgs
      

Edit the Windows torrc as appropriate. It's a bit kludgey, but it probably beats batchfiles and stub executables, and it's relatively robust, and having that there is relatively harmless across all platforms (assuming that missing argument is treated as a non-fatal error).

comment:5 in reply to:  4 Changed 4 years ago by dcf

Replying to yawning:

Hey, that's really neat.

Replying to dcf:

On linux and mac, we can use symbolic links to the executable. I don't know what is done on Windows. Maybe batch files, or stub programs that change argv[0] and then exec?

NTFS kind of sort of has symlinks but assuming that the target filesystem is NTFS instead of FAT32/exFAT will end in tears.

How about adding code to the begining of main() that:

  • Looks at os.Args[1] for --progname=<name> (Do this manually, and not with the flag package.)

Or just always switch on argv[1], and ignore argv[0]. For example,

ClientTransportPlugin obfs4 exec TorBrowser\Tor\PluggableTransports\multi obfs4proxy

and do the same thing on all platforms:

ClientTransportPlugin obfs4 exec ./TorBrowser/Tor/PluggableTransports/multi obfs4proxy

comment:6 Changed 17 months ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

Note: See TracTickets for help on using tickets.