Use IDA Pro and/or BinDiff to inspect releases

added component::applications/quality assurance and testing owner::cypherpunks priority::high resolution::worksforme severity::normal status::closed type::project volunteer labels

Trac:
Parent: N/A to #5292 (moved)

Trac:
Keywords: N/A deleted, volunteer added
Summary: Recruit volunteers with access to IDA Pro and/or BinDiff to inspect releases to Use IDA Pro and/or BinDiff to inspect releases

Trac:
Description: We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky and may require its own instructions which Sebastian is working on writing for Windows and Linux, and already exist in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups for Mac OS.
BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should be substantially more similar than Windows, especially if you build on Debian/Lenny or MacOS 10.7 with Xcode 4. Start with using the 'cmp' (or hexdump and diff) Unix commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Text diff the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing existing, signed TBB copies from https://www.torproject.org/dist/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today.

Note that the above is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

to

We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky and may require its own instructions which Sebastian is working on writing for Windows and Linux, and already exist in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups for Mac OS.
BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should be substantially more similar than Windows, especially if you build on Debian/Lenny or MacOS 10.7 with Xcode 4. Start with using the 'cmp' (or hexdump and diff) Unix commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Text diff the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today.

Note that the above is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

Trac:
Description: We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky and may require its own instructions which Sebastian is working on writing for Windows and Linux, and already exist in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups for Mac OS.
BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should be substantially more similar than Windows, especially if you build on Debian/Lenny or MacOS 10.7 with Xcode 4. Start with using the 'cmp' (or hexdump and diff) Unix commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Text diff the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today.

Note that the above is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

to

We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky. Sebastian has instructions for Mac OS in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups. Windows and Linux instructions should arrive there soon.
BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should produce substantially more similar builds than Windows, especially if you build the Linux TBB on Debian Lenny and the Mac one on MacOS 10.7 with Xcode 4 (which are our build machine setups). Start with using the 'cmp' (and/or hexdump and diff) UNIX commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections. More advanced tools to find minimal binary edit differences may also be useful if the differences the basic tools find are large. http://jojodiff.sourceforge.net/ is an example of one such tool, but there probably are others.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Use normal UNIX diff on the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today.

Note that the above is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

Trac:
Description: We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky. Sebastian has instructions for Mac OS in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups. Windows and Linux instructions should arrive there soon.
BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should produce substantially more similar builds than Windows, especially if you build the Linux TBB on Debian Lenny and the Mac one on MacOS 10.7 with Xcode 4 (which are our build machine setups). Start with using the 'cmp' (and/or hexdump and diff) UNIX commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections. More advanced tools to find minimal binary edit differences may also be useful if the differences the basic tools find are large. http://jojodiff.sourceforge.net/ is an example of one such tool, but there probably are others.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Use normal UNIX diff on the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today.

Note that the above is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

to

We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky. Sebastian has instructions for Mac OS in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups. Windows and Linux instructions should arrive there soon.
BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should produce substantially more similar builds than Windows, especially if you build the Linux TBB on Debian Lenny and the Mac one on MacOS 10.7 with Xcode 4 (which are our build machine setups). Start with using the 'cmp' (and/or hexdump and diff) UNIX commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections. More advanced tools to find minimal binary edit differences may also be useful if the differences the basic tools find are large. http://jojodiff.sourceforge.net/ is an example of one such tool, but there probably are others.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Use normal UNIX diff on the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today. The source tarballs are in that directory as well, with -src.tar.gz instead of a language and arch. However, to get the actual git sources for an old release:

git clone git://git.torproject.org/torbrowser.git
cd torbrowser
git tag -l
git checkout torbrowser-2.2.35-9.1

Git will tell you that you can't commit anything that way, but that shouldn't matter to you.

Note that this whole process is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

Trac:
Cc: Sebastian to Sebastian, g.koppen@jondos.de

If you succeed in producing an independent build, it will be useful to post a link to it here for others to begin analyzing. However, you'll want to use a VM when inspecting random anonymous people's builds: IDA Pro is of course not immune to exploits against itself.

Trac:
Description: We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky. Sebastian has instructions for Mac OS in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups. Windows and Linux instructions should arrive there soon.
BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should produce substantially more similar builds than Windows, especially if you build the Linux TBB on Debian Lenny and the Mac one on MacOS 10.7 with Xcode 4 (which are our build machine setups). Start with using the 'cmp' (and/or hexdump and diff) UNIX commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections. More advanced tools to find minimal binary edit differences may also be useful if the differences the basic tools find are large. http://jojodiff.sourceforge.net/ is an example of one such tool, but there probably are others.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Use normal UNIX diff on the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today. The source tarballs are in that directory as well, with -src.tar.gz instead of a language and arch. However, to get the actual git sources for an old release:

git clone git://git.torproject.org/torbrowser.git
cd torbrowser
git tag -l
git checkout torbrowser-2.2.35-9.1

Git will tell you that you can't commit anything that way, but that shouldn't matter to you.

Note that this whole process is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

to

We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky. Sebastian has instructions for Mac OS in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups. Windows and Linux instructions should arrive there soon.

If you succeed in producing an independent build, it will be useful to post a link to it here for others to begin analyzing. However, you'll want to use a VM when inspecting random anonymous people's builds: IDA Pro is of course not immune to exploits against itself.

BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should produce substantially more similar builds than Windows, especially if you build the Linux TBB on Debian Lenny and the Mac one on MacOS 10.7 with Xcode 4 (which are our build machine setups). Start with using the 'cmp' (and/or hexdump and diff) UNIX commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections. More advanced tools to find minimal binary edit differences may also be useful if the differences the basic tools find are large. http://jojodiff.sourceforge.net/ is an example of one such tool, but there probably are others.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Use normal UNIX diff on the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today. The source tarballs are in that directory as well, with -src.tar.gz instead of a language and arch. However, to get the actual git sources for an old release:

git clone git://git.torproject.org/torbrowser.git
cd torbrowser
git tag -l
git checkout torbrowser-2.2.35-9.1

Git will tell you that you can't commit anything that way, but that shouldn't matter to you.

Note that this whole process is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

Trac:
Cc: Sebastian, g.koppen@jondos.de to Sebastian, g.koppen@jondos.de, tom@ritter.vg

Changes to description: Give a few more hints to illuminate the diffing process. We may not actually want to rely fully on what BinDiff and IDA say.

Trac:
Description: We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky. Sebastian has instructions for Mac OS in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups. Windows and Linux instructions should arrive there soon.

If you succeed in producing an independent build, it will be useful to post a link to it here for others to begin analyzing. However, you'll want to use a VM when inspecting random anonymous people's builds: IDA Pro is of course not immune to exploits against itself.

BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should produce substantially more similar builds than Windows, especially if you build the Linux TBB on Debian Lenny and the Mac one on MacOS 10.7 with Xcode 4 (which are our build machine setups). Start with using the 'cmp' (and/or hexdump and diff) UNIX commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections. More advanced tools to find minimal binary edit differences may also be useful if the differences the basic tools find are large. http://jojodiff.sourceforge.net/ is an example of one such tool, but there probably are others.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Use normal UNIX diff on the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today. The source tarballs are in that directory as well, with -src.tar.gz instead of a language and arch. However, to get the actual git sources for an old release:

git clone git://git.torproject.org/torbrowser.git
cd torbrowser
git tag -l
git checkout torbrowser-2.2.35-9.1

Git will tell you that you can't commit anything that way, but that shouldn't matter to you.

Note that this whole process is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

to

We have some concerning issues with our build processes. It's hard to really know for sure that the build machines are secure and not tampered with. See for example #5689 (closed). The most likely situation there is that AV signature engines fucking suck, and were false positives. However, the real fix is not simply scanning the build machines with more shitty AV software (although it was a fine, rational first step -- I'm not hating on you, Erinn). The real fix is unfortunately much harder: We need to ensure that the same source produces the same binaries everywhere: #3688 (closed).

Other than build reproduction, it's nearly fucking impossible to know that the binaries you get actually come from the source you wrote and audited. A handful of Open Source projects have figured it out (see comments in #3688 (closed)), but it's actually pretty darn hard on Windows, if not impossible.

As a stop-gap solution, it's occurred to me that we might be able to crowdsource hobbyists and people looking to learn reversing (or if we're really lucky, some people who already have these skills) to both help verify previous builds, and give us the input we need to proceed on #3688 (closed).

If you have access to IDA Pro (BinDiff will help, but is an extra $200 extension to IDA Pro and is not required), you can participate as follows:

Download a signed TBB release from https://www.torproject.org/dist/torbrowser/.
Compile your own TBB bundle. This is somewhat tricky. Sebastian has instructions for Mac OS in https://gitweb.torproject.org/torbrowser.git/tree/master:/docs/buildmachine_setups. Windows and Linux instructions should arrive there soon.
Analyze/BinDiff the resulting packaging exe as well as the exes contained therein.

If you lack BinDiff, you should consider focusing your efforts on MacOS and Linux packages, which should produce substantially more similar builds than Windows, especially if you build the Linux TBB on Debian Lenny and the Mac one on MacOS 10.7 with Xcode 4 (which are our build machine setups). Start with using the 'cmp' (and/or hexdump and diff) UNIX commands to find the differing sections of the binaries, and then use IDA Pro to inspect those sections (quickest way: Just use the search tool for a long hex string surrounding the difference).

More advanced tools to find minimal binary edit differences may also be useful if the differences the basic tools find are large. http://jojodiff.sourceforge.net/ is an example of one such tool, but there probably are others.

Be sure to verify the difference that IDA and/or BinDiff decodes is the same as a simple diff tool finds (cmp or jojodiff). The simple diff tool is less likely to be fooled than IDA.

You should be able to use IDA Pro for any platform to analyze binaries for any other platform. You do not need to buy the MacOS copy to analyze MacOS binaries.

Use normal UNIX diff on the non-exe components (for example, conf files, shell scripts, contents of Firefox jars and addon sources).
Write up the significant differences.

We need to make the above a semi-regular part of our development process to periodically verify our build integrity.

Writing up the differences you find will also be useful for us to help us work towards #3688 (closed), especially for Mac and Linux builds (which are likely pretty darn close.. It's probably only some timestamps and symbol names that differ if you use the same platform and compiler).

If you want to be extra-helpful, you can write up a detailed howto (and/or just paste the ones you find that already exist) on the process itself to make it even easier for others to learn and participate.

We'll need to do this often.. Perhaps not for every release, but at least randomly on some builds every once and a while. Also, BinDiffing archived signed TBB copies from https://archive.torproject.org/tor-package-archive/torbrowser/ will be useful, too, so no need to wait for future releases. You can start today. The source tarballs are in that directory as well, with -src.tar.gz instead of a language and arch. However, to get the actual git sources for an old release:

git clone git://git.torproject.org/torbrowser.git
cd torbrowser
git tag -l
git checkout torbrowser-2.2.35-9.1

Git will tell you that you can't commit anything that way, but that shouldn't matter to you.

Note that this whole process is a fantastic exercise in learning basic reverse engineering skills. Diffing vendor patches and malware updates is a common practice in the field. It's a useful skill to learn, if you have the time.

Good luck!

I have IDA Pro on OS X and am happy to help, but don't have a license to BinDiff, is there another plugin/addon that can be used?

I have used Patch2Diff but I realize that just shows difference in functions vs. raw opcode changes etc...

Trac:
Username: Tanaris4

Replying to Tanaris4:

I have IDA Pro on OS X and am happy to help, but don't have a license to BinDiff, is there another plugin/addon that can be used?

See instructions in the ticket description. I've updated them a couple of times, and they should now cover this. If they don't please ask for specific clarification.

I have used Patch2Diff but I realize that just shows difference in functions vs. raw opcode changes etc...

Awesome. If you get to this point, you're ahead of the curve. Please just tell us offsets of the differences. I can take it from there.

If you're not at that point yet, just post a binary somewhere. Every that would save us precious time.

Another fine tool would be too use http://www.darungrim.org/ which is free and other people might me more familiar with.

If any differences are found with these tools (bindiff as example here), in what type of format should we upload the results? .idb ida files?

Cheers.

I just want to echo mikeperry's "just post a binary somewhere." I've never gotten a sufficiently complicated build process on Windows to work, ever - but I have IDA and BinDiff and can produce some results for people to look at fairly quickly if someone can give me a binary.

Ok, so to help with build reproduction part: The Windows TBBs are built with Mingw "0.1-alpha5" that was installed on Feb 11, 2011. The gcc in use calls itself 4.5.2. The Windows build machine itself uses Windows 7 Professional.

However, Sebastian and Erinn are going to rebuild the build machine to better document the full state of all of the toolchain components in use for future releases. I imagine it will show up in that buildmachine_setup directory where the osx description is.

Trac:
Cc: Sebastian, g.koppen@jondos.de, tom@ritter.vg to Sebastian, g.koppen@jondos.de, tom@ritter.vg, erinn

Trac:
Username: Tichodroma
Cc: Sebastian, g.koppen@jondos.de, tom@ritter.vg, erinn to Sebastian, g.koppen@jondos.de, tom@ritter.vg, erinn, tichodroma@posteo.de

Trac:
Cc: Sebastian, g.koppen@jondos.de, tom@ritter.vg, erinn, tichodroma@posteo.de to Sebastian, g.koppen@jondos.de, tom@ritter.vg, erinn, tichodroma@posteo.de, drwhax@2600nl.net

Trac:
Component: Tor bundles/installation to Quality Assurance and Testing

Trac:
Cc: Sebastian, g.koppen@jondos.de, tom@ritter.vg, erinn, tichodroma@posteo.de, drwhax@2600nl.net to Sebastian, g.koppen@jondos.de, tom@ritter.vg, erinn, tichodroma@posteo.de, drwhax@2600nl.net, Shondoit

Woo-hoo! Shondoit posted independent re-builds of the latest TBB stable Windows and Linux in #3688 (closed) (https://people.torproject.org/~shondoit/builds/).

https://code.google.com/p/patchdiff2/ is apparently a free IDA Pro plugin for diffing binaries. BinDiff itself appears to still be commercial.

I don't have a smoking gun "safe" or "not safe" (obviously) - what I do have is some prelimiary results that hopefully will help with the process. I do have BinDiff. I tried to generate reports for non-matching dlls/binaries, but bindiff hung. I'm investigating. Here's what I do have:

http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/

The installer matched perfectly: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/(tbb)tor-browser-2.2.37-2_en-US%20vs%20(tbb)shon-tor-browser-2.2.37-2_en-US%20difference.html

As did tbb-firefox: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/(Firefox)tbb-firefox%20vs%20(Firefox)tbb-firefox%20difference.html

And plugin-container: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/(Firefox)plugin-container%20vs%20(Firefox)plugin-container%20difference.html

libeay had a lot of changes. But it seemed to be mostly due to inlining on a couple I spot-checked. For example: On the left, inlined, on the right: no. http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/libeay-digestfinal-bindiff.PNG Official inlined: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/libeay-digestfinal-official.PNG Shon's Not Inlined: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/libeay-digestfinal-shon.PNG Official's _ex function: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/libeay-digestfinal_ex-official.PNG Shon's ex function: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/libeay-digestfinal_ex-shon.PNG

That makes a lot of things to go through. There may be some way to do this in an automated way (in fact, I'm sure there is, and Rolf Rolles has probably done it) but I'm not good enough/don't have to free time to figure it out....

Tor: 438 out of 3560 matched at 95% similarity or less. 40 unmatched in official, 22 in shon. Here's an example of one difference: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/tor-tlscallback-bindiff.PNG

I also went after the monster, xul.dll: 3,241 of 79970 functions did not match perfectly, 24 unmatched in official, 28 in shon There's definetly some odd things things, like this: http://ritter.vg/misc/stuff/tbb-diff-2012-08-02/xul-random1-bindiff.PNG

It sounds like a lot of the above are due to either differing compiler versions, or different compiler options.

Shondoit: Did you use the official build scripts + options to build those? Is it just compiler version changes that likely caused most of the above?

Replying to mikeperry:

Shondoit: Did you use the official build scripts + options to build those? Is it just compiler version changes that likely caused most of the above? I definitely used the official build scripts. I even checked out the 2.2.37-2 tagged commit. Only the build setup should be different.

Is there anything I can do to help here? Would it be useful for me to rebuild that tag to see if I get differences from the same build machine?

Replying to erinn:

Is there anything I can do to help here? Would it be useful for me to rebuild that tag to see if I get differences from the same build machine?

Because the end goal is determining whether or not the build process is tampered with, performing bindiffs will not do it. There are techniques to backdoor a binary without modifying the primary code (via DWARF exception chains. I'm sure there are others.) It'd have to be via hash matches or a different comparison process.

However, I think step 1 to getting identical binaries from two build machines is getting identical assembly. So if anyone can post binaries from two different machines that were built with the same compiler flags and the same compiler version I (or anyone) could do the bindiff process again and we can see how close that gets us. It may be necessary to use the compiler options mentioned in #3688 (closed).

As it is, the quantity of changes seen above make it unrealistic for a manual diff comparison.

Replying to erinn:

Is there anything I can do to help here? Would it be useful for me to rebuild that tag to see if I get differences from the same build machine? Maybe you can give --version outputs for the different tools? That way I can try to build as close to official as possible. Seeing as Tom compared Windows versions we can start with that.

Also, doing another build from the same tag is appreciated. It will reveal temporal differences which would definitely be of interest. I know at least Firefox has some diffs regarding the build date.

-- Shondoit

Is this ticket still actual for reproducible builds? Probably to transform this to analysis of decompilated binaries, as long as they are reproducible but not true deterministic yet. If all compilers in the whole universe was tampered to tamper only specific target then no test case could help to find it, only independent reverse engineering of bundle could help.

Lets to fill ticket by links to open-source tools with academic background or free ones at least.

I think this can indeed get closed thanks to our reproducible builds.

Trac:
Cc: Sebastian, g.koppen@jondos.de, tom@ritter.vg, erinn, tichodroma@posteo.de, drwhax@2600nl.net, Shondoit to Sebastian, gk, tom@ritter.vg, erinn, tichodroma@posteo.de, drwhax@2600nl.net, Shondoit
Severity: N/A to Normal
Reviewer: N/A to N/A
Sponsor: N/A to N/A
Resolution: N/A to worksforme
Status: new to closed

Trac:
Parent: #5292 (moved) to N/A

closed

mentioned in issue #5863 (closed)

moved to tpo/applications/tor-browser-bundle-testsuite#5837 (closed)

Use IDA Pro and/or BinDiff to inspect releases

Child items ...

Activity