Evaluate Tor's Embedding API when integrating Tor Launcher

[sysrqb]

We should look at the embedding API because maybe it is a better fit for us instead of fork/exec-ing a new process.

​https://lists.torproject.org/pipermail/tbb-dev/2018-July/000876.html

[gk]

sisbell: That might be something you may be interested in as well.

[sysrqb]

Moving to second-alpha.

[sysrqb]

Okay, this is looking good. #24204 is really nice.

I created a test program that takes the standard tor command line as its own command line arguments, but it runs tor_run_main() twice, and calls tor_configuration_set_command_line() twice, too.

First, it begins by taking the provided command line arguments and injecting --verify-config as the second argument (in a newly allocate argv). It then creates a control fd, and calls tor_run_main() using the modified command line for verifying the config is valid.

After that returns, it resets the configuration using the original provided command line and calls tor_run_main() again using the original arguments. After tor starts, SIGNAL TERM is sent on the control FD.

$ ./threaded_verify_config --Log "notice stdout" --AvoidDiskWrites 1 --SocksPort unix:${HOME}/test_socks --DisableNetwork 1
Running with 'tor 0.3.5.0-alpha-dev'
Sep 18 19:12:57.516 [notice] Tor 0.3.5.0-alpha-dev (git-b729bc202c74113c) running on Linux with Libevent 2.0.22-stable, NSS 3.36.1, Zlib 1.2.11, Liblzma N/A, and Libzstd N/A.
Sep 18 19:12:57.516 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Sep 18 19:12:57.516 [notice] This version is not a stable Tor release. Expect more bugs than usual.
Sep 18 19:12:57.516 [notice] Configuration file "/usr/local/etc/tor/torrc" not present, using reasonable defaults.
Configuration was valid
Tor returned successfully
Sep 18 19:12:57.572 [notice] Tor 0.3.5.0-alpha-dev (git-b729bc202c74113c) running on Linux with Libevent 2.0.22-stable, NSS 3.36.1, Zlib 1.2.11, Liblzma N/A, and Libzstd N/A.
Sep 18 19:12:57.572 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Sep 18 19:12:57.572 [notice] This version is not a stable Tor release. Expect more bugs than usual.
Sep 18 19:12:57.572 [notice] Configuration file "/usr/local/etc/tor/torrc" not present, using reasonable defaults.
Sep 18 19:12:57.575 [notice] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
Sep 18 19:12:57.000 [notice] Bootstrapped 0%: Starting
Sep 18 19:12:57.000 [notice] Starting with guard context "default"
Sep 18 19:12:57.000 [notice] Delaying directory fetches: DisableNetwork is set.
Sep 18 19:12:57.000 [notice] Catching signal TERM, exiting cleanly.
Tor returned successfully

I compiled tor (with and without --enable-restart-debugging) with openssl:

$ ./configure --disable-asciidoc --enable-pic --enable-android --enable-fatal-warnings

and for fun, with NSS:

$ ./configure --disable-asciidoc --enable-pic --enable-android --enable-fatal-warnings --enable-nss

and I compiled the program using:

gcc -Wall -Werror -g -o threaded_verify_config -Isrc/feature/api src/tools/threaded_verify_config.c -lpthread -lz -lm -levent -lssl -lcrypto -ldl `make show-libs`

and with NSS:

gcc -Wall -Werror -g -o threaded_verify_config -Isrc/feature/api src/tools/threaded_verify_config.c -lpthread -lz -lm -levent -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -lcrypto -ldl `make show-libs`

I put the program in src/tool/threaded_verify_config.c simply because it was easy.

The test programs are available in branch testing_26653 in my tor user repo. These don't fully bootstrap because SIGNAL TERM is sent as soon as the control fd is writable. I confirmed tor does fully bootstrap if it is not terminated immediately, though.

Ex.

diff --git a/src/tools/threaded_verify_config.c b/src/tools/threaded_verify_config.c
index 9ba637888..48b7249d8 100644
--- a/src/tools/threaded_verify_config.c
+++ b/src/tools/threaded_verify_config.c
@@ -121,6 +121,7 @@ int main(const int argc, char *argv[]) {
     pfd[0].events = POLLIN;
 
     if (poll(pfd, 1, -1)) {
+        sleep(60);
         read(ctx.control_fd, buf, 100);
         printf("%s", buf);
     }

​https://gitweb.torproject.org/user/sysrqb/tor.git/commit/?h=testing_26653&id=621ef35eb5464bcf63857deebd2be6728483e8b6

[sysrqb]

I don't see any problems with using this for embedding tor in a background service thread in the Android app.

Linking against all the shared libraries is a little annoying, so I'll test compiling with --enable-static-tor and see if that's any better. Overall, the API is very small, but the controller interface already provides all the features we need, so exposing some of them in the API would be redundant. In the future, I expect developers may ask for this, but I don't think we need it for TBA at this point. #24204 gives us immediate controller access, and its simple - I really like it.

[sysrqb]

Replying to sysrqb:

I don't see any problems with using this for embedding tor in a background service thread in the Android app.

Linking against all the shared libraries is a little annoying, so I'll test compiling with --enable-static-tor and see if that's any better.

Nope. It still requires them. It was worth a try.

[sysrqb]

Replying to sysrqb:

Replying to sysrqb:

I don't see any problems with using this for embedding tor in a background service thread in the Android app.

Linking against all the shared libraries is a little annoying, so I'll test compiling with --enable-static-tor and see if that's any better.

Nope. It still requires them. It was worth a try.

Right. Of course that didn't matter (those symbols aren't resolved until the linking stage...and that's not executed until the executable is created and the references must be resolved).

[sysrqb]

Although this won't be helpful for TBA, one feature that may be useful is adding an interface for creating a connection instead of requiring the use of SOCKS. This would be similar to tor_main_configuration_setup_control_socket(), where the function takes as parameters the ipv[46] addr or hostname, port number, and an isolation parameter, and it returns a file descriptor. I don't believe there's currently a feature provided by the controller interface. REDIRECTSTREAM is similar, but not the same.

/**
 * Tells Tor to open a socket for a client connection to the requested
 * destination.  Return the socket.
 */
SOCKS_SOCKET tor_main_create_connection(tor_main_configuration_t *cfg,
                                        const char * hostname,
                                        uint16_t port,
                                        const char * stream_isolation);

I wonder if the easiest way this could be implemented is creating a SOCKS connection itself (rather than the program making the SOCKS connection directly, but it'd ease the application development process because the API provides a default SOCKS client implementation.

[sysrqb]

Okay. I have an example Java client called jverify_config (doing the same thing as the threaded_verify_config example). It's in the same branch as above.

I don't think we need anything else here.

[gk]

Sponsor8 in 2018/09.