Opened 3 weeks ago

Last modified 8 days ago

#32299 new enhancement

read GeoIPFile and GeoIP6File straight out of ZIP (.apk, .jar, .zip, .ipa)

Reported by: eighthave Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: Android, iOS, tbb-mobile, needs-proposal
Cc: n8fr8 Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Right now, the geoip and geoip6 files are packed into the APK, which is basically a ZIP/JAR file. Then when Orbot/TorBrowser/etc starts, it unzips those files to a local folder. Since those files are read only, they could be read directly out of the zip file. This would mean faster startup times and less disk space in use. Right now, there is a full ZIP read, then a full write, then a full read from disk.

I believe that tor is already linked against zlib, so the functions are there. In any case, zlib is one of the few libs that are provided by the NDK.

Android already does things like this. For example, PNG files are stored uncompressed in the APK. The whole ZIP file is byte aligned for each PNG reading. Then the APK maintains a list of pointers to the byte position in the APK file a PNG file can be read from.

Child Tickets

Change History (6)

comment:1 Changed 8 days ago by teor

Keywords: needs-proposal added

It looks like this feature would change the tor specs, because they currently assume that the GeoIP* files are read straight from disk. (It also looks like this feature would be useful on any disk-constrained system: lots of Tor's directory documents are sent compressed, and only read into RAM once.)

So let's design this change using the Tor proposal process:
https://gitweb.torproject.org/torspec.git/tree/proposals/001-process.txt#n39

Here are some design notes:

In Tor, we currently use prefixes for special file names:

  • unix sockets: "unix:(path)"

We could do something similar for archives:

  • zip files: "zip:(path to file)/(path within archive)"

That would be easy to extend to other formats as needed.

comment:2 Changed 8 days ago by eighthave

I imagine OpenWRT is one of the disk-constrained systems you are talking about, though these days perhaps it makes sense to also include Debian e.g. Raspberry Pi.

  • zip:/path/to/my.zip/path/to/geoip makes sense to me. .apk and .jar files are both ZIP files with some extra standards, so it might make sense to include jar: and apk: as aliases for zip:. Most people are not aware that .apk and .jar files are ZIP format. iOS's .ipa format is also a ZIP container, and it looks like iOS also reads PNGs straight from that file once installed.
  • .deb/.udeb are .ar files with compressed .tar files (.gz, bz2, .lzma, .xz). Supporting that case would probably require deb:/path/to/tor_0.4.1.5-1.deb/path/to/geoip. libs for .gz, .lzma and .xz are already reguarly linked into tor. .ar and .tar support would need to be added.
  • OpenWRT's .ipk files seem to follow the same structure as .deb, but I couldn't find clear docs on that. But it might use tar instead of ar.
  • .rpm are compressed cpio files, the compression can be with gzip, zstd, lzma, xz, .bzip2. RPM 5.0 allows xar instead of cpio.

comment:3 Changed 8 days ago by eighthave

Keywords: iOS added

comment:4 Changed 8 days ago by eighthave

Summary: read GeoIPFile and GeoIP6File straight out of APK file (Android-only)read GeoIPFile and GeoIP6File straight out of ZIP (.apk, .jar, .zip, .ipa)

Seems to me that this proposal should focus on zip:, since the other package formats would work quite a bit differently.

Another thing to note: although APK and JAR files are always readable as ZIP files, some strict ZIP libraries might get caught up on some aspects of the APK format, which are not part of the ZIP standard. For example zipalign adds padding between entries using a method that is not specified in the ZIP standard (as far as I remember). I do know that rewriting an APK with libzip will break that padding, and can make APKs not work. I am pretty sure that breakage only comes when writing, and everything should read APKs just fine. There might be some warnings or error messages though, depending on the reader config.

comment:5 Changed 8 days ago by eighthave

I guess I didn't understand enough about iOS, which is a very different scenario than Android. tla says:

From my point of few, this looks like an optimization which should be at the very end of any development process. geoip: 4.3MB, geoip6: 2.5MB. Minimal disk size in any iOS device: 8 GB. Contained resource files in IPAs are automatically unpacked. High chance of app access to IPA file is restricted/forbidden, high chance of no mechanism to actually stop iOS from unpacking contained resources.

comment:6 Changed 8 days ago by dgoulet

Milestone: Tor: unspecified
Note: See TracTickets for help on using tickets.