wiki:doc/OONI/CensorshipDetectionTools/Netalyzr

Netalyzr

Checklist

Is the tool open source?

No, but it's written in Java.

Is the data collected made public?

Not unless you count their map. Data from a user's session, both clientside and serverside transcripts, are made available for download after the tests are completed. Before running the tests, a user may view an example result, clientside transcript, and serverside transcript.

Is the data format that is used for publication easy to interact with?

Is is sorted by timestamp, similarly to dmesg output. One thing that could make it easier to read in this format would be a translation between the client and server timestamps.

Are the methodologies explained?

The types of tests conducted are listed, and substantial effort is put into adding optional "more info" buttons which give explanations balanced for both laypersons and the technically-inclined.

Is the tool to be used by the general public?

Yes.

If so, is the user warned of possible risks that he may incur when running the tool?

No, however risks for most users are incredibly low.

Does the data collected by the tool include potentially sensitive information?

Yes.

What kind of tests does the tool perform?

If the user's JVM security settings allow all tests to be run, the tests extensively test network settings, bandwidth, and configuration, as well as possible proxies and captive portals, MITMs for certain protocols, firewalls and gateway settings, existence of NAT and uPnP devices, kernel settings such as UDP buffer size and clock drift, and the configuration of local DNS resolvers.

How accurate are the tests?

There appears to be minimal false positives, and information that could be erroneous is explained in the results.

What claims does the tool make?

They claim to only collect the information which is displayed to the user in the results.

Are the claims satisfied?

Yes.

How does the reporting system work?

Several HTTP POSTs throughout the test procedure of the information to netalyzr.icsi.berkeley.edu. Both the web applet and the commandline client report in this way.

Is confidentiality and integrity of data being reported maintained?

Not at all. Plaintext over the wire.

What are it's strengths?

It's quite thorough in its tests of TCP, UDP, and DNS quirks.

What are it's weaknesses?

Java. And the HTTP POSTs.

What tests it performes

HTTP

  • Tries 0x20 hack on HTTP headers and checks raw response for identical capitalization
  • checks for reordered HTTP headers
  • checks for changed/added/unsolicited HTTP headers, checks headers lengths
  • checks raw HTTP content length
  • checks if google 302 redirects to a different google page
  • check for "<!DOCTYPE HTML PUBLIC <insert w3 info and url here>" and "<title>Google</title>" in response page
  • If a potential proxy was discovered via ICSI-Client-Addr, attempts an HTTP connection to the proxy and requests http://www.google.com, if a response comes back, check again for the ICSI-Client-Addr header
  • Does an HTTP GET for fake files of known size with .exe, .torrent, and .mp3 extensions, check for a fetching failure, an I/O exception, file corruption, checks the headers on the fetch's response, checks the size of the returned file
  • Checks that 404s are properly received for bogus urls: <servername>.<tld>/notfound/mode-<randomstring>/<agentID>.html
  • HTTP GET to <servername> for an image with known size and locally calculated MD5sum with the following extra headers set:

If-Modified-Since: If-None-Match: Pragma: no-cache Cache-Control: no-cache/no-store

If a proxy was previously discovered, first queries the proxy directly, and checks the response headers, and also the "Last-Modified" and "ETag" response headers, then compares MD5sums.

  • Checks for strong and weak Etag validation. Etags can be used for tracking purposes.

DNS

  • Checks DNS by requesting the inet adress of a domain (constructed from $nonce.<servername>.<tld>:<serverport>) by name on the locally configured DNS resolver, then requests IP of domain with domain's returned inet address. Domain's actual expected IP is hardcoded.
  • Checks DNS resolution for bogus domain ('return-false.abcd'), expects an UnknownHostException
  • Checks DNS MTU by checking to see if ednspadding-0.xdnsmtu.$nonce.<servername>.<tld>:<serverport> is reachable, then making 513B, 12801B, and 1473B requests, then trying the maximum 4000B, then slowly decreasing to get the actual max if 4000B doesn't work.
  • Tries to resolves each of these domains (we're first on the list?) twice, then gives up:

"www.torproject.org", "www.meebo.com", "www.paypal.com", "www.google.com", "mail.google.com", "www.google-analytics.com", "encrypted.google.com", "www.yahoo.com", "mail.yahoo.com", "search.yahoo.com", "www.bing.com", "www.microsoft.com", "windowsupdate.microsoft.com", "mail.live.com", "www.bankofamerica.com", "www.chase.com", "chaseonline.chase.com", "www.wamu.com", "www.wellsfargo.com", "online.wellsfargo.com", "www.f-secure.com", "www.trendmicro.com", "www.secureworks.com", "www.citibank.com", "online.citibank.com", "www.citi.com", "www.citimortgage.com", "www.smithbarney.com", "ad.doubleclick.net", "smartzone.comcast.net", "partner.googleadservices.com", "wireless.att.com", "www.ticketmaster.com", "www.facebook.com", "www.schwab.com", "www.etrade.com", "us.etrade.com", "www.tdameritrade.com", "www.ameritrade.com", "www.abbey.co.uk", "www.alliance-leicester.co.uk", "www.amazon.com", "www.bankofscotland.co.uk", "www.bankofthewest.com", "www.barclays.co.uk", "www.capitalone.com", "www.careerbuilder.com", "www.cnn.com", "www.desjardins.com", "www.deutsche-bank.de", "www.e-gold.com", "www.ebay.com", "www.fdic.gov", "www.friendfinder.com", "www.halifax.co.uk", "www.hsbc.co.uk", "www.irs.gov", "www.jpmorganchase.com", "www.lloydstsb.com", "www.mbna.com", "www.mbna.net", "messenger.yahoo.com", "www.nationwide.co.uk", "www.networksolutions.com", "www.newegg.com", "www.nordea.fi", "www.orange.fr", "www.postbank.de", "www.rbs.co.uk", "www.sears.com", "www.sparkasse.de", "www.sterlingsavingsbank.com", "www.usbank.com", "www.verisign.com", "www.visa.com", "www.wachovia.com", "westernunion.com", "tinyurl.com", "bit.ly", "www.youtube.com", "www.twitter.com", "www.thepiratebay.org", "thepiratebay.org", "www.twitpic.com", "hootsuite.com", "appspot.com", "www.blogimg.jp", "blogpark.jp", "6park.com", "www.6park.com"

  • Does most of the above DNS checks over TCP
  • Makes CHAOS queries to attempt to discover anycast DNS root servers, see this paper. The list of hardcoded DNS root server IPs is: 198.41.0.4 (Verisign) 192.228.79.201 (BRoot) 192.33.4.12 (Cogent Communications) 128.8.10.90 (U. of Maryland) 192.203.230.10 (NASA) 192.5.5.241 (ISC) 192.112.36.4 (DoD) 128.63.2.53 (US ISC Headquarters) 192.36.148.17 (Netnod, Sweden) 192.58.128.30 (Verisign) 193.0.14.129 (KRoot) 199.7.83.42 (ICANN) 202.12.27.33 (U. of Tokyo)
  • Checks for NAT DNS resolvers, and for 2wire devices, if found, checks for MITMing of DNS queries
  • Checks for DNS resolver port randomization, for more info see this commit in my dnstamper branch of ooni, specifically the DNSTamperResolver class.
  • Does a bunch of the same checks for IPv6 DNS
  • Check if local DNS resolver (lDNSr) will accept exact, internal, and exteral glue entries. Check if lDNSr will try to resolve a CNAME for an NS
  • Check if lDNSr passes 0x20 queries correctly
  • Check if lDNSr uses ANY for A records
  • Check if lDNSr will resolve AAAA records for IPv6only hosts
  • Check if lDNSr support DNSSEC and/or EDNS, if so, then also check EDNS MTU
  • Checks if lDNSr will handle different sized TXT records
  • Checks if lDNSr wildcards NXDOMAIN errors by seeing if "www.yahoo.cmo" falsely returns as "www.yahoo.com"
  • Checks if lDNSr DNS wildcarding extends to "typos" at the sld level
  • Checks that random hostnames properly resolve to NXDOMAIN
  • Check if DNS works over TCP
  • If lDNSr accepts glue records, test creation of glue records with TTL 0 and 1
  • Try to query an IPv6only DNS server
  • Checks if SOA records can be retrieved
  • Make queries to lDNSr for random hostnames which are as yet uncached, calculate median uncached lookup time, then re-request all of them again and calculate median cached lookup time
  • Checks if lDNSr supports a sufficient ammount of DNS transactionID entropy, which lDNSr should be doing to avoid the subdomain cache poisoning attack.
  • Checks if an external DNS proxy can be used, and if so, does all the previous DNS tests for that proxy
  • Checks if non-DNS UDP packets can be sent over port 53

Misc

  • Sends useragent, language, encoding, charset, agentID, nonce
  • Tries direct TCP/IPv4 and TCP/IPv6 connections on "$TCP_ECHO_PORT" (?), 21, 22, 25, 53, 80, 110, 135, 139, 143, 161, 443, 465, 465, 585, 587, 993, 995, 1194, 1723, 5060, 6681, 9001
  • Sends info on client network interfaces: addrs, hostnames, inetaddrs, and available bandwidth
  • Checks for ability to send raw UDP, also checks path MTU holesby sending and receiving large (1471B) UDP packets
  • Checks for ability to send and recieve fragmented UDP packets, and for max MTU
  • Checks which ports UDP is available on, including common ports: 53 (DNS), 123 (NTP), 137 (NetBIOS), 138 (NetBIOS DGM), 500 (IKE), 1194 (OpenVPN), 1434 (Slammer), 1701 (L2 tunnels), 4500 (IPSec NAT), 5004 (RTP), 5005 (RTCP), 5060 (SIP), 7078 (VoIP), 7082 (VoIP), 9899 (SCTP), 27005 & 27015 (Steam)
  • Checks if the device in behind NAT
  • Builds a multicast socket on local inet and tries sending a broadcast datagram host discovery query
  • Tries to discover local uPnP devices, their NAT addressses, and location URLs
  • HTTP POST to a server over port 80 of the agentID, all uPnP devices, and the uPnP device descriptions
  • gets previous POST results and uses them to try to measure one-way-delay, compensating for clock drift
  • calculates kernel UDP buffer size
  • Checks and sends (over http:80 again) WAN info: connection type, IP address, WANPPP connection info and IP (if available), external IP
  • Does UDP bursts and measures loss
  • calculates RTT
  • server sends back an ICSI-Client-Header, checks for match against external IP, and != then that client is potential proxy
  • checks send/recv UDP MTU (again, I believe?), allowed fragmentation size, and does a binary search to attempt to find the bottleneck IP
  • Checks if traceroutes work, and tries to identify bad hops for IPv4 and IPv6
  • Tries to send ICMP packets above the max size (~1KB)
Last modified 5 years ago Last modified on Jun 18, 2012, 8:44:28 AM