Expand tutorial to include exit scanning

added component::archived/stem owner::atagar priority::medium resolution::implemented status::closed type::enhancement website labels

Hey, This is the code we've talked about. There might be some structural changes happening so I held back with commenting everything just yet.

https://gist.github.com/ragwater/1dbe7596eb11d68f7538

Trac:
Username: ragwater
Status: new to needs_review

Hi ragwater, sorry about the delay. This script looks great! Feedback below...

from stem.util import term

The first example already demos the term util and printing bootstrap messages. Dropping this from later examples simplifies the scripts and makes them more focused on just what we're trying to show. I did this for another recent addition...

https://stem.torproject.org/tutorials/to_russia_with_love.html#reading-twitter

import pycurl

Lets use SocksiPy instead since its usage is considerably less verbose. See the current tutorial page for an example.

import Queue

PEP8 (python's style guide) defines a specific order for our imports. Please see...

http://www.python.org/dev/peps/pep-0008/#imports

Running 'pep8 --ignore E251,E501,E111,E127' over your script will highlight these issues. For an explanation of the parts of pep8 that we're ignoring see...

https://gitweb.torproject.org/stem.git/blob/HEAD:/test/settings.cfg#l95

STREAM_STATUS_CLOSED = "CLOSED"
STREAM_STATUS_NEW = "NEW"
STREAM_STATUS_DETACHED = "DETACHED"

Not necessary. These are part of the stem.StreamStatus enumeration. Stem's enums are actually strings so...

>>> from stem import StreamStatus
>>> StreamStatus.DETACHED
'DETACHED'

results = {
    'SUCCESS' : 0,
    'FAILED' : 0,
    'TIMEOUT' : 0,
    'DESTROYED' : 0,}

Minor nitpicks but please...

do two space indents
put the colon right after the key
put the ending brace on a new line, since this makes copy-and-paste easier

For instance in this case...

results = {
  'SUCCESS': 0,
  'FAILED': 0,
  'TIMEOUT': 0,
  'DESTROYED': 0,
}

return stem.process.launch_tor_with_config(config = config,
  init_msg_handler = print_bootstrap_lines)

Keyword based functions like launch_tor_with_config() can also benefit from that kind of layout...

return stem.process.launch_tor_with_config(
  config = config,
  init_msg_handler = print_bootstrap_lines,
)

Though as mentioned above we should probably drop init_msg_handler, so guess there's no point in having this span multiple lines.

total=sum(value for value in data.values())

Another pep8 thing but the '=' should have a space on either side of it.

# I was considering checking allowed ports (443) at this point by
# getting microdescriptors for each node, we can either filter it here
# or wait for it to happen at getting response stage (it will FAIL then).
node_gen = controller.get_network_statuses()
filtered_nodes = filter_nodes(controller, node_gen) # [-20:]
circ_waiting_list = Queue.Queue()

Good idea about filtering based on the exit policy. Lets do this via server descriptors since microdescriptors make this a real pita. If you add "FetchUselessDescriptors 1" to the config then this should do the trick...

from stem.control import Controller

with Controller.from_port(port = 9051) as controller:
  controller.authenticate()

  russian_exits = []

  for desc in controller.get_server_descriptors():
    if desc.exit_policy.can_exit_to(port = 443):
      # check if this exit is in russia
           
      exit_locale = controller.get_info('ip-to-country/%s' % desc.address)

      if exit_locale == 'ru':
        russian_exits.append(desc)

  print "Russian Exits:"

  for desc in russian_exits:
    print "  %s (%s)" % (desc.nickname, desc.address)

There's actually no need to honour the 'Exit' flag. That is just an indicator from the directory authorities that a relay allows certain ports and is relatively quick. Lacking the flag doesn't necessarily mean that a relay can't serve as an exit (that's determined by its exit policy).

One gotcha though - there doesn't appear to be any Russian exits. sigh

$ python example.py 
Russian Exits:

For testing lets use another locale, then we'll note in the tutorial that they might need to pick someplace else.

Cheers! -Damian

Trac:
Status: needs_review to needs_revision

Hey, thanks for the feedback!

Dropping this from later examples simplifies the scripts and makes them more focused on just what we're trying to show.

Should the info text showcasing the stream events go as well? I mean...

Checking exit node ididnteditheconfig6 (128.8.24.14)
Assigned to circuit 7:
0 ('C2F9D30118BEBF3EFEE6D96252374082CA73C054', 'st0nerhenge') 83.170.92.9
1 ('4F8D80A0F768A2A29856A8F26B05D35DEAA39850', 'wannabe') 96.47.226.19
2 ('594D6B7B1DEE6B23476C7378D0D26F2DB61251A5', 'HackThisSite') 198.148.81.167
3 ('001F9D6558A2CA6204A53B5BC7EF5C5F9637DA2D', 'ididnteditheconfig6') 128.8.24.14
| Stream 10 is being assigned to circuit 7
| Stream 10 assigned to circuit 7
 \Status: SENTCONNECT, destination: 173.194.113.66
| Stream 10 assigned to circuit 7
 \Status: REMAP, destination: 173.194.113.66
| Stream 10 assigned to circuit 7
 \Status: SUCCEEDED, destination: 173.194.113.66

One gotcha though - there doesn't appear to be any Russian exits.

That actually got me worried a bit, because I was definitely exiting through Russian nodes in my testing.

Turns out exit_policy.can_exit_to() responds in an interesting manner when only the port parameter is provided, and when I run your script against the US locale, it returned ~10 exit nodes, none of them being really usable as far as I can tell (e.g. https://atlas.torproject.org/#details/810962DC3C058C757070420E9A9497532BB8185A or https://atlas.torproject.org/#details/315D5684D5343580D409F16119F78D776A58AEFB).

Anyhow passing strict = True along (or specifying the address) with desired port solves the issue and gives us a healthy number of exit nodes.

  from stem import StreamStatus

I have no idea how I've managed to miss it, thanks for the catch.

I'm yet unsure how to get the cert checking in urllib with socks but I should figure it out in a few days' time and deliver the code, pep8-approved this time around ;).

Thanks, Tom

Trac:
Username: ragwater

Should the info text showcasing the stream events go as well?

That's definitely an interesting idea, but lets initially keep the example simple initially. Maybe just printing the exits as we use them (with a message if we encounter an issue).

Turns out exit_policy.can_exit_to() responds in an interesting manner when only the port parameter is provided, and when I run your script against the US locale, it returned ~10 exit nodes, none of them being really usable as far as I can tell

Oops, I should have included the strict flag. Those relays are being returned because they do allow exiting on port 443... but only for one IP.

Exit policy of 810962DC3C058C757070420E9A9497532BB8185A:
  accept 69.28.72.78:443
  accept 69.28.72.78:6697
  reject *:*

Exit policy of 315D5684D5343580D409F16119F78D776A58AEFB:
  accept 173.236.34.122:80
  accept 173.236.34.122:443
  accept 173.236.34.123:80
  accept 173.236.34.123:443
  reject *:*

Originally 'strict' was the default so it would only match if the relay accepted all 443 traffic but some users found that unintuitive.

Anyhow passing strict = True along (or specifying the address) with desired port solves the issue and gives us a healthy number of exit nodes.

Glad to hear! When you provide an IP address the 'strict' flag doesn't do anything. As the pydocs mention that only effects how an undefined address or port is handled.

I'm yet unsure how to get the cert checking in urllib with socks but I should figure it out in a few days' time and deliver the code, pep8-approved this time around ;).

Great. Looking forward to it!

Trac:
Cc: N/A to r.a@posteo.net

As talked on IRC it might be a good idea to split 'sslstrip' up into two seperated scripts: a) use SETCONF (ExitNodes and StrictExitNodes) and NEWNYM signal to let Tor build a new circuit (path length might be even shortened to 2). b) manually circuit construction with rotating exit and fixed guard node for use as an example

Went with a bandwidth-authority style scanner instead since a demo of this is also useful to Roger. Done!

Trac:
Resolution: N/A to implemented
Status: needs_revision to closed

closed

mentioned in issue #14979 (closed)

Expand tutorial to include exit scanning

Child items 0

Activity