Opened 8 years ago

Closed 22 months ago

#4097 closed defect (wontfix)

KeyError in self.node_map[exit_node[1:]]

Reported by: aagbsn Owned by: mikeperry
Priority: Medium Milestone:
Component: Core Tor/Torflow Version:
Severity: Keywords:
Cc: aagbsn@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

My ExitAuthority died with Traceback:

ERROR[Fri Sep 23 06:55:02 2011]:An unexpected error occured.
Traceback (most recent call last):
  File "soat.py", line 2914, in <module>
    main(sys.argv)
  File "soat.py", line 2855, in main
    result = test.run_test()
  File "soat.py", line 1085, in run_test
    result = self.check_http(address,filetype)
  File "soat.py", line 1251, in check_http
    result = HttpTestResult(self.node_map[exit_node[1:]],
KeyError: 'E7C49248095E77968BA4390DA2BBAC9D7E4560F3'
INFO[Fri Sep 23 06:55:02 2011]:Resetting __LeaveStreamsUnattached=0 and FetchUselessDescriptors=1

Child Tickets

Change History (8)

comment:1 Changed 8 years ago by aagbsn

(soat)exitscanner@tor:~/torflow/NetworkScanners/ExitAuthority$ grep "E7C49248095E77968BA4390DA2BBAC9D7E4560F3" data/soat.log | grep -v set 
NOTICE[Thu Sep 22 12:31:40 2011]:Running router rutabagatrousers=E7C49248095E77968BA4390DA2BBAC9D7E4560F3 has no descriptor
INFO[Thu Sep 22 13:49:59 2011]:Router descriptor for E7C49248095E77968BA4390DA2BBAC9D7E4560F3 does not match ns fingerprint (NS @ 2011-09-22 18:58:02 vs Desc @ 2011-09-22 17:14:20)
INFO[Thu Sep 22 13:50:01 2011]:Expiring non-running router E7C49248095E77968BA4390DA2BBAC9D7E4560F3
DEBUG[Fri Sep 23 06:52:44 2011]:Got Setexit: $E7C49248095E77968BA4390DA2BBAC9D7E4560F3
DEBUG[Fri Sep 23 06:52:44 2011]:Added exit restriction for $E7C49248095E77968BA4390DA2BBAC9D7E4560F3
DEBUG[Fri Sep 23 06:52:44 2011]:Applying Setexit: $E7C49248095E77968BA4390DA2BBAC9D7E4560F3
DEBUG[Fri Sep 23 06:52:44 2011]:Chose to run 1 tests via E7C49248095E77968BA4390DA2BBAC9D7E4560F3 (tests share 312 exit nodes)
DEBUG[Fri Sep 23 06:52:46 2011]:Circ refcount 1 for E7C49248095E77968BA4390DA2BBAC9D7E4560F3
DEBUG[Fri Sep 23 06:52:46 2011]:Set last exit to E7C49248095E77968BA4390DA2BBAC9D7E4560F3
DEBUG[Fri Sep 23 06:52:49 2011]:0.000303983688354 CIRC 35494 EXTENDED $53EA7E124038799F0047FE94C7362092DEF9CA2E,$E7C49248095E77968BA4390DA2BBAC9D7E4560F3
DEBUG[Fri Sep 23 06:52:49 2011]:0.000473022460938 CIRC 35494 BUILT $53EA7E124038799F0047FE94C7362092DEF9CA2E,$E7C49248095E77968BA4390DA2BBAC9D7E4560F3
DEBUG[Fri Sep 23 06:52:50 2011]:Got last exit of E7C49248095E77968BA4390DA2BBAC9D7E4560F3
INFO[Fri Sep 23 06:52:50 2011]:SSL success at E7C49248095E77968BA4390DA2BBAC9D7E4560F3. This makes 64/91 node successes for p10.secure.hostingprod.com
DEBUG[Fri Sep 23 06:52:50 2011]:Got last exit of E7C49248095E77968BA4390DA2BBAC9D7E4560F3
INFO[Fri Sep 23 06:52:50 2011]:Marking E7C49248095E77968BA4390DA2BBAC9D7E4560F3 with result 0
DEBUG[Fri Sep 23 06:52:50 2011]:Keeping node E7C49248095E77968BA4390DA2BBAC9D7E4560F3. 312 nodes remain. Tests: 1/5
INFO[Fri Sep 23 06:52:52 2011]:SSL test via E7C49248095E77968BA4390DA2BBAC9D7E4560F3 has result 0
DEBUG[Fri Sep 23 06:52:52 2011]:Got last exit of E7C49248095E77968BA4390DA2BBAC9D7E4560F3
NOTICE[Fri Sep 23 06:52:52 2011]:$E7C49248095E77968BA4390DA2BBAC9D7E4560F3 had error -15.0 fetching content for h
KeyError: 'E7C49248095E77968BA4390DA2BBAC9D7E4560F3'

comment:2 Changed 8 years ago by aagbsn

INFO[Fri Sep 23 06:52:52 2011]:HTTPTest decided to fetch 1 urls of types: [u't']
INFO[Fri Sep 23 06:52:52 2011]:[(u'h', u't')]
INFO[Fri Sep 23 06:52:52 2011]:Conducting an http test with destination h
DEBUG[Fri Sep 23 06:52:52 2011]:Starting request for: h
Traceback (most recent call last):
  File "soat.py", line 379, in http_request
    reply = opener.open(request)
  File "/usr/lib/python2.6/urllib2.py", line 383, in open
    protocol = req.get_type()
  File "/usr/lib/python2.6/urllib2.py", line 244, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: h
INFO[Fri Sep 23 06:52:52 2011]:Completed HTTP Reqest for: h
DEBUG[Fri Sep 23 06:52:52 2011]:Got last exit of E7C49248095E77968BA4390DA2BBAC9D7E4560F3
NOTICE[Fri Sep 23 06:52:52 2011]:$E7C49248095E77968BA4390DA2BBAC9D7E4560F3 had error -15.0 fetching content for h
DEBUG[Fri Sep 23 06:52:52 2011]:Starting request for: h
Traceback (most recent call last):
  File "soat.py", line 379, in http_request
    reply = opener.open(request)
  File "/usr/lib/python2.6/urllib2.py", line 383, in open
    protocol = req.get_type()
  File "/usr/lib/python2.6/urllib2.py", line 244, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: h

comment:3 Changed 8 years ago by aagbsn

Cc: aagbsn@… added

maybe get_search_urls_for_filetype(self, filetype, number) does not return a tuple and is used inconsistently by SearchBasedHTTPTest.refill_targets()

  def refill_targets(self):
    for ftype in self.scan_filetypes:
      targets_needed = self.results_per_type - len(self.targets.bykey(ftype))
      if targets_needed > 0:
        plog("NOTICE", self.proto+" scanner short on "+ftype+" targets. Adding more")
        map(self.add_target, self.get_search_urls_for_filetype(ftype,targets_needed))

for comparison, see SearchBasedHTTPTest.get_targets():

  def get_targets(self):
    '''
    construct a list of urls based on the wordlist, filetypes and protocol.
    '''
    plog('INFO', 'Searching for relevant sites...')

    urllist = set([])
    for filetype in self.scan_filetypes:
      urllist.update(map(lambda x: (x, filetype), self.get_search_urls_for_filetype(filetype, self.results_per_type)))

    return list(urllist)

Testing the following modification:

index be0dde0..aabb3e4 100755
--- a/NetworkScanners/ExitAuthority/soat.py
+++ b/NetworkScanners/ExitAuthority/soat.py
@@ -1883,9 +1883,12 @@ class SearchBasedHTTPTest(SearchBasedTest, BaseHTTPTest):
   def refill_targets(self):
     for ftype in self.scan_filetypes:
       targets_needed = self.results_per_type - len(self.targets.bykey(ftype))
+      urllist = set([])
       if targets_needed > 0:
         plog("NOTICE", self.proto+" scanner short on "+ftype+" targets. Adding more")
-        map(self.add_target, self.get_search_urls_for_filetype(ftype,targets_needed))
+        #map(self.add_target, self.get_search_urls_for_filetype(ftype,targets_needed))
+        urllist.update(map(lambda x: (x, ftype), self.get_search_urls_for_filetype(ftype, targets_needed)))
+        map(self.add_target, urllist)

comment:5 Changed 8 years ago by aagbsn

In hindsight, this should be tracked as a separate issue: #4127

comment:6 Changed 8 years ago by aagbsn

Yep, she's back:

NOTICE[Mon Oct 03 05:33:59 2011]:Got extra NEWDESC event for router teeks992=2091E306B8240B1592E0F690DD9BD62071BA204B
DEBUG[Mon Oct 03 05:34:00 2011]:36269.470582 Read 1 ND => 1570 routers. Update: True
DEBUG[Mon Oct 03 05:34:00 2011]:E = 1402230000, G = 2147006000, T = 2596476000, g_ratio = 0.826892295558, e_ratio = 0.540051207868, bw_per_hop = 1298238000.0
DEBUG[Mon Oct 03 05:34:01 2011]:Bw: 10955110560/2596476000. The exit-weight is: 0.0741618707345, guard weight is: 0.395326328851
DEBUG[Mon Oct 03 05:34:01 2011]:E = 1402230000, G = 2147006000, T = 2596476000, g_ratio = 0.826892295558, e_ratio = 0.540051207868, bw_per_hop = 1298238000.0
DEBUG[Mon Oct 03 05:34:01 2011]:Bw: 10955110560/2596476000. The exit-weight is: 0.0741618707345, guard weight is: 0.395326328851
DEBUG[Mon Oct 03 05:34:01 2011]:Added exit restriction for $725E414D5A86C8FA3418F6005787646C5F43589A
DEBUG[Mon Oct 03 05:34:01 2011]:newdesc_event end
DEBUG[Mon Oct 03 05:34:01 2011]:newdesc_event begin
NOTICE[Mon Oct 03 05:34:01 2011]:Got extra NEWDESC event for router ah3z=24FB8281D98B624523AF965D761635CEE53F26F9
INFO[Mon Oct 03 05:34:02 2011]:SSL Request done for addrress: shop.foebud.org
DEBUG[Mon Oct 03 05:34:02 2011]:Removed 127.0.0.1:34380 from our local port list
DEBUG[Mon Oct 03 05:34:02 2011]:Got last exit of FFFEAC4B93184447F9A4B4EA795EBF1A443E8533
ERROR[Mon Oct 03 05:34:03 2011]:An unexpected error occured.
Traceback (most recent call last):
  File "soat.py", line 2918, in <module>
DEBUG[Mon Oct 03 05:34:03 2011]:36273.1197581 Read 1 ND => 1570 routers. Update: True
DEBUG[Mon Oct 03 05:34:04 2011]:E = 1402230000, G = 2147006000, T = 2596476000, g_ratio = 0.826892295558, e_ratio = 0.540051207868, bw_per_hop = 1298238000.0
DEBUG[Mon Oct 03 05:34:04 2011]:Bw: 11639804970/2596476000. The exit-weight is: 0.0741618707345, guard weight is: 0.395326328851
DEBUG[Mon Oct 03 05:34:04 2011]:E = 1402230000, G = 2147006000, T = 2596476000, g_ratio = 0.826892295558, e_ratio = 0.540051207868, bw_per_hop = 1298238000.0
DEBUG[Mon Oct 03 05:34:04 2011]:Bw: 11639804970/2596476000. The exit-weight is: 0.0741618707345, guard weight is: 0.395326328851
DEBUG[Mon Oct 03 05:34:05 2011]:Added exit restriction for $725E414D5A86C8FA3418F6005787646C5F43589A
    main(sys.argv)
  File "soat.py", line 2859, in main
    result = test.run_test()
  File "soat.py", line 1493, in run_test
    return self.check_ssl(random.choice(self.targets)[0])
  File "soat.py", line 1646, in check_ssl
    result = SSLTestResult(self.node_map[exit_node[1:]],
KeyError: 'FFFEAC4B93184447F9A4B4EA795EBF1A443E8533'
INFO[Mon Oct 03 05:41:37 2011]:Resetting __LeaveStreamsUnattached=0 and FetchUselessDescriptors=1
DEBUG[Mon Oct 03 05:42:23 2011]:newdesc_event end
DEBUG[Mon Oct 03 05:42:42 2011]:newdesc_event begin
NOTICE[Mon Oct 03 05:43:25 2011]:Got extra NEWDESC event for router ChubbyRelay=2786E37E2E538AC63950657240E58901F098DA1E

comment:7 Changed 8 years ago by mikeperry

I committed a potential fix for this in TorCtl origin/master at e15ed99df8e8e04cadd0f3792e42a621df6fbe02

comment:8 Changed 22 months ago by tom

Resolution: wontfix
Status: newclosed

ExitAuthority is not maintained anymore

Note: See TracTickets for help on using tickets.