Opened 11 years ago

Last modified 7 years ago

#930 closed defect (Fixed)

router_parse_entry_from_string (s=0x135 <Address 0x135 out of bounds>

Reported by: phobos Owned by:
Priority: Low Milestone: 0.2.1.x-final
Component: Core Tor/Tor Version: 0.2.1.12-alpha
Severity: Keywords:
Cc: phobos, nickm, edmanm, arma Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

gdb bt below. tor 0.2.0.34-stable does not have this issue.

Feb 22 18:48:53.906 [notice] Bootstrapped 90%: Establishing a Tor circuit.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fab760756e0 (LWP 18369)]
0x00007fab74d11015 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007fab74d11015 in raise () from /lib/libc.so.6
#1 0x00007fab74d12b83 in abort () from /lib/libc.so.6
#2 0x00007fab74d57a80 in ?? () from /lib/libc.so.6
#3 0x00000000004ad39d in memarea_drop_all (area=0x21a47f0) at memarea.c:102
#4 0x0000000000493c64 in router_parse_entry_from_string (s=0x135 <Address 0x135 out of bounds>, end=0x9a8 <Address 0x9a8 out of bounds>, cache_copy=1,

allow_annotations=295, prepend_annotations=0x135 <Address 0x135 out of bounds>) at routerparse.c:1438

#5 0x0000000000494eb4 in router_parse_list_from_string (s=0x7fff7e093e38, eos=0x7fab75ec6357 "", dest=0x21ecb10, saved_location=SAVED_NOWHERE,

want_extrainfo=0, allow_annotations=0, prepend_annotations=0x7fff7e093f70 "@downloaded-at 2009-02-22 23:48:56\n@source \"194.105.99.31\"\n")
at routerparse.c:1061

#6 0x000000000048cf05 in router_load_routers_from_string (

s=0x7fab75ebc614 "router che 81.233.224.95 443 0 80\nplatform Tor 0.2.0.33 (r18212) on Linux i686\nopt protocols Link 1 2 Circuit 1\npublished 2009-02-22 21:26:03\nopt fingerprint D5F2 C65F 4131 A146 8D5B 67A8 838A 9B7E D8"..., eos=0x0, saved_location=SAVED_NOWHERE, requested_fingerprints=0x2264440,
descriptor_digests=1, prepend_annotations=0x7fff7e093f70 "@downloaded-at 2009-02-22 23:48:56\n@source \"194.105.99.31\"\n") at routerlist.c:3500

#7 0x0000000000447b16 in connection_dir_client_reached_eof (conn=0x21a9440) at directory.c:1376
#8 0x00000000004486de in connection_dir_reached_eof (conn=0x21a9440) at directory.c:2041
#9 0x000000000042bde8 in connection_handle_read (conn=0x21a9440) at connection.c:2909
#10 0x0000000000461bc0 in conn_read_callback (fd=<value optimized out>, event=<value optimized out>, _conn=<value optimized out>) at main.c:456
#11 0x00007fab75a4e67d in event_base_loop () from /usr/lib/libevent-1.3e.so.1
#12 0x00000000004617a6 in do_main_loop () at main.c:1435
#13 0x00000000004619f5 in tor_main (argc=1, argv=<value optimized out>) at main.c:2060
#14 0x00007fab74cfc466 in libc_start_main () from /lib/libc.so.6
#15 0x0000000000407469 in _start ()

[Automatically added by flyspray2trac: Operating System: Other Linux]

Child Tickets

Attachments (1)

flyspray-930-datadir.tgz (1.1 MB) - added by phobos 11 years ago.
cached* files from my datadir

Download all attachments as: .zip

Change History (24)

comment:1 Changed 11 years ago by phobos

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fa75f0c86e0 (LWP 18176)]
0x00007fa75dd64015 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007fa75dd64015 in raise () from /lib/libc.so.6
#1 0x00007fa75dd65b83 in abort () from /lib/libc.so.6
#2 0x00007fa75ddaaa80 in ?? () from /lib/libc.so.6
#3 0x00000000004ad56d in memarea_drop_all (area=0x2456000) at memarea.c:102
#4 0x0000000000493cf4 in router_parse_entry_from_string (s=0x135 <Address 0x135 out of bounds>, end=0x9a8 <Address 0x9a8 out of bounds>, cache_copy=1,

allow_annotations=295, prepend_annotations=0x135 <Address 0x135 out of bounds>) at routerparse.c:1438

#5 0x0000000000494f44 in router_parse_list_from_string (s=0x7fff670e8e48, eos=0x7fa75f0852d7 "", dest=0x234e920, saved_location=SAVED_NOWHERE,

want_extrainfo=0, allow_annotations=0, prepend_annotations=0x7fff670e8f80 "@downloaded-at 2009-02-22 23:45:17\n@source \"90.230.56.89\"\n")
at routerparse.c:1061

#6 0x000000000048cf95 in router_load_routers_from_string (

s=0x7fa75f07b594 "router che 81.233.224.95 443 0 80\nplatform Tor 0.2.0.33 (r18212) on Linux i686\nopt protocols Link 1 2 Circuit 1\npublished 2009-02-22 21:26:03\nopt fingerprint D5F2 C65F 4131 A146 8D5B 67A8 838A 9B7E D8"..., eos=0x0, saved_location=SAVED_NOWHERE, requested_fingerprints=0x2442100,
descriptor_digests=1, prepend_annotations=0x7fff670e8f80 "@downloaded-at 2009-02-22 23:45:17\n@source \"90.230.56.89\"\n") at routerlist.c:3506

#7 0x00000000004478d6 in connection_dir_client_reached_eof (conn=0x209c100) at directory.c:1376
#8 0x000000000044849e in connection_dir_reached_eof (conn=0x209c100) at directory.c:2041
#9 0x000000000042bb18 in connection_handle_read (conn=0x209c100) at connection.c:2909
#10 0x00000000004619b0 in conn_read_callback (fd=<value optimized out>, event=<value optimized out>, _conn=<value optimized out>) at main.c:456
#11 0x00007fa75eaa167d in event_base_loop () from /usr/lib/libevent-1.3e.so.1
#12 0x0000000000461596 in do_main_loop () at main.c:1435
#13 0x00000000004617e5 in tor_main (argc=1, argv=<value optimized out>) at main.c:2060
#14 0x00007fa75dd4f466 in libc_start_main () from /lib/libc.so.6
#15 0x00000000004074b9 in _start ()

comment:2 Changed 11 years ago by nickm

So the abort seems to be coming from inside the free() in memarea_drop_all. Heap corruption, maybe?

comment:3 Changed 11 years ago by nickm

The s=0x135 thing is _probably_ a red herring; it looks like the kind of thing thatt happens when GCC optimizes out
no-longer-live variables.

comment:4 Changed 11 years ago by nickm

If this happens repeatably, and you can run Tor under valgrind, that'd probably say what the trouble is.

comment:5 Changed 11 years ago by phobos

Here's a bt and print *s from #5:

Feb 23 14:46:59.662 [notice] Parsing GEOIP file.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f97f774f6e0 (LWP 22120)]
0x00007f97f63eb015 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007f97f63eb015 in raise () from /lib/libc.so.6
#1 0x00007f97f63ecb83 in abort () from /lib/libc.so.6
#2 0x00007f97f6431a80 in ?? () from /lib/libc.so.6
#3 0x00000000004ad39d in memarea_drop_all (area=0x1ba71b0) at memarea.c:102
#4 0x0000000000493c64 in router_parse_entry_from_string (s=0x118 <Address 0x118 out of bounds>, end=0x8c0 <Address 0x8c0 out of bounds>, cache_copy=0,

allow_annotations=265, prepend_annotations=0x118 <Address 0x118 out of bounds>) at routerparse.c:1438

#5 0x0000000000494eb4 in router_parse_list_from_string (s=0x7fffff770788, eos=0x7f97f565f7ca "", dest=0x1ba63f0, saved_location=SAVED_IN_CACHE,

want_extrainfo=0, allow_annotations=1, prepend_annotations=0x0) at routerparse.c:1061

#6 0x000000000048cf05 in router_load_routers_from_string (

s=0x7f97f552d50d "@downloaded-at 2009-02-23 12:45:15\n@source \"12.166.156.100\"\nrouter MYCROFTsOtherChild 68.57.205.126 995 0 443\nplatform Tor 0.2.1.12-alpha (r18423) on FreeBSD i386\nopt protocols Link 1 2 Circuit 1\npubl"..., eos=0x7f97f565f7ca "", saved_location=SAVED_IN_CACHE,
requested_fingerprints=0x0, descriptor_digests=0, prepend_annotations=0x0) at routerlist.c:3500

#7 0x000000000048d37c in router_reload_router_list_impl (store=0x185b590) at routerlist.c:799
#8 0x000000000048d4b2 in router_reload_router_list () at routerlist.c:849
#9 0x00000000004616c5 in do_main_loop () at main.c:1396
#10 0x00000000004619f5 in tor_main (argc=3, argv=<value optimized out>) at main.c:2060
#11 0x00007f97f63d6466 in libc_start_main () from /lib/libc.so.6
#12 0x0000000000407469 in _start ()

#5 0x0000000000494eb4 in router_parse_list_from_string (s=0x7fffff770788, eos=0x7f97f565f7ca "", dest=0x1ba63f0, saved_location=SAVED_IN_CACHE,

want_extrainfo=0, allow_annotations=1, prepend_annotations=0x0) at routerparse.c:1061

1061 in routerparse.c
(gdb) print *s
$1 = 0x7f97f552d50d "@downloaded-at 2009-02-23 12:45:15\n@source \"12.166.156.100\"\nrouter MYCROFTsOtherChild 68.57.205.126 995 0 443\nplatform Tor 0.2.1.12-alpha (r18423) on FreeBSD i386\nopt protocols Link 1 2 Circuit 1\npubl"...

comment:6 Changed 11 years ago by phobos

with MAX_FREELIST_LEN 0 in memarea.c:

Feb 23 14:59:15.565 [notice] Opening Control listener on 127.0.0.1:9051
[New Thread 0x7fa49e75d6e0 (LWP 28321)]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fa49e75d6e0 (LWP 28321)]
0x00007fa49d3f9015 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007fa49d3f9015 in raise () from /lib/libc.so.6
#1 0x00007fa49d3fab83 in abort () from /lib/libc.so.6
#2 0x00007fa49d43fa80 in ?? () from /lib/libc.so.6
#3 0x00000000004ad12d in memarea_drop_all (area=0x23007c0) at memarea.c:102
#4 0x00000000004939f4 in router_parse_entry_from_string (s=0x19 <Address 0x19 out of bounds>, end=0xc8 <Address 0xc8 out of bounds>, cache_copy=0,

allow_annotations=11, prepend_annotations=0x19 <Address 0x19 out of bounds>) at routerparse.c:1438

#5 0x0000000000494c44 in router_parse_list_from_string (s=0x7fffa677c738, eos=0x7fa49cd8e7ca "", dest=0x22feee0, saved_location=SAVED_IN_CACHE,

want_extrainfo=0, allow_annotations=1, prepend_annotations=0x0) at routerparse.c:1061

#6 0x000000000048cc95 in router_load_routers_from_string (

s=0x7fa49cba77bd "@downloaded-at 2009-02-23 12:45:33\n@source \"97.115.129.188\"\nrouter torxmission 166.70.207.2 9001 0 0\nplatform Tor 0.1.1.23 on Linux i686\npublished 2009-02-22 23:18:23\nopt fingerprint 7B0F A328 23D3 7B"..., eos=0x7fa49cd8e7ca "", saved_location=SAVED_IN_CACHE,
requested_fingerprints=0x0, descriptor_digests=0, prepend_annotations=0x0) at routerlist.c:3500

#7 0x000000000048d10c in router_reload_router_list_impl (store=0x22764b0) at routerlist.c:799
#8 0x000000000048d242 in router_reload_router_list () at routerlist.c:849
#9 0x0000000000461415 in do_main_loop () at main.c:1396
#10 0x0000000000461745 in tor_main (argc=3, argv=<value optimized out>) at main.c:2060
#11 0x00007fa49d3e4466 in libc_start_main () from /lib/libc.so.6
#12 0x0000000000407469 in _start ()

#5 0x0000000000494c44 in router_parse_list_from_string (s=0x7fffa677c738, eos=0x7fa49cd8e7ca "", dest=0x22feee0, saved_location=SAVED_IN_CACHE,

want_extrainfo=0, allow_annotations=1, prepend_annotations=0x0) at routerparse.c:1061

1061 router = router_parse_entry_from_string(*s, end,
(gdb) print *s
$1 = 0x7fa49cba77bd "@downloaded-at 2009-02-23 12:45:33\n@source \"97.115.129.188\"\nrouter torxmission 166.70.207.2 9001 0 0\nplatform Tor 0.1.1.23 on Linux i686\npublished 2009-02-22 23:18:23\nopt fingerprint 7B0F A328 23D3 7B"...

comment:7 Changed 11 years ago by phobos

And with #define DEBUG_AREA_ALLOC in routerparse.c:

Feb 23 15:04:16.012 [notice] Opening Control listener on 127.0.0.1:9051
[New Thread 0x7fe37ecab6e0 (LWP 1862)]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fe37ecab6e0 (LWP 1862)]
0x00007fe37d947015 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007fe37d947015 in raise () from /lib/libc.so.6
#1 0x00007fe37d948b83 in abort () from /lib/libc.so.6
#2 0x00007fe37d98da80 in ?? () from /lib/libc.so.6
#3 0x00000000004ad66d in memarea_drop_all (area=0x2423800) at memarea.c:102
#4 0x0000000000493d87 in router_parse_entry_from_string (s=0x118 <Address 0x118 out of bounds>, end=0x8c0 <Address 0x8c0 out of bounds>, cache_copy=0,

allow_annotations=265, prepend_annotations=0x118 <Address 0x118 out of bounds>) at routerparse.c:1438

#5 0x0000000000495024 in router_parse_list_from_string (s=0x7fff86ccac88, eos=0x7fe37d2dc7ca "", dest=0x2421f20, saved_location=SAVED_IN_CACHE,

want_extrainfo=0, allow_annotations=1, prepend_annotations=0x0) at routerparse.c:1061

#6 0x000000000048cc95 in router_load_routers_from_string (

s=0x7fe37d1aa50d "@downloaded-at 2009-02-23 12:45:15\n@source \"12.166.156.100\"\nrouter MYCROFTsOtherChild 68.57.205.126 995 0 443\nplatform Tor 0.2.1.12-alpha (r18423) on FreeBSD i386\nopt protocols Link 1 2 Circuit 1\npubl"..., eos=0x7fe37d2dc7ca "", saved_location=SAVED_IN_CACHE,
requested_fingerprints=0x0, descriptor_digests=0, prepend_annotations=0x0) at routerlist.c:3500

#7 0x000000000048d10c in router_reload_router_list_impl (store=0x23972f0) at routerlist.c:799
#8 0x000000000048d242 in router_reload_router_list () at routerlist.c:849
#9 0x0000000000461415 in do_main_loop () at main.c:1396
#10 0x0000000000461745 in tor_main (argc=3, argv=<value optimized out>) at main.c:2060
#11 0x00007fe37d932466 in libc_start_main () from /lib/libc.so.6
#12 0x0000000000407469 in _start ()

#5 0x0000000000495024 in router_parse_list_from_string (s=0x7fff86ccac88, eos=0x7fe37d2dc7ca "", dest=0x2421f20, saved_location=SAVED_IN_CACHE,

want_extrainfo=0, allow_annotations=1, prepend_annotations=0x0) at routerparse.c:1061

1061 router = router_parse_entry_from_string(*s, end,
(gdb) print *s
$1 = 0x7fe37d1aa50d "@downloaded-at 2009-02-23 12:45:15\n@source \"12.166.156.100\"\nrouter MYCROFTsOtherChild 68.57.205.126 995 0 443\nplatform Tor 0.2.1.12-alpha (r18423) on FreeBSD i386\nopt protocols Link 1 2 Circuit 1\npubl"...

Changed 11 years ago by phobos

Attachment: flyspray-930-datadir.tgz added

cached* files from my datadir

comment:8 Changed 11 years ago by phobos

added the cached* files from my datadir.

comment:9 Changed 11 years ago by nickm

Hm. I have no idea what's going on here; this bug is totally weird. Would it be hard to figure out which of the
0.2.1.x-alpha series introduced this problem?

comment:10 Changed 11 years ago by phobos

Every single version in the 0.2.1.x series fails with the same error. I suspect it's my libc that broke things somehow.

Looking into what's changed in libc.

comment:11 Changed 11 years ago by nickm

But 0.2.0.34 still works? Okay, if you have time, it might be neat to go through the commits between when 0.2.1.x
forked and when 0.2.1.1-alpha came out. One of those must be to blame.

comment:12 Changed 11 years ago by edmanm

Just wanted to add a "Yeah, Tor is busted for me too on FC10" to this ticket. My bt looks quite
similar to the ones phobos pasted.

Mar 17 19:08:35.682 [notice] Tor v0.2.1.13-alpha-dev (r19068). This is experimental software. Do not rely on it for strong anonymity. (Running on Linux x86_64)
Mar 17 19:08:35.682 [notice] Configuration file "/usr/local/etc/tor/torrc" not present, using reasonable defaults.
Mar 17 19:08:35.682 [notice] Initialized libevent version 1.4.5-stable using method epoll. Good.
Mar 17 19:08:35.682 [notice] Opening Socks listener on 127.0.0.1:9050

(gdb) bt
#0 0x0000003729a32f05 in raise () from /lib64/libc.so.6
#1 0x0000003729a34a73 in abort () from /lib64/libc.so.6
#2 0x0000003729a77ef0 in malloc_printerr () from /lib64/libc.so.6
#3 0x00000000004a9b8d in chunk_free () at memarea.c:102
#4 memarea_drop_all (area=0x7875e0) at memarea.c:123
#5 0x000000000048fe34 in router_parse_entry_from_string (s=0x135 <Address 0x135 out of bounds>,

end=0x9a8 <Address 0x9a8 out of bounds>, cache_copy=0, allow_annotations=295,
prepend_annotations=0x135 <Address 0x135 out of bounds>) at routerparse.c:1438

#6 0x000000000049106c in router_parse_list_from_string (s=0x7fffffffdd18, eos=0x7ffff7fd1ac6 "",

dest=0x7878f0, saved_location=SAVED_IN_CACHE, want_extrainfo=0, allow_annotations=1,
prepend_annotations=0x0) at routerparse.c:1061

#7 0x0000000000489482 in router_load_routers_from_string (

s=0x7ffff7e7ed44 "@downloaded-at 2009-02-28 23:08:12\n@source \"81.25.55.235\"\nrouter che 81.233.224.95 443 0 80\nplatform Tor 0.2.0.33 (r18212) on Linux i686\nopt protocols Link 1 2 Circuit 1\npublished 2009-02-28 10:30:11\n"..., eos=0x7ffff7fd1ac6 "", saved_location=SAVED_IN_CACHE,
requested_fingerprints=0x0, descriptor_digests=0, prepend_annotations=0x0) at routerlist.c:3507

#8 0x00000000004898fc in router_reload_router_list_impl (store=0x746d30) at routerlist.c:805
#9 0x0000000000489a32 in router_reload_router_list () at routerlist.c:855
#10 0x000000000045ea15 in do_main_loop () at main.c:1396
#11 0x000000000045ed45 in tor_main (argc=1, argv=<value optimized out>) at main.c:2060
#12 0x0000003729a1e576 in libc_start_main () from /lib64/libc.so.6
#13 0x00000000004069e9 in _start ()

comment:13 Changed 11 years ago by nickm

edmanm: can _you_ figure out which Tor revision broke it for you, or use valgrind to figure out what's up?
(See above) I don't have the hardware to reproduce this.

comment:14 Changed 11 years ago by phobos

Any tor which uses memarea.c fails, this seems to be the entire 0.2.1.x-alpha branch. I notice libc-2.8.90.so seems to

work fine. If I upgrade it via backports, tor starts crashing again.

The version that works is filename: pool/main/g/glibc/libc6_2.8~20080505-0ubuntu7_amd64.deb
, even better, http://packages.ubuntu.com/intrepid/amd64/libc6 is working
http://packages.ubuntu.com/intrepid-updates/amd64/libc6 fails.

The diff between versions may help elucidate the issue, http://archive.ubuntu.com/ubuntu/pool/main/g/glibc/glibc_2.8~20080505-0ubuntu9.diff.gz

comment:15 Changed 11 years ago by nickm

r19074 adds some assertion statements that could conceivably help. Do they trigger for you guys?

Another thing to try: does replacing the definition of tor_malloc_roundup in util.h with the following help?

#define tor_malloc_roundup(szp) _tor_malloc((*szp) DMALLOC_ARGS)

comment:16 Changed 11 years ago by nickm

Also, does this happen really quickly, or do you have to wait a while for it to trigger?

comment:17 Changed 10 years ago by edmanm

It always happened for me immediately on startup, like in the log I pasted above (just after opening
the SOCKS listener).

Unfortunately, I made the mistake of running an instance of Tor 0.2.0.34 without first backing up my
~/.tor/ (I had some Vidalia stuff I had to test and needed a working Tor). Tor 0.2.0.34 ran fine, and
after going back to 0.2.1.13-alpha-dev I can't reproduce the crash anymore. Frowny face.

comment:18 Changed 10 years ago by nickm

Anything new on this bug? Is it still happening? The malloc_printerr() part of Matt's more-verbose backtrace is
pretty neat.

comment:19 Changed 10 years ago by nickm

d9650cfa50772d52b7f9ad7a6deeb5b3fdc751f3 on master adds sentinel values that might detect some possible causes
of this bug.

comment:20 Changed 10 years ago by nickm

I just added a fix for a bug that moria ran into that may be related.

comment:21 Changed 10 years ago by nickm

Okay. I've fixed a few seemingly 64-bit only allocation bugs since this appeared, and it doesn't seem to be
reprodicible any more. please repoen this or start a new bug if this comes back.

comment:22 Changed 10 years ago by nickm

flyspray2trac: bug closed.

comment:23 Changed 7 years ago by nickm

Component: Tor ClientTor
Note: See TracTickets for help on using tickets.