Opened 10 years ago

Closed 10 years ago

Last modified 7 years ago

#1342 closed defect (invalid)

Page allocation failure on KVM

Reported by: micah Owned by:
Priority: Low Milestone:
Component: Core Tor/Tor Version: 0.2.1.25
Severity: Keywords:
Cc: micah, Sebastian, arma, nickm Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I've setup tor on a KVM instance to act as an exit node. It works really well doing 30-40mbit/sec for hours.
My upstream is moving me to a 10gb/sec switch to help with that. The problem is that it runs for a few hours and
then dies, and it dies in an unhappy way... the networking stack on the kvm instance totally tanks, in otherwords
it stops responding on the net completely. No ping, no traffic can be observed using tcpdump, the counters on the
interface no longer change (although the interface stays up).

To get things back, I have to restart the kvm instance, no amount of module reloading, networking reconfiguring
will bring things back, it is one unhappy camper. No, there is no firewall involved. We've been going through replacing hardware, in case that is
the issue, but so far nothing has improved.

I tried to search for a similar bug on this issue, but I couldn't find anything. This very well could be a KVM
issue, or hardware, or something other than a tor issue, however I'm at a loss of where to start reporting this bug
so we'll start here and see what you think and then if you have determined it is not a tor issue, I can shop
it around other places, until we've chased it into a corner where we can kill it.

This is what the interface looks like:

eth0 Link encap:Ethernet HWaddr 52:54:00:43:ae:3d

inet addr:77.109.139.87 Bcast:77.109.139.95 Mask:255.255.255.240
inet6 addr: 2001:1620:2018:2::4d6d:8b57/64 Scope:Global
inet6 addr: fe80::5054:ff:fe43:ae3d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12059348 errors:0 dropped:0 overruns:0 frame:0
TX packets:12971887 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8170402534 (7.6 GiB) TX bytes:8712200168 (8.1 GiB)

There is a kernel crash that is produced everytime, here is a copy of one of them:

[34826.883040] tor: page allocation failure. order:0, mode:0x20
[34826.884390] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34826.885721] Call Trace:
[34826.886438] <IRQ> [<ffffffff80297eed>] ? alloc_pages_internal+0x404/0x427
[34826.887852] [<ffffffffa0213e68>] ? try_fill_recv+0x98/0x179 [virtio_net]
[34826.889281] [<ffffffffa0214a4f>] ? virtnet_poll+0x52d/0x5a2 [virtio_net]
[34826.890633] [<ffffffff80412048>] ? net_rx_action+0xa8/0x19f
[34826.891824] [<ffffffffa0048215>] ? vring_interrupt+0x26/0x2f [virtio_ring]
[34826.893320] [<ffffffff80246f81>] ?
do_softirq+0xac/0x173
[34826.894501] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34826.895713] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34826.897000] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34826.898077] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34826.899105] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34826.900207] <EOI> <6>Mem-Info:
[34826.900981] Node 0 DMA per-cpu:
[34826.901782] CPU 0: hi: 0, btch: 1 usd: 0
[34826.902883] Node 0 DMA32 per-cpu:
[34826.903680] CPU 0: hi: 90, btch: 15 usd: 75
[34826.904712] Active_anon:12736 active_file:10956 inactive_anon:14498
[34826.904714] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34826.904719] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34826.908663] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34826.912440] lowmem_reserve[]: 0 236 236 236
[34826.913482] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34826.917520] lowmem_reserve[]: 0 0 0 0
[34826.918627] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34826.921154] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34826.923867] 22182 total pagecache pages
[34826.924716] 4532 pages in swap cache
[34826.925531] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34826.926934] Free swap = 2058480kB
[34826.927694] Total swap = 2093048kB
[34826.930623] 65520 pages RAM
[34826.931501] 2926 pages reserved
[34826.932394] 3684 pages shared
[34826.933143] 58136 pages non-shared
[34826.934169] tor: page allocation failure. order:0, mode:0x20
[34826.935583] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34826.936385] Call Trace:
[34826.936385] <IRQ> [<ffffffff80297eed>] ? alloc_pages_internal+0x404/0x427
[34826.936385] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34826.936385] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34826.936385] [<ffffffff802bb4ec>] ?
cache_alloc_node+0x110/0x12f
[34826.936385] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34826.936385] [<ffffffff8040b68e>] ?
alloc_skb+0x3f/0x138
[34826.936385] [<ffffffff80406e84>] ? sock_wmalloc+0x27/0x5f
[34826.936385] [<ffffffff80435ed9>] ? ip_append_data+0x552/0x9be
[34826.936385] [<ffffffff804353e7>] ? ip_reply_glue_bits+0x0/0x4f
[34826.936385] [<ffffffff80430d03>] ? ip_route_output_flow+0x77/0x1cb
[34826.936385] [<ffffffff804372fa>] ? ip_send_reply+0x1b0/0x242
[34826.936385] [<ffffffff80406c26>] ? sock_def_readable+0x37/0x62
[34826.936385] [<ffffffff8044b39d>] ? tcp_v4_send_reset+0x1dd/0x21e
[34826.936385] [<ffffffff80439e42>] ? inet_lookup_established+0x43/0x20f
[34826.936385] [<ffffffff8044d82f>] ? tcp_v4_rcv+0x6a9/0x6db
[34826.936385] [<ffffffff80433189>] ? ip_local_deliver_finish+0x0/0x1e9
[34826.936385] [<ffffffff80433189>] ? ip_local_deliver_finish+0x0/0x1e9
[34826.936385] [<ffffffff804332cf>] ? ip_local_deliver_finish+0x146/0x1e9
[34826.936385] [<ffffffff80432e45>] ? ip_rcv_finish+0x321/0x33b
[34826.936385] [<ffffffffa02149e6>] ? virtnet_poll+0x4c4/0x5a2 [virtio_net]
[34826.936385] [<ffffffff80412048>] ? net_rx_action+0xa8/0x19f
[34826.936385] [<ffffffffa0048215>] ? vring_interrupt+0x26/0x2f [virtio_ring]
[34826.936385] [<ffffffff80246f81>] ?
do_softirq+0xac/0x173
[34826.936385] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34826.936385] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34826.936385] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34826.936385] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34826.936385] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34826.936385] <EOI> <6>Mem-Info:
[34826.936385] Node 0 DMA per-cpu:
[34826.936385] CPU 0: hi: 0, btch: 1 usd: 0
[34826.936385] Node 0 DMA32 per-cpu:
[34826.936385] CPU 0: hi: 90, btch: 15 usd: 75
[34826.936385] Active_anon:12736 active_file:10956 inactive_anon:14498
[34826.936385] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34826.936385] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34826.936385] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34826.936385] lowmem_reserve[]: 0 236 236 236
[34826.936385] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34826.936385] lowmem_reserve[]: 0 0 0 0
[34826.936385] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34826.936385] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34826.936385] 22182 total pagecache pages
[34826.936385] 4532 pages in swap cache
[34826.936385] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34826.936385] Free swap = 2058480kB
[34826.936385] Total swap = 2093048kB
[34826.936385] 65520 pages RAM
[34826.936385] 2926 pages reserved
[34826.936385] 3684 pages shared
[34826.936385] 58136 pages non-shared
[34827.003183] tor: page allocation failure. order:0, mode:0x20
[34827.004520] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34827.005687] Call Trace:
[34827.006325] <IRQ> [<ffffffff80297eed>] ? alloc_pages_internal+0x404/0x427
[34827.006938] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34827.006938] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34827.006938] [<ffffffff802bb4ec>] ?
cache_alloc_node+0x110/0x12f
[34827.006938] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34827.006938] [<ffffffff8040b68e>] ?
alloc_skb+0x3f/0x138
[34827.006938] [<ffffffff80406f66>] ? sock_alloc_send_pskb+0xaa/0x2c3
[34827.006938] [<ffffffff80435e9b>] ? ip_append_data+0x514/0x9be
[34827.006938] [<ffffffff8045569b>] ? icmp_glue_bits+0x0/0x76
[34827.006938] [<ffffffff8045d7a8>] ? inet_dev_addr_type+0xaf/0xbf
[34827.006938] [<ffffffff804555f2>] ? icmp_push_reply+0x5c/0x105
[34827.006938] [<ffffffff8045648c>] ? icmp_send+0x524/0x558
[34827.006938] [<ffffffff804461e1>] ? tcp_rcv_established+0x688/0x6d9
[34827.006938] [<ffffffff804527a0>] ?
udp4_lib_rcv+0x45a/0x5dc
[34827.006938] [<ffffffff80433189>] ? ip_local_deliver_finish+0x0/0x1e9
[34827.006938] [<ffffffff804332cf>] ? ip_local_deliver_finish+0x146/0x1e9
[34827.006938] [<ffffffff80432e45>] ? ip_rcv_finish+0x321/0x33b
[34827.006938] [<ffffffffa02149e6>] ? virtnet_poll+0x4c4/0x5a2 [virtio_net]
[34827.006938] [<ffffffff80412048>] ? net_rx_action+0xa8/0x19f
[34827.006938] [<ffffffffa0048215>] ? vring_interrupt+0x26/0x2f [virtio_ring]
[34827.006938] [<ffffffff80246f81>] ? do_softirq+0xac/0x173
[34827.006938] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34827.006938] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34827.006938] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34827.006938] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34827.006938] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34827.006938] <EOI> <6>Mem-Info:
[34827.006938] Node 0 DMA per-cpu:
[34827.006938] CPU 0: hi: 0, btch: 1 usd: 0
[34827.006938] Node 0 DMA32 per-cpu:
[34827.006938] CPU 0: hi: 90, btch: 15 usd: 75
[34827.006938] Active_anon:12736 active_file:10956 inactive_anon:14498
[34827.006938] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34827.006938] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34827.006938] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34827.006938] lowmem_reserve[]: 0 236 236 236
[34827.006938] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34827.006938] lowmem_reserve[]: 0 0 0 0
[34827.006938] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34827.006938] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34827.006938] 22182 total pagecache pages
[34827.006938] 4532 pages in swap cache
[34827.006938] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34827.006938] Free swap = 2058480kB
[34827.006938] Total swap = 2093048kB
[34827.006938] 65520 pages RAM
[34827.006938] 2926 pages reserved
[34827.006938] 3684 pages shared
[34827.006938] 58136 pages non-shared
[34827.091968] tor: page allocation failure. order:0, mode:0x20
[34827.093195] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34827.094105] Call Trace:
[34827.094105] <IRQ> [<ffffffff80297eed>] ?
alloc_pages_internal+0x404/0x427
[34827.094105] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34827.094105] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34827.094105] [<ffffffff802bb4ec>] ? cache_alloc_node+0x110/0x12f
[34827.094105] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34827.094105] [<ffffffff8040b68e>] ? alloc_skb+0x3f/0x138
[34827.094105] [<ffffffff80433189>] ? ip_local_deliver_finish+0x0/0x1e9
[34827.094105] [<ffffffff8040c25d>] ?
netdev_alloc_skb+0x29/0x43
[34827.094105] [<ffffffffa0213e19>] ? try_fill_recv+0x49/0x179 [virtio_net]
[34827.094105] [<ffffffffa0214a4f>] ? virtnet_poll+0x52d/0x5a2 [virtio_net]
[34827.094105] [<ffffffff80412048>] ? net_rx_action+0xa8/0x19f
[34827.094105] [<ffffffffa0048215>] ? vring_interrupt+0x26/0x2f [virtio_ring]
[34827.094105] [<ffffffff80246f81>] ? do_softirq+0xac/0x173
[34827.094105] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34827.094105] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34827.094105] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34827.094105] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34827.094105] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34827.094105] <EOI> <6>Mem-Info:
[34827.094105] Node 0 DMA per-cpu:
[34827.094105] CPU 0: hi: 0, btch: 1 usd: 0
[34827.094105] Node 0 DMA32 per-cpu:
[34827.094105] CPU 0: hi: 90, btch: 15 usd: 75
[34827.094105] Active_anon:12736 active_file:10956 inactive_anon:14498
[34827.094105] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34827.094105] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34827.094105] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34827.094105] lowmem_reserve[]: 0 236 236 236
[34827.094105] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34827.094105] lowmem_reserve[]: 0 0 0 0
[34827.094105] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34827.094105] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34827.094105] 22182 total pagecache pages
[34827.094105] 4532 pages in swap cache
[34827.094105] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34827.094105] Free swap = 2058480kB
[34827.094105] Total swap = 2093048kB
[34827.094105] 65520 pages RAM
[34827.094105] 2926 pages reserved
[34827.094105] 3684 pages shared
[34827.094105] 58136 pages non-shared
[34827.148063] tor: page allocation failure. order:0, mode:0x20
[34827.149429] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34827.150608] Call Trace:
[34827.151259] <IRQ> [<ffffffff80297eed>] ?
alloc_pages_internal+0x404/0x427
[34827.152014] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34827.152014] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34827.152014] [<ffffffff802bb4ec>] ? cache_alloc_node+0x110/0x12f
[34827.152014] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34827.152014] [<ffffffff8040b68e>] ? alloc_skb+0x3f/0x138
[34827.152014] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.152014] [<ffffffff80447cd1>] ? tcp_send_ack+0x23/0xf4
[34827.152014] [<ffffffff8044a7bc>] ? tcp_delack_timer+0x18e/0x1fe
[34827.152014] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.152014] [<ffffffff8024aa5f>] ? run_timer_softirq+0x193/0x210
[34827.152014] [<ffffffff80246f81>] ?
do_softirq+0xac/0x173
[34827.152014] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34827.152014] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34827.152014] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34827.152014] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34827.152014] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34827.152014] <EOI> <6>Mem-Info:
[34827.152014] Node 0 DMA per-cpu:
[34827.152014] CPU 0: hi: 0, btch: 1 usd: 0
[34827.152014] Node 0 DMA32 per-cpu:
[34827.152014] CPU 0: hi: 90, btch: 15 usd: 75
[34827.152014] Active_anon:12736 active_file:10956 inactive_anon:14498
[34827.152014] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34827.152014] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34827.152014] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34827.152014] lowmem_reserve[]: 0 236 236 236
[34827.152014] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34827.152014] lowmem_reserve[]: 0 0 0 0
[34827.152014] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34827.152014] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34827.152014] 22182 total pagecache pages
[34827.152014] 4532 pages in swap cache
[34827.152014] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34827.152014] Free swap = 2058480kB
[34827.152014] Total swap = 2093048kB
[34827.152014] 65520 pages RAM
[34827.152014] 2926 pages reserved
[34827.152014] 3684 pages shared
[34827.152014] 58136 pages non-shared
[34827.203077] tor: page allocation failure. order:0, mode:0x20
[34827.204420] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34827.205629] Call Trace:
[34827.206290] <IRQ> [<ffffffff80297eed>] ? alloc_pages_internal+0x404/0x427
[34827.206898] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34827.206898] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34827.206898] [<ffffffff802bb4ec>] ?
cache_alloc_node+0x110/0x12f
[34827.206898] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34827.206898] [<ffffffff8040b68e>] ?
alloc_skb+0x3f/0x138
[34827.206898] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.206898] [<ffffffff80447cd1>] ? tcp_send_ack+0x23/0xf4
[34827.206898] [<ffffffff8044a7bc>] ? tcp_delack_timer+0x18e/0x1fe
[34827.206898] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.206898] [<ffffffff8024aa5f>] ? run_timer_softirq+0x193/0x210
[34827.206898] [<ffffffff80246f81>] ? do_softirq+0xac/0x173
[34827.206898] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34827.206898] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34827.206898] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34827.206898] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34827.206898] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34827.206898] <EOI> <6>Mem-Info:
[34827.206898] Node 0 DMA per-cpu:
[34827.206898] CPU 0: hi: 0, btch: 1 usd: 0
[34827.206898] Node 0 DMA32 per-cpu:
[34827.206898] CPU 0: hi: 90, btch: 15 usd: 75
[34827.206898] Active_anon:12736 active_file:10956 inactive_anon:14498
[34827.206898] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34827.206898] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34827.206898] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34827.206898] lowmem_reserve[]: 0 236 236 236
[34827.206898] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34827.206898] lowmem_reserve[]: 0 0 0 0
[34827.206898] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34827.206898] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34827.206898] 22182 total pagecache pages
[34827.206898] 4532 pages in swap cache
[34827.206898] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34827.206898] Free swap = 2058480kB
[34827.206898] Total swap = 2093048kB
[34827.206898] 65520 pages RAM
[34827.206898] 2926 pages reserved
[34827.206898] 3684 pages shared
[34827.206898] 58136 pages non-shared
[34827.257994] tor: page allocation failure. order:0, mode:0x20
[34827.259096] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34827.260325] Call Trace:
[34827.260985] <IRQ> [<ffffffff80297eed>] ?
alloc_pages_internal+0x404/0x427
[34827.261933] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34827.261933] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34827.261933] [<ffffffff802bb4ec>] ? cache_alloc_node+0x110/0x12f
[34827.261933] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34827.261933] [<ffffffff8040b68e>] ? alloc_skb+0x3f/0x138
[34827.261933] [<ffffffff8024b217>] ?
mod_timer+0xc2/0xd1
[34827.261933] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.261933] [<ffffffff80447cd1>] ? tcp_send_ack+0x23/0xf4
[34827.261933] [<ffffffff8044a7bc>] ? tcp_delack_timer+0x18e/0x1fe
[34827.261933] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.261933] [<ffffffff8024aa5f>] ? run_timer_softirq+0x193/0x210
[34827.261933] [<ffffffff80246f81>] ? do_softirq+0xac/0x173
[34827.261933] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34827.261933] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34827.261933] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34827.261933] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34827.261933] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34827.261933] <EOI> <6>Mem-Info:
[34827.261933] Node 0 DMA per-cpu:
[34827.261933] CPU 0: hi: 0, btch: 1 usd: 0
[34827.261933] Node 0 DMA32 per-cpu:
[34827.261933] CPU 0: hi: 90, btch: 15 usd: 75
[34827.261933] Active_anon:12736 active_file:10956 inactive_anon:14498
[34827.261933] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34827.261933] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34827.261933] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34827.261933] lowmem_reserve[]: 0 236 236 236
[34827.261933] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34827.261933] lowmem_reserve[]: 0 0 0 0
[34827.261933] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34827.261933] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34827.261933] 22182 total pagecache pages
[34827.261933] 4532 pages in swap cache
[34827.261933] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34827.261933] Free swap = 2058480kB
[34827.261933] Total swap = 2093048kB
[34827.261933] 65520 pages RAM
[34827.261933] 2926 pages reserved
[34827.261933] 3684 pages shared
[34827.261933] 58136 pages non-shared
[34827.337350] tor: page allocation failure. order:0, mode:0x20
[34827.338718] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34827.340014] Call Trace:
[34827.340716] <IRQ> [<ffffffff80297eed>] ?
alloc_pages_internal+0x404/0x427
[34827.341184] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34827.341184] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34827.341184] [<ffffffff802bb4ec>] ? cache_alloc_node+0x110/0x12f
[34827.341184] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34827.341184] [<ffffffff8040b68e>] ? alloc_skb+0x3f/0x138
[34827.341184] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.341184] [<ffffffff80447cd1>] ? tcp_send_ack+0x23/0xf4
[34827.341184] [<ffffffff8044a7bc>] ? tcp_delack_timer+0x18e/0x1fe
[34827.341184] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.341184] [<ffffffff8024aa5f>] ? run_timer_softirq+0x193/0x210
[34827.341184] [<ffffffff80246f81>] ?
do_softirq+0xac/0x173
[34827.341184] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34827.341184] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34827.341184] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34827.341184] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34827.341184] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34827.341184] <EOI> <6>Mem-Info:
[34827.341184] Node 0 DMA per-cpu:
[34827.341184] CPU 0: hi: 0, btch: 1 usd: 0
[34827.341184] Node 0 DMA32 per-cpu:
[34827.341184] CPU 0: hi: 90, btch: 15 usd: 75
[34827.341184] Active_anon:12736 active_file:10956 inactive_anon:14498
[34827.341184] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34827.341184] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34827.341184] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34827.341184] lowmem_reserve[]: 0 236 236 236
[34827.341184] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34827.341184] lowmem_reserve[]: 0 0 0 0
[34827.341184] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34827.341184] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34827.341184] 22182 total pagecache pages
[34827.341184] 4532 pages in swap cache
[34827.341184] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34827.341184] Free swap = 2058480kB
[34827.341184] Total swap = 2093048kB
[34827.341184] 65520 pages RAM
[34827.341184] 2926 pages reserved
[34827.341184] 3684 pages shared
[34827.341184] 58136 pages non-shared
[34827.395146] tor: page allocation failure. order:0, mode:0x20
[34827.396357] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34827.397573] Call Trace:
[34827.398269] <IRQ> [<ffffffff80297eed>] ? alloc_pages_internal+0x404/0x427
[34827.399084] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34827.399084] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34827.399084] [<ffffffff802bb4ec>] ?
cache_alloc_node+0x110/0x12f
[34827.399084] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34827.399084] [<ffffffff8040b68e>] ?
alloc_skb+0x3f/0x138
[34827.399084] [<ffffffff8024b217>] ? mod_timer+0xc2/0xd1
[34827.399084] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.399084] [<ffffffff80447cd1>] ? tcp_send_ack+0x23/0xf4
[34827.399084] [<ffffffff8044a7bc>] ? tcp_delack_timer+0x18e/0x1fe
[34827.399084] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.399084] [<ffffffff8024aa5f>] ? run_timer_softirq+0x193/0x210
[34827.399084] [<ffffffff80246f81>] ?
do_softirq+0xac/0x173
[34827.399084] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34827.399084] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34827.399084] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34827.399084] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34827.399084] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34827.399084] <EOI> <6>Mem-Info:
[34827.399084] Node 0 DMA per-cpu:
[34827.399084] CPU 0: hi: 0, btch: 1 usd: 0
[34827.399084] Node 0 DMA32 per-cpu:
[34827.399084] CPU 0: hi: 90, btch: 15 usd: 75
[34827.399084] Active_anon:12736 active_file:10956 inactive_anon:14498
[34827.399084] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34827.399084] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34827.399084] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no
[34827.399084] lowmem_reserve[]: 0 236 236 236
[34827.399084] Node 0 DMA32 free:676kB min:1928kB low:2408kB high:2892kB active_anon:50940kB inactive_anon:57936kB active_file:43164kB inactive_file:26020kB unevictable:0kB present:242336kB pages_scanned:0 all_unreclaimable? no
[34827.399084] lowmem_reserve[]: 0 0 0 0
[34827.399084] Node 0 DMA: 9*4kB 9*8kB 6*16kB 0*32kB 6*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 972kB
[34827.399084] Node 0 DMA32: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 676kB
[34827.399084] 22182 total pagecache pages
[34827.399084] 4532 pages in swap cache
[34827.399084] Swap cache stats: add 78235, delete 73703, find 259918/266546
[34827.399084] Free swap = 2058480kB
[34827.399084] Total swap = 2093048kB
[34827.399084] 65520 pages RAM
[34827.399084] 2926 pages reserved
[34827.399084] 3684 pages shared
[34827.399084] 58136 pages non-shared
[34827.453135] tor: page allocation failure. order:0, mode:0x20
[34827.454275] Pid: 3309, comm: tor Not tainted 2.6.30-bpo.2-amd64 #1
[34827.455476] Call Trace:
[34827.456200] <IRQ> [<ffffffff80297eed>] ? alloc_pages_internal+0x404/0x427
[34827.457103] [<ffffffff802bb14c>] ? kmem_getpages+0x68/0x12b
[34827.457103] [<ffffffff802bb362>] ? fallback_alloc+0x153/0x1cd
[34827.457103] [<ffffffff802bb4ec>] ?
cache_alloc_node+0x110/0x12f
[34827.457103] [<ffffffff802bb5b6>] ? kmem_cache_alloc_node+0xab/0x11e
[34827.457103] [<ffffffff8040b68e>] ?
alloc_skb+0x3f/0x138
[34827.457103] [<ffffffff8024b217>] ? mod_timer+0xc2/0xd1
[34827.457103] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.457103] [<ffffffff80447cd1>] ? tcp_send_ack+0x23/0xf4
[34827.457103] [<ffffffff8044a7bc>] ? tcp_delack_timer+0x18e/0x1fe
[34827.457103] [<ffffffff8044a62e>] ? tcp_delack_timer+0x0/0x1fe
[34827.457103] [<ffffffff8024aa5f>] ? run_timer_softirq+0x193/0x210
[34827.457103] [<ffffffff80246f81>] ?
do_softirq+0xac/0x173
[34827.457103] [<ffffffff80210bcc>] ? call_softirq+0x1c/0x30
[34827.457103] [<ffffffff802125fa>] ? do_softirq+0x3a/0x7e
[34827.457103] [<ffffffff80246cfe>] ? irq_exit+0x3f/0x80
[34827.457103] [<ffffffff80211ece>] ? do_IRQ+0xa9/0xbf
[34827.457103] [<ffffffff80210453>] ? ret_from_intr+0x0/0x11
[34827.457103] <EOI> <6>Mem-Info:
[34827.457103] Node 0 DMA per-cpu:
[34827.457103] CPU 0: hi: 0, btch: 1 usd: 0
[34827.457103] Node 0 DMA32 per-cpu:
[34827.457103] CPU 0: hi: 90, btch: 15 usd: 75
[34827.457103] Active_anon:12736 active_file:10956 inactive_anon:1449842:11
[34827.457103] inactive_file:6686 unevictable:0 dirty:0 writeback:0 unstable:0
[34827.457103] free:412 slab:9571 mapped:1765 pagetables:343 bounce:0
[34827.457103] Node 0 DMA free:972kB min:72kB low:88kB high:108kB active_anon:4kB inactive_anon:56kB active_file:660kB inactive_file:724kB unevictable:0kB present:9384kB pages_scanned:0 all_unreclaimable? no

[Automatically added by flyspray2trac: Operating System: Other Linux]

Child Tickets

Change History (4)

comment:1 Changed 10 years ago by micah

I've installed 0.2.2.10-alpha-2 to see if that changes things. I shall also install a newer kernel and newer kvm, if there is any to see if it helps.

comment:2 Changed 10 years ago by micah

Ok, 0.2.2.10-alpha-2 didn't change anything.
host and kvm instance upgraded to 2.6.32-bpo.2-amd64, no change
upgraded to kvm 0.12.3+dfsg-4~bpo50+1, no change

comment:3 Changed 10 years ago by micah

Resolution: Noneinvalid
Status: newclosed

Ok, after a lot of experimenting over the last few weeks, I've come to determine that this is a kernel problem with Debian's kernel, specifically the virtio network driver. If I switch the kvm guest to using the e1000e driver instead, the page allocation faults go away.

This may be a relevant fix for Redhat: https://bugzilla.redhat.com/show_bug.cgi?id=554078

Closing as this is not a tor bug.

comment:4 Changed 7 years ago by nickm

Component: Tor RelayTor
Note: See TracTickets for help on using tickets.