Opened 9 years ago

Closed 9 years ago

Last modified 7 years ago

#1789 closed defect (fixed)

Wake-up from Hibernation Occurs Day 1 Each Month

Reported by: BarkerJr Owned by:
Priority: Low Milestone: Tor: 0.2.2.x-final
Component: Core Tor/Tor Version: Tor: 0.2.2.14-alpha
Severity: Keywords: tor-relay
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

This is happening on several relays, on both 64-bit and 32-bit CentOS Linux.

Apr 18 12:48:15.014 [notice] Configured hibernation. This interval begins at 2010-04-18 00:00:00 and ends at 2010-04-19 00:00:00. We have no prior estimate for bandwidth, so we will start out awake and hibernate when we exhaust our quota.
Apr 18 13:04:04.244 [notice] Configured hibernation. This interval begins at 2010-04-18 00:00:00 and ends at 2010-04-19 00:00:00. We have no prior estimate for bandwidth, so we will start out awake and hibernate when we exhaust our quota.
Apr 19 00:00:00.995 [notice] Configured hibernation. This interval began at 2010-04-19 00:00:00; the scheduled wake-up time was 2010-04-19 00:00:00; we expect to exhaust our quota for this interval around 2010-04-20 00:00:00; the next interval begins at 2010-04-20 00:00:00 (all times local)
Apr 19 10:55:19.797 [notice] Configured hibernation. This interval began at 2010-04-01 00:00:00; the scheduled wake-up time was 2010-04-01 00:00:00; we expect to exhaust our quota for this interval around 2010-05-01 00:00:00; the next interval begins at 2010-05-01 00:00:00 (all times local)
Apr 21 01:39:36.221 [notice] Configured hibernation. This interval began at 2010-04-01 00:00:00; the scheduled wake-up time was 2010-04-01 00:00:00; we expect to exhaust our quota for this interval around 2010-05-01 00:00:00; the next interval begins at 2010-05-01 00:00:00 (all times local)
Apr 22 21:11:26.494 [notice] Configured hibernation. This interval began at 2010-04-01 00:00:00; the scheduled wake-up time was 2010-04-01 00:00:00; we expect to exhaust our quota for this interval around 2010-05-01 00:00:00; the next interval begins at 2010-05-01 00:00:00 (all times local)
Apr 24 22:00:39.128 [notice] Configured hibernation. This interval began at 2010-04-01 00:00:00; the scheduled wake-up time was 2010-04-01 00:00:00; we expect to exhaust our quota for this interval around 2010-05-01 00:00:00; the next interval begins at 2010-05-01 00:00:00 (all times local)
Apr 25 01:37:16.835 [notice] Configured hibernation. This interval began at 2010-04-01 00:00:00; the scheduled wake-up time was 2010-04-03 20:22:28; we expect to exhaust our quota for this interval around 2010-04-27 11:45:28; the next interval begins at 2010-05-01 00:00:00 (all times local)
Apr 27 09:33:10.377 [notice] Bandwidth soft limit reached; commencing hibernation.
May 01 00:00:00.636 [notice] Configured hibernation. This interval began at 2010-05-01 00:00:00; the scheduled wake-up time is 2010-05-06 15:25:09; we expect to exhaust our quota for this interval around 2010-05-18 16:29:09; the next interval begins at 2010-06-01 00:00:00 (all times local)
May 01 00:00:00.667 [notice] Accounting period ended. Commencing hibernation until 2010-05-06 15:25:09 GMT
May 02 19:38:30.791 [notice] Configured hibernation. This interval began at 2010-05-01 00:00:00; the scheduled wake-up time is 2010-05-06 15:25:09; we expect to exhaust our quota for this interval around 2010-05-18 16:29:09; the next interval begins at 2010-06-01 00:00:00 (all times local)
May 02 19:38:31.263 [notice] Commencing hibernation. We will wake up at 2010-05-06 15:25:09 local time.
May 06 15:25:09.423 [notice] Hibernation period ended. Resuming normal activity.
May 08 22:06:49.155 [notice] Bandwidth soft limit reached; commencing hibernation.
Jun 01 00:00:00.981 [notice] Configured hibernation. This interval began at 2010-06-01 00:00:00; the scheduled wake-up time was 2010-06-01 00:00:00; we expect to exhaust our quota for this interval around 2010-07-01 00:00:00; the next interval begins at 2010-07-01 00:00:00 (all times local)
Jun 01 00:00:01.052 [notice] Hibernation period ended. Resuming normal activity.
Jun 02 10:59:45.826 [notice] Configured hibernation. This interval began at 2010-06-01 00:00:00; the scheduled wake-up time was 2010-06-01 00:00:00; we expect to exhaust our quota for this interval around 2010-07-01 00:00:00; the next interval begins at 2010-07-01 00:00:00 (all times local)
Jun 05 07:24:03.517 [notice] Bandwidth soft limit reached; commencing hibernation.
Jun 07 00:39:45.337 [notice] Configured hibernation. This interval began at 2010-06-01 00:00:00; the scheduled wake-up time was 2010-06-01 00:00:00; we expect to exhaust our quota for this interval around 2010-07-01 00:00:00; the next interval begins at 2010-07-01 00:00:00 (all times local)
Jun 07 00:39:46.019 [notice] Bandwidth soft limit reached; commencing hibernation.
Jun 20 17:38:44.317 [notice] Configured hibernation. This interval began at 2010-06-01 00:00:00; the scheduled wake-up time was 2010-06-01 00:00:00; we expect to exhaust our quota for this interval around 2010-07-01 00:00:00; the next interval begins at 2010-07-01 00:00:00 (all times local)
Jun 20 17:38:47.640 [notice] Bandwidth soft limit reached; commencing hibernation.
Jul 01 00:00:00.388 [notice] Configured hibernation. This interval began at 2010-07-01 00:00:00; the scheduled wake-up time is 2010-07-01 04:07:55; we expect to exhaust our quota for this interval around 2010-07-31 22:47:55; the next interval begins at 2010-08-01 00:00:00 (all times local)
Jul 01 00:00:00.528 [notice] Accounting period ended. Commencing hibernation until 2010-07-01 04:07:55 GMT
Jul 01 04:07:55.607 [notice] Hibernation period ended. Resuming normal activity.
Jul 08 07:12:20.943 [notice] Bandwidth soft limit reached; commencing hibernation.
Jul 15 00:12:31.450 [notice] Configured hibernation. This interval began at 2010-07-01 00:00:00; the scheduled wake-up time was 2010-07-01 04:07:55; we expect to exhaust our quota for this interval around 2010-07-31 22:47:55; the next interval begins at 2010-08-01 00:00:00 (all times local)
Jul 15 00:12:32.688 [notice] Bandwidth soft limit reached; commencing hibernation.
Aug 01 00:00:00.152 [notice] Configured hibernation. This interval began at 2010-08-01 00:00:00; the scheduled wake-up time was 2010-08-01 00:00:00; we expect to exhaust our quota for this interval around 2010-09-01 00:00:00; the next interval begins at 2010-09-01 00:00:00 (all times local)
Aug 01 00:00:00.184 [notice] Hibernation period ended. Resuming normal activity.

Child Tickets

Change History (32)

comment:1 Changed 9 years ago by nickm

What is the accounting and bandwidth configuration that produces this output?

Having wake-up happen one day each month is not necessarily surprising on its own. Suppose that you configure your BandwidthRate equal to your AccountingMax / (24*60*60) -- this pretty much means that you're okay with using up your bandwidth in a single day.

The part of this that could be a bug, though, is the part where we keep waking up at the start of the month. It's supposed to be a random point in the month.

comment:2 Changed 9 years ago by BarkerJr

This is my only related setting:
AccountingMax 200GB

comment:3 Changed 9 years ago by nickm

What's the bandwidth on this host?

comment:4 in reply to:  3 Changed 9 years ago by BarkerJr

I'm not really sure. I just downloaded a 100MB file from within the data centre at 130mbit, which is probably just the speed of the disk drive.

comment:5 Changed 9 years ago by nickm

I mean, is there a bandwidthrate line (or a bandwidth anything) in your torrc ?

comment:6 Changed 9 years ago by nickm

Milestone: Tor: 0.2.2.x-final

comment:7 in reply to:  description Changed 9 years ago by mwenge

Replying to BarkerJr:

This is happening on several relays, on both 64-bit and 32-bit CentOS Linux.

Jun 01 00:00:00.981 [notice] Configured hibernation. This interval began at 2010-06-01 00:00:00; the scheduled wake-up time was 2010-06-01 00:00:00; we expect to exhaust our quota for this interval around 2010-07-01 00:00:00; the next interval begins at 2010-07-01 00:00:00 (all times local)
Jul 01 00:00:00.388 [notice] Configured hibernation. This interval began at 2010-07-01 00:00:00; the scheduled wake-up time is 2010-07-01 04:07:55; we expect to exhaust our quota for this interval around 2010-07-31 22:47:55; the next interval begins at 2010-08-01 00:00:00 (all times local)
Aug 01 00:00:00.152 [notice] Configured hibernation. This interval began at 2010-08-01 00:00:00; the scheduled wake-up time was 2010-08-01 00:00:00; we expect to exhaust our quota for this interval around 2010-09-01 00:00:00; the next interval begins at 2010-09-01 00:00:00 (all times local)
Aug 01 00:00:00.184 [notice] Hibernation period ended. Resuming normal activity.

I think the reason it woke up from hibernation at the start of each month is that it was agnostic in June and then very optimistic in July and August about your bandwidth usage vs your stated capacity of 200GB for the period. So in order to squeeze all that capacity on a relay that has quite low observed bandwidth it allotted an accounting period of almost the entire month in each case. That behaviour conforms with my understanding of the code.

The real problem, it seems to me, is why it got your expected usage so wrong in July when it had the whole of June to learn. I also don't understand why it went into low bandwidth mode twice in June and July.

Jun 05 07:24:03.517 [notice] Bandwidth soft limit reached; commencing hibernation.
Jun 20 17:38:47.640 [notice] Bandwidth soft limit reached; commencing hibernation.
Jul 08 07:12:20.943 [notice] Bandwidth soft limit reached; commencing hibernation.
Jul 15 00:12:32.688 [notice] Bandwidth soft limit reached; commencing hibernation.

comment:8 Changed 9 years ago by mwenge

OK, the log supplied relates to relay BarkerJrCoast1. Here is the bandwidth info reported by the descriptors for that node in July:

BarkerJrCoast1,2010-07-01,04:08:00,5242880,10485760,1753
BarkerJrCoast1,2010-07-01,04:20:41,5242880,10485760,53320
BarkerJrCoast1,2010-07-01,08:50:07,5242880,10485760,159642
BarkerJrCoast1,2010-07-01,23:35:39,5242880,10485760,369674
BarkerJrCoast1,2010-07-02,01:35:32,5242880,10485760,369674
BarkerJrCoast1,2010-07-02,12:05:57,5242880,10485760,978369
BarkerJrCoast1,2010-07-03,06:06:41,5242880,10485760,978369
BarkerJrCoast1,2010-07-04,00:07:25,5242880,10485760,732043
BarkerJrCoast1,2010-07-04,18:08:11,5242880,10485760,1363632
BarkerJrCoast1,2010-07-05,12:08:54,5242880,10485760,1964809
BarkerJrCoast1,2010-07-06,06:09:37,5242880,10485760,1477788
BarkerJrCoast1,2010-07-07,00:10:21,5242880,10485760,1257557
BarkerJrCoast1,2010-07-07,18:11:04,5242880,10485760,1298376
BarkerJrCoast1,2010-07-08,07:12:53,5242880,10485760,0
BarkerJrCoast1,2010-07-09,01:13:37,5242880,10485760,0
BarkerJrCoast1,2010-07-09,01:35:33,5242880,10485760,0
BarkerJrCoast1,2010-07-09,19:35:44,5242880,10485760,0
BarkerJrCoast1,2010-07-10,13:36:31,5242880,10485760,0
BarkerJrCoast1,2010-07-11,07:37:16,5242880,10485760,0
BarkerJrCoast1,2010-07-12,01:38:00,5242880,10485760,0
BarkerJrCoast1,2010-07-12,19:38:54,5242880,10485760,0
BarkerJrCoast1,2010-07-13,13:39:43,5242880,10485760,0
BarkerJrCoast1,2010-07-14,07:40:30,5242880,10485760,0

comment:9 Changed 9 years ago by mwenge

More spam-stats. This time for all the reporter's relays. A bunch of them stopped reporting descriptors around the 14th of July.

BarkerJrCoast1,2010-07-01,04:08:00,5242880,10485760,1753
BarkerJrCoast1,2010-07-01,04:20:41,5242880,10485760,53320
BarkerJrCoast1,2010-07-01,08:50:07,5242880,10485760,159642
BarkerJrCoast1,2010-07-01,23:35:39,5242880,10485760,369674
BarkerJrCoast1,2010-07-02,01:35:32,5242880,10485760,369674
BarkerJrCoast1,2010-07-02,12:05:57,5242880,10485760,978369
BarkerJrCoast1,2010-07-03,06:06:41,5242880,10485760,978369
BarkerJrCoast1,2010-07-04,00:07:25,5242880,10485760,732043
BarkerJrCoast1,2010-07-04,18:08:11,5242880,10485760,1363632
BarkerJrCoast1,2010-07-05,12:08:54,5242880,10485760,1964809
BarkerJrCoast1,2010-07-06,06:09:37,5242880,10485760,1477788
BarkerJrCoast1,2010-07-07,00:10:21,5242880,10485760,1257557
BarkerJrCoast1,2010-07-07,18:11:04,5242880,10485760,1298376
BarkerJrCoast1,2010-07-08,07:12:53,5242880,10485760,0
BarkerJrCoast1,2010-07-09,01:13:37,5242880,10485760,0
BarkerJrCoast1,2010-07-09,01:35:33,5242880,10485760,0
BarkerJrCoast1,2010-07-09,19:35:44,5242880,10485760,0
BarkerJrCoast1,2010-07-10,13:36:31,5242880,10485760,0
BarkerJrCoast1,2010-07-11,07:37:16,5242880,10485760,0
BarkerJrCoast1,2010-07-12,01:38:00,5242880,10485760,0
BarkerJrCoast1,2010-07-12,19:38:54,5242880,10485760,0
BarkerJrCoast1,2010-07-13,13:39:43,5242880,10485760,0
BarkerJrCoast1,2010-07-14,07:40:30,5242880,10485760,0
BarkerJrCoast2,2010-07-01,08:49:05,5242880,10485760,1684
BarkerJrCoast2,2010-07-01,09:04:35,5242880,10485760,55937
BarkerJrCoast2,2010-07-01,12:03:32,5242880,10485760,322958
BarkerJrCoast2,2010-07-01,19:18:41,5242880,10485760,737753
BarkerJrCoast2,2010-07-02,13:19:25,5242880,10485760,871499
BarkerJrCoast2,2010-07-03,07:20:10,5242880,10485760,629035
BarkerJrCoast2,2010-07-04,01:20:54,5242880,10485760,694318
BarkerJrCoast2,2010-07-04,04:53:55,5242880,10485760,694318
BarkerJrCoast2,2010-07-04,07:48:16,5242880,10485760,1437453
BarkerJrCoast2,2010-07-05,01:49:00,5242880,10485760,1703634
BarkerJrCoast2,2010-07-05,19:49:45,5242880,10485760,1898005
BarkerJrCoast2,2010-07-06,13:50:30,5242880,10485760,1078605
BarkerJrCoast2,2010-07-07,07:51:13,5242880,10485760,1117988
BarkerJrCoast2,2010-07-08,01:51:56,5242880,10485760,1483263
BarkerJrCoast2,2010-07-08,04:25:27,5242880,10485760,0
BarkerJrCoast2,2010-07-08,22:26:12,5242880,10485760,0
BarkerJrCoast2,2010-07-09,16:26:58,5242880,10485760,0
BarkerJrCoast2,2010-07-10,10:27:44,5242880,10485760,0
BarkerJrCoast2,2010-07-11,04:28:27,5242880,10485760,0
BarkerJrCoast2,2010-07-11,04:53:56,5242880,10485760,0
BarkerJrCoast2,2010-07-11,22:54:37,5242880,10485760,0
BarkerJrCoast2,2010-07-12,16:55:22,5242880,10485760,0
BarkerJrCoast2,2010-07-13,10:56:07,5242880,10485760,0
BarkerJrCoast2,2010-07-14,04:56:55,5242880,10485760,0
BarkerJrCoast2,2010-07-14,22:57:40,5242880,10485760,0
BarkerJrGarden1,2010-07-01,00:05:18,5242880,10485760,1676
BarkerJrGarden1,2010-07-01,00:17:22,5242880,10485760,53642
BarkerJrGarden1,2010-07-01,10:01:57,5242880,10485760,147320
BarkerJrGarden1,2010-07-01,14:31:22,5242880,10485760,326889
BarkerJrGarden1,2010-07-02,08:32:06,5242880,10485760,485508
BarkerJrGarden1,2010-07-03,02:32:51,5242880,10485760,837670
BarkerJrGarden1,2010-07-03,02:46:29,5242880,10485760,837670
BarkerJrGarden1,2010-07-03,20:46:47,5242880,10485760,1188813
BarkerJrGarden1,2010-07-04,14:40:24,5242880,10485760,0
BarkerJrGarden1,2010-07-05,08:41:10,5242880,10485760,0
BarkerJrGarden1,2010-07-06,02:41:55,5242880,10485760,0
BarkerJrGarden1,2010-07-06,20:41:57,5242880,10485760,0
BarkerJrGarden1,2010-07-07,14:42:00,5242880,10485760,0
BarkerJrGarden1,2010-07-08,08:42:46,5242880,10485760,0
BarkerJrGarden1,2010-07-09,02:42:47,5242880,10485760,0
BarkerJrGarden1,2010-07-09,20:42:51,5242880,10485760,0
BarkerJrGarden1,2010-07-10,02:46:31,5242880,10485760,0
BarkerJrGarden1,2010-07-10,20:46:35,5242880,10485760,0
BarkerJrGarden1,2010-07-11,14:47:23,5242880,10485760,0
BarkerJrGarden1,2010-07-12,08:48:10,5242880,10485760,0
BarkerJrGarden1,2010-07-13,02:48:55,5242880,10485760,0
BarkerJrGarden1,2010-07-13,20:49:43,5242880,10485760,0
BarkerJrGarden1,2010-07-14,14:50:39,5242880,10485760,0
BarkerJrGarden2,2010-07-01,09:53:39,5242880,10485760,1606
BarkerJrGarden2,2010-07-01,10:03:26,5242880,10485760,50114
BarkerJrGarden2,2010-07-01,11:48:09,5242880,10485760,129714
BarkerJrGarden2,2010-07-01,20:31:45,5242880,10485760,836293
BarkerJrGarden2,2010-07-02,14:32:28,5242880,10485760,949128
BarkerJrGarden2,2010-07-03,02:46:30,5242880,10485760,949128
BarkerJrGarden2,2010-07-03,20:47:13,5242880,10485760,1005137
BarkerJrGarden2,2010-07-04,09:15:29,5242880,10485760,0
BarkerJrGarden2,2010-07-05,03:16:15,5242880,10485760,0
BarkerJrGarden2,2010-07-05,21:17:01,5242880,10485760,0
BarkerJrGarden2,2010-07-06,15:17:47,5242880,10485760,0
BarkerJrGarden2,2010-07-07,09:18:21,5242880,10485760,0
BarkerJrGarden2,2010-07-08,03:19:14,5242880,10485760,0
BarkerJrGarden2,2010-07-08,21:20:06,5242880,10485760,0
BarkerJrGarden2,2010-07-09,15:20:11,5242880,10485760,0
BarkerJrGarden2,2010-07-10,02:46:31,5242880,10485760,0
BarkerJrGarden2,2010-07-10,20:47:16,5242880,10485760,0
BarkerJrGarden2,2010-07-11,14:48:03,5242880,10485760,0
BarkerJrGarden2,2010-07-12,08:48:52,5242880,10485760,0
BarkerJrGarden2,2010-07-13,02:48:56,5242880,10485760,0
BarkerJrGarden2,2010-07-13,20:49:49,5242880,10485760,0
BarkerJrGarden2,2010-07-14,14:50:34,5242880,10485760,0
BarkerJrNet,2010-07-01,00:29:32,51200,655360,111623
BarkerJrNet,2010-07-01,18:30:18,51200,655360,111623
BarkerJrNet,2010-07-02,00:24:18,51200,655360,111623
BarkerJrNet,2010-07-02,18:24:56,51200,655360,111629
BarkerJrNet,2010-07-03,12:25:46,51200,655360,111630
BarkerJrNet,2010-07-04,06:26:32,51200,655360,111623
BarkerJrNet,2010-07-05,00:27:18,51200,655360,111716
BarkerJrNet,2010-07-05,18:28:03,51200,655360,111631
BarkerJrNet,2010-07-06,12:28:50,51200,655360,111630
BarkerJrNet,2010-07-07,06:29:39,51200,655360,111623
BarkerJrNet,2010-07-08,00:30:27,51200,655360,111628
BarkerJrNet,2010-07-08,18:31:13,51200,655360,111629
BarkerJrNet,2010-07-09,00:24:19,51200,655360,111629
BarkerJrNet,2010-07-09,18:24:46,51200,655360,111722
BarkerJrNet,2010-07-10,12:25:33,51200,655360,111629
BarkerJrNet,2010-07-11,06:26:21,51200,655360,111631
BarkerJrNet,2010-07-12,00:27:11,51200,655360,111628
BarkerJrNet,2010-07-12,18:27:56,51200,655360,111623
BarkerJrNet,2010-07-13,12:28:42,51200,655360,111631
BarkerJrNet,2010-07-14,06:29:30,51200,655360,111631
BarkerJrNet,2010-07-15,00:30:15,51200,655360,111631
BarkerJrNet,2010-07-15,18:31:03,51200,655360,111630
BarkerJrNet,2010-07-16,00:24:20,51200,655360,111630
BarkerJrNet,2010-07-16,18:24:39,51200,655360,111629
BarkerJrNet,2010-07-17,12:25:23,51200,655360,111630
BarkerJrNet,2010-07-18,06:26:14,51200,655360,111721
BarkerJrNet,2010-07-19,00:27:00,51200,655360,111721
BarkerJrNet,2010-07-19,18:27:46,51200,655360,111799
BarkerJrNet,2010-07-20,12:28:31,51200,655360,111799
BarkerJrNet,2010-07-21,06:29:17,51200,655360,111630
BarkerJrNet,2010-07-22,00:30:08,51200,655360,111623
BarkerJrNet,2010-07-22,18:30:53,51200,655360,111630
BarkerJrNet,2010-07-23,00:24:21,51200,655360,111630
BarkerJrNet,2010-07-23,18:24:28,51200,655360,111631
BarkerJrNet,2010-07-24,12:25:18,51200,655360,111631
BarkerJrNet,2010-07-25,06:26:06,51200,655360,111629
BarkerJrNet,2010-07-26,00:26:52,51200,655360,111629
BarkerJrNet,2010-07-26,18:27:39,51200,655360,111720
BarkerJrNet,2010-07-27,12:28:24,51200,655360,111720
BarkerJrNet,2010-07-28,06:29:13,51200,655360,111722
BarkerJrNet,2010-07-29,00:30:01,51200,655360,111722
BarkerJrNet,2010-07-29,18:30:48,51200,655360,111631
BarkerJrNet,2010-07-30,00:24:22,51200,655360,111631
BarkerJrNet,2010-07-30,18:25:20,51200,655360,111630
BarkerJrNet,2010-07-31,12:26:04,51200,655360,111630
BarkerJrParis1a,2010-07-01,01:59:31,1703936,1966080,1900595
BarkerJrParis1a,2010-07-01,19:59:41,1703936,1966080,1890383
BarkerJrParis1a,2010-07-02,14:00:27,1703936,1966080,1890383
BarkerJrParis1a,2010-07-03,08:01:19,1703936,1966080,1890509
BarkerJrParis1a,2010-07-04,02:02:10,1703936,1966080,1883393
BarkerJrParis1a,2010-07-04,20:02:56,1703936,1966080,1888140
BarkerJrParis1a,2010-07-05,14:03:50,1703936,1966080,1889036
BarkerJrParis1a,2010-07-06,08:04:39,1703936,1966080,1885880
BarkerJrParis1a,2010-07-07,02:05:23,1703936,1966080,1893458
BarkerJrParis1a,2010-07-07,20:06:07,1703936,1966080,1893458
BarkerJrParis1a,2010-07-08,01:59:33,1703936,1966080,1883476
BarkerJrParis1a,2010-07-08,19:59:40,1703936,1966080,1892170
BarkerJrParis1a,2010-07-09,14:00:26,1703936,1966080,1892758
BarkerJrParis1a,2010-07-10,08:01:12,1703936,1966080,1896473
BarkerJrParis1a,2010-07-11,02:01:56,1703936,1966080,1894911
BarkerJrParis1a,2010-07-11,20:02:42,1703936,1966080,1893457
BarkerJrParis1a,2010-07-12,14:03:28,1703936,1966080,1894618
BarkerJrParis1a,2010-07-13,08:04:15,1703936,1966080,1886802
BarkerJrParis1a,2010-07-14,02:05:08,1703936,1966080,1897460
BarkerJrParis1a,2010-07-14,20:05:53,1703936,1966080,1894842
BarkerJrParis1a,2010-07-15,01:59:34,1703936,1966080,1891282
BarkerJrParis1a,2010-07-15,19:59:38,1703936,1966080,1890611
BarkerJrParis1a,2010-07-16,14:00:23,1703936,1966080,1895499
BarkerJrParis1a,2010-07-17,08:01:15,1703936,1966080,1887688
BarkerJrParis1a,2010-07-18,02:02:03,1703936,1966080,1890495
BarkerJrParis1a,2010-07-18,20:02:48,1703936,1966080,1889033
BarkerJrParis1a,2010-07-19,14:03:36,1703936,1966080,1888203
BarkerJrParis1a,2010-07-20,08:04:24,1703936,1966080,1881617
BarkerJrParis1a,2010-07-21,02:05:11,1703936,1966080,1888809
BarkerJrParis1a,2010-07-21,20:05:58,1703936,1966080,1893108
BarkerJrParis1a,2010-07-22,01:59:35,1703936,1966080,1892968
BarkerJrParis1a,2010-07-22,20:00:36,1703936,1966080,1896188
BarkerJrParis1a,2010-07-23,14:01:21,1703936,1966080,1890728
BarkerJrParis1a,2010-07-24,08:02:09,1703936,1966080,1885575
BarkerJrParis1a,2010-07-25,02:02:54,1703936,1966080,1880891
BarkerJrParis1a,2010-07-25,20:03:38,1703936,1966080,1885038
BarkerJrParis1a,2010-07-26,14:04:37,1703936,1966080,1896973
BarkerJrParis1a,2010-07-27,08:05:25,1703936,1966080,1891933
BarkerJrParis1a,2010-07-28,02:06:14,1703936,1966080,1880998
BarkerJrParis1a,2010-07-28,20:07:01,1703936,1966080,1892385
BarkerJrParis1a,2010-07-29,01:59:36,1703936,1966080,1894042
BarkerJrParis1a,2010-07-29,20:00:33,1703936,1966080,1893639
BarkerJrParis1a,2010-07-30,14:01:16,1703936,1966080,1884204
BarkerJrParis1a,2010-07-31,08:02:01,1703936,1966080,1889781

comment:10 Changed 9 years ago by mwenge

BarkerJrCoast1 and BarkerJrCoast2 look like instances on the same machine. They both hibernated at roughly the same time. The same for BarkerJrGarden1 and 2. That seems odd to me, though maybe they're just encountering the same bug due to similar configuration.

I really find it hard to believe that the AccountingMax on any of these four machines was 200GB. Next step is to take a look at the extra-infos to see the bandwidth reported there.

comment:11 Changed 9 years ago by mwenge

I did a quick scan of July's descriptors to try and find anyone else with behaviour that resembled this problem. Here is the list of 65 nodes in July that hibernated for at least 1/3rd of the month:

http://roberthogan.net/stuff/hibernatingrouters-july.txt

There are a couple of funnies there, but torentrybergerm and giga seem to exhibit the same pattern as the routers reported here. (note there are a few false positives in the list as I matched on routername when pulling the detailed list.)

comment:12 Changed 9 years ago by mwenge

This is the bandwidth on the router according to the extra-info. You can see it drop off as soon as it reaches the bw soft limit. I'm actually surprised there's so much of it after the soft limit was reached.

TIME Router Write Read

(KB) (KB)

2010-07-01T00:05 BarkerJrCoast1-74491A 0 0
2010-07-01T00:20 BarkerJrCoast1-74491A 0 1
2010-07-01T00:35 BarkerJrCoast1-74491A 17 312
2010-07-01T00:50 BarkerJrCoast1-74491A 0 0
2010-07-01T01:05 BarkerJrCoast1-74491A 0 1
2010-07-01T01:20 BarkerJrCoast1-74491A 0 1
2010-07-01T01:35 BarkerJrCoast1-74491A 0 0
2010-07-01T01:50 BarkerJrCoast1-74491A 0 0
2010-07-01T02:05 BarkerJrCoast1-74491A 5 207
2010-07-01T02:20 BarkerJrCoast1-74491A 0 0
2010-07-01T02:35 BarkerJrCoast1-74491A 0 2
2010-07-01T02:50 BarkerJrCoast1-74491A 8 212
2010-07-01T03:05 BarkerJrCoast1-74491A 0 1
2010-07-01T03:20 BarkerJrCoast1-74491A 0 0
2010-07-01T03:35 BarkerJrCoast1-74491A 0 121
2010-07-01T03:50 BarkerJrCoast1-74491A 5 84
2010-07-01T04:05 BarkerJrCoast1-74491A 624 601
2010-07-01T04:20 BarkerJrCoast1-74491A 48 22
2010-07-01T04:35 BarkerJrCoast1-74491A 30 22
2010-07-01T04:50 BarkerJrCoast1-74491A 26 19
2010-07-01T05:05 BarkerJrCoast1-74491A 30 142
2010-07-01T05:20 BarkerJrCoast1-74491A 38 31
2010-07-01T05:35 BarkerJrCoast1-74491A 1099 1075
2010-07-01T05:50 BarkerJrCoast1-74491A 125 118
2010-07-01T06:05 BarkerJrCoast1-74491A 9360 9286
2010-07-01T06:20 BarkerJrCoast1-74491A 675 663
2010-07-01T06:35 BarkerJrCoast1-74491A 1334 1443
2010-07-01T06:50 BarkerJrCoast1-74491A 752 739
2010-07-01T07:05 BarkerJrCoast1-74491A 2177 2161
2010-07-01T07:20 BarkerJrCoast1-74491A 4356 4343
2010-07-01T07:35 BarkerJrCoast1-74491A 7489 7388
2010-07-01T07:50 BarkerJrCoast1-74491A 22738 22488
2010-07-01T08:05 BarkerJrCoast1-74491A 3637 3590
2010-07-01T08:20 BarkerJrCoast1-74491A 1751 1709
2010-07-01T08:35 BarkerJrCoast1-74491A 5874 5860
2010-07-01T08:50 BarkerJrCoast1-74491A 3717 4044
2010-07-01T09:05 BarkerJrCoast1-74491A 1342 1139
2010-07-01T09:20 BarkerJrCoast1-74491A 1676 1641
2010-07-01T09:35 BarkerJrCoast1-74491A 3004 2946
2010-07-01T09:50 BarkerJrCoast1-74491A 7148 7188
2010-07-01T10:05 BarkerJrCoast1-74491A 5585 5544
2010-07-01T10:20 BarkerJrCoast1-74491A 7954 7893
2010-07-01T10:35 BarkerJrCoast1-74491A 2845 2803
2010-07-01T10:50 BarkerJrCoast1-74491A 8454 8345
2010-07-01T11:05 BarkerJrCoast1-74491A 7472 7376
2010-07-01T11:20 BarkerJrCoast1-74491A 20104 19782
2010-07-01T11:35 BarkerJrCoast1-74491A 9344 9384
2010-07-01T11:50 BarkerJrCoast1-74491A 9297 9214
2010-07-01T12:05 BarkerJrCoast1-74491A 16231 16041
2010-07-01T12:20 BarkerJrCoast1-74491A 28431 28113
2010-07-01T12:35 BarkerJrCoast1-74491A 40115 39780
2010-07-01T12:50 BarkerJrCoast1-74491A 82837 83061
2010-07-01T13:05 BarkerJrCoast1-74491A 13424 13413
2010-07-01T13:20 BarkerJrCoast1-74491A 17535 17368
2010-07-01T13:35 BarkerJrCoast1-74491A 9551 9396
2010-07-01T13:50 BarkerJrCoast1-74491A 15714 15442
2010-07-01T14:05 BarkerJrCoast1-74491A 6548 6577
2010-07-01T14:20 BarkerJrCoast1-74491A 56797 56266
2010-07-01T14:35 BarkerJrCoast1-74491A 77304 76227
2010-07-01T14:50 BarkerJrCoast1-74491A 17843 17777
2010-07-01T15:05 BarkerJrCoast1-74491A 21898 21622
2010-07-01T15:20 BarkerJrCoast1-74491A 24224 23865
2010-07-01T15:35 BarkerJrCoast1-74491A 24241 23852
2010-07-01T15:50 BarkerJrCoast1-74491A 22528 22261
2010-07-01T16:05 BarkerJrCoast1-74491A 36932 36316
2010-07-01T16:20 BarkerJrCoast1-74491A 15291 15154
2010-07-01T16:35 BarkerJrCoast1-74491A 14932 14692
2010-07-01T16:50 BarkerJrCoast1-74491A 30551 30087
2010-07-01T17:05 BarkerJrCoast1-74491A 14956 14660
2010-07-01T17:20 BarkerJrCoast1-74491A 24367 23907
2010-07-01T17:35 BarkerJrCoast1-74491A 19264 18827
2010-07-01T17:50 BarkerJrCoast1-74491A 18986 18581
2010-07-01T18:05 BarkerJrCoast1-74491A 17744 17780
2010-07-01T18:20 BarkerJrCoast1-74491A 42190 41828
2010-07-01T18:35 BarkerJrCoast1-74491A 17676 17827
2010-07-01T18:50 BarkerJrCoast1-74491A 20727 20459
2010-07-01T19:05 BarkerJrCoast1-74491A 35287 34820
2010-07-01T19:20 BarkerJrCoast1-74491A 64084 63110
2010-07-01T19:35 BarkerJrCoast1-74491A 77791 76637
2010-07-01T19:50 BarkerJrCoast1-74491A 65230 64168
2010-07-01T20:05 BarkerJrCoast1-74491A 59523 58488
2010-07-01T20:20 BarkerJrCoast1-74491A 57214 56355
2010-07-01T20:35 BarkerJrCoast1-74491A 53501 52809
2010-07-01T20:50 BarkerJrCoast1-74491A 15528 15472
2010-07-01T21:05 BarkerJrCoast1-74491A 20435 20091
2010-07-01T21:20 BarkerJrCoast1-74491A 51977 51245
2010-07-01T21:35 BarkerJrCoast1-74491A 83765 82428
2010-07-01T21:50 BarkerJrCoast1-74491A 45468 44849
2010-07-01T22:05 BarkerJrCoast1-74491A 42385 41829
2010-07-01T22:20 BarkerJrCoast1-74491A 19149 18868
2010-07-01T22:35 BarkerJrCoast1-74491A 24062 24206
2010-07-01T22:50 BarkerJrCoast1-74491A 41739 40658
2010-07-01T23:05 BarkerJrCoast1-74491A 30805 30743
2010-07-01T23:20 BarkerJrCoast1-74491A 28275 27868
2010-07-01T23:35 BarkerJrCoast1-74491A 27501 27467
2010-07-01T23:50 BarkerJrCoast1-74491A 52175 51675
2010-07-02T00:05 BarkerJrCoast1-74491A 18323 18240
2010-07-02T00:20 BarkerJrCoast1-74491A 13458 13246
2010-07-02T00:35 BarkerJrCoast1-74491A 9873 10037
2010-07-02T00:50 BarkerJrCoast1-74491A 18291 17893
2010-07-02T01:05 BarkerJrCoast1-74491A 33619 33322
2010-07-02T01:20 BarkerJrCoast1-74491A 14726 14568
2010-07-02T01:35 BarkerJrCoast1-74491A 32043 31729
2010-07-02T01:50 BarkerJrCoast1-74491A 51967 52182
2010-07-02T02:05 BarkerJrCoast1-74491A 53447 52481
2010-07-02T02:20 BarkerJrCoast1-74491A 40439 40052
2010-07-02T02:35 BarkerJrCoast1-74491A 35803 35398
2010-07-02T02:50 BarkerJrCoast1-74491A 52400 51630
2010-07-02T03:05 BarkerJrCoast1-74491A 23713 23436
2010-07-02T03:20 BarkerJrCoast1-74491A 22741 22801
2010-07-02T03:35 BarkerJrCoast1-74491A 29430 29071
2010-07-02T03:50 BarkerJrCoast1-74491A 51990 51418
2010-07-02T04:05 BarkerJrCoast1-74491A 52611 51914
2010-07-02T04:20 BarkerJrCoast1-74491A 24405 24258
2010-07-02T04:35 BarkerJrCoast1-74491A 45026 44966
2010-07-02T04:50 BarkerJrCoast1-74491A 53089 53162
2010-07-02T05:05 BarkerJrCoast1-74491A 26325 26008
2010-07-02T05:20 BarkerJrCoast1-74491A 21990 21642
2010-07-02T05:35 BarkerJrCoast1-74491A 23460 23120
2010-07-02T05:50 BarkerJrCoast1-74491A 52511 52226
2010-07-02T06:05 BarkerJrCoast1-74491A 40435 40110
2010-07-02T06:20 BarkerJrCoast1-74491A 48404 47961
2010-07-02T06:35 BarkerJrCoast1-74491A 78870 78015
2010-07-02T06:50 BarkerJrCoast1-74491A 85783 85448
2010-07-02T07:05 BarkerJrCoast1-74491A 36201 36125
2010-07-02T07:20 BarkerJrCoast1-74491A 64862 64190
2010-07-02T07:35 BarkerJrCoast1-74491A 51996 51565
2010-07-02T07:50 BarkerJrCoast1-74491A 39773 39757
2010-07-02T08:05 BarkerJrCoast1-74491A 46691 46639
2010-07-02T08:20 BarkerJrCoast1-74491A 37998 37442
2010-07-02T08:35 BarkerJrCoast1-74491A 26908 26514
2010-07-02T08:50 BarkerJrCoast1-74491A 64888 64110
2010-07-02T09:05 BarkerJrCoast1-74491A 30846 30213
2010-07-02T09:20 BarkerJrCoast1-74491A 73978 72963
2010-07-02T09:35 BarkerJrCoast1-74491A 70443 69685
2010-07-02T09:50 BarkerJrCoast1-74491A 64046 63317
2010-07-02T10:05 BarkerJrCoast1-74491A 76562 75662
2010-07-02T10:20 BarkerJrCoast1-74491A 71460 70477
2010-07-02T10:35 BarkerJrCoast1-74491A 90895 89967
2010-07-02T10:50 BarkerJrCoast1-74491A 42548 42044
2010-07-02T11:05 BarkerJrCoast1-74491A 89555 88567
2010-07-02T11:20 BarkerJrCoast1-74491A 118884 117421
2010-07-02T11:35 BarkerJrCoast1-74491A 80474 79486
2010-07-02T11:50 BarkerJrCoast1-74491A 178446 176567
2010-07-02T12:05 BarkerJrCoast1-74491A 132823 131556
2010-07-02T12:20 BarkerJrCoast1-74491A 85727 84708
2010-07-02T12:35 BarkerJrCoast1-74491A 53484 52987
2010-07-02T12:50 BarkerJrCoast1-74491A 46158 45729
2010-07-02T13:05 BarkerJrCoast1-74491A 59727 59114
2010-07-02T13:20 BarkerJrCoast1-74491A 60665 60403
2010-07-02T13:35 BarkerJrCoast1-74491A 49512 48896
2010-07-02T13:50 BarkerJrCoast1-74491A 97362 96993
2010-07-02T14:05 BarkerJrCoast1-74491A 81925 81183
2010-07-02T14:20 BarkerJrCoast1-74491A 184958 183069
2010-07-02T14:35 BarkerJrCoast1-74491A 150935 149669
2010-07-02T14:50 BarkerJrCoast1-74491A 204911 203239
2010-07-02T15:05 BarkerJrCoast1-74491A 175937 173249
2010-07-02T15:20 BarkerJrCoast1-74491A 139652 137825
2010-07-02T15:35 BarkerJrCoast1-74491A 165535 163593
2010-07-02T15:50 BarkerJrCoast1-74491A 202879 201134
2010-07-02T16:05 BarkerJrCoast1-74491A 203124 201169
2010-07-02T16:20 BarkerJrCoast1-74491A 155977 154220
2010-07-02T16:35 BarkerJrCoast1-74491A 159004 156687
2010-07-02T16:50 BarkerJrCoast1-74491A 184684 182315
2010-07-02T17:05 BarkerJrCoast1-74491A 122203 120632
2010-07-02T17:20 BarkerJrCoast1-74491A 215345 212865
2010-07-02T17:35 BarkerJrCoast1-74491A 223373 220088
2010-07-02T17:50 BarkerJrCoast1-74491A 187588 185550
2010-07-02T18:05 BarkerJrCoast1-74491A 156652 154786
2010-07-02T18:20 BarkerJrCoast1-74491A 216602 214685
2010-07-02T18:35 BarkerJrCoast1-74491A 121661 120002
2010-07-02T18:50 BarkerJrCoast1-74491A 108711 107738
2010-07-02T19:05 BarkerJrCoast1-74491A 129799 128586
2010-07-02T19:20 BarkerJrCoast1-74491A 138453 136516
2010-07-02T19:35 BarkerJrCoast1-74491A 182538 181129
2010-07-02T19:50 BarkerJrCoast1-74491A 340962 336335
2010-07-02T20:05 BarkerJrCoast1-74491A 320000 315510
2010-07-02T20:20 BarkerJrCoast1-74491A 248850 246930
2010-07-02T20:35 BarkerJrCoast1-74491A 125302 123677
2010-07-02T20:50 BarkerJrCoast1-74491A 123100 122670
2010-07-02T21:05 BarkerJrCoast1-74491A 150039 149685
2010-07-02T21:20 BarkerJrCoast1-74491A 237477 235585
2010-07-02T21:35 BarkerJrCoast1-74491A 263373 260320
2010-07-02T21:50 BarkerJrCoast1-74491A 375210 370146
2010-07-02T22:05 BarkerJrCoast1-74491A 274778 271652
2010-07-02T22:20 BarkerJrCoast1-74491A 212325 210434
2010-07-02T22:35 BarkerJrCoast1-74491A 291519 288114
2010-07-02T22:50 BarkerJrCoast1-74491A 212766 210103
2010-07-02T23:05 BarkerJrCoast1-74491A 201272 199293
2010-07-02T23:20 BarkerJrCoast1-74491A 161550 160772
2010-07-02T23:35 BarkerJrCoast1-74491A 194433 192368
2010-07-02T23:50 BarkerJrCoast1-74491A 139961 139054
2010-07-03T00:05 BarkerJrCoast1-74491A 116717 115949
2010-07-03T00:20 BarkerJrCoast1-74491A 135194 133725
2010-07-03T00:35 BarkerJrCoast1-74491A 209930 207431
2010-07-03T00:50 BarkerJrCoast1-74491A 143951 142208
2010-07-03T01:05 BarkerJrCoast1-74491A 168179 166074
2010-07-03T01:20 BarkerJrCoast1-74491A 168920 166707
2010-07-03T01:35 BarkerJrCoast1-74491A 145603 144261
2010-07-03T01:50 BarkerJrCoast1-74491A 115600 114472
2010-07-03T02:05 BarkerJrCoast1-74491A 198855 196273
2010-07-03T02:20 BarkerJrCoast1-74491A 238208 234929
2010-07-03T02:35 BarkerJrCoast1-74491A 177867 175218
2010-07-03T02:50 BarkerJrCoast1-74491A 188863 186499
2010-07-03T03:05 BarkerJrCoast1-74491A 164229 163317
2010-07-03T03:20 BarkerJrCoast1-74491A 162194 161262
2010-07-03T03:35 BarkerJrCoast1-74491A 181775 180590
2010-07-03T03:50 BarkerJrCoast1-74491A 240895 238735
2010-07-03T04:05 BarkerJrCoast1-74491A 242250 240044
2010-07-03T04:20 BarkerJrCoast1-74491A 274830 272242
2010-07-03T04:35 BarkerJrCoast1-74491A 253170 249959
2010-07-03T04:50 BarkerJrCoast1-74491A 192002 189470
2010-07-03T05:05 BarkerJrCoast1-74491A 186383 184211
2010-07-03T05:20 BarkerJrCoast1-74491A 221533 219002
2010-07-03T05:35 BarkerJrCoast1-74491A 154930 153238
2010-07-03T05:50 BarkerJrCoast1-74491A 179082 177136
2010-07-03T06:05 BarkerJrCoast1-74491A 210232 207475
2010-07-03T06:20 BarkerJrCoast1-74491A 284579 280555
2010-07-03T06:35 BarkerJrCoast1-74491A 168166 166122
2010-07-03T06:50 BarkerJrCoast1-74491A 185290 183340
2010-07-03T07:05 BarkerJrCoast1-74491A 192425 190418
2010-07-03T07:20 BarkerJrCoast1-74491A 235650 232748
2010-07-03T07:35 BarkerJrCoast1-74491A 192288 189932
2010-07-03T07:50 BarkerJrCoast1-74491A 224768 222114
2010-07-03T08:05 BarkerJrCoast1-74491A 260946 259685
2010-07-03T08:20 BarkerJrCoast1-74491A 196817 194767
2010-07-03T08:35 BarkerJrCoast1-74491A 206348 204414
2010-07-03T08:50 BarkerJrCoast1-74491A 165459 164041
2010-07-03T09:05 BarkerJrCoast1-74491A 176278 173856
2010-07-03T09:20 BarkerJrCoast1-74491A 186063 183775
2010-07-03T09:35 BarkerJrCoast1-74491A 206725 204583
2010-07-03T09:50 BarkerJrCoast1-74491A 263495 259851
2010-07-03T10:05 BarkerJrCoast1-74491A 319373 315457
2010-07-03T10:20 BarkerJrCoast1-74491A 329120 324806
2010-07-03T10:35 BarkerJrCoast1-74491A 248402 245244
2010-07-03T10:50 BarkerJrCoast1-74491A 165061 163579
2010-07-03T11:05 BarkerJrCoast1-74491A 180899 178841
2010-07-03T11:20 BarkerJrCoast1-74491A 195417 193456
2010-07-03T11:35 BarkerJrCoast1-74491A 282937 279377
2010-07-03T11:50 BarkerJrCoast1-74491A 352601 349667
2010-07-03T12:05 BarkerJrCoast1-74491A 364836 361809
2010-07-03T12:20 BarkerJrCoast1-74491A 261451 258399
2010-07-03T12:35 BarkerJrCoast1-74491A 266925 263908
2010-07-03T12:50 BarkerJrCoast1-74491A 265766 261662
2010-07-03T13:05 BarkerJrCoast1-74491A 188757 185817
2010-07-03T13:20 BarkerJrCoast1-74491A 243791 240314
2010-07-03T13:35 BarkerJrCoast1-74491A 117614 116034
2010-07-03T13:50 BarkerJrCoast1-74491A 52932 52695
2010-07-03T14:05 BarkerJrCoast1-74491A 143856 142092
2010-07-03T14:20 BarkerJrCoast1-74491A 162182 160120
2010-07-03T14:35 BarkerJrCoast1-74491A 164908 162379
2010-07-03T14:50 BarkerJrCoast1-74491A 142698 140754
2010-07-03T15:05 BarkerJrCoast1-74491A 123167 121903
2010-07-03T15:20 BarkerJrCoast1-74491A 161174 160041
2010-07-03T15:35 BarkerJrCoast1-74491A 166663 165487
2010-07-03T15:50 BarkerJrCoast1-74491A 140112 138192
2010-07-03T16:05 BarkerJrCoast1-74491A 183599 181701
2010-07-03T16:20 BarkerJrCoast1-74491A 257447 255903
2010-07-03T16:35 BarkerJrCoast1-74491A 285276 282771
2010-07-03T16:50 BarkerJrCoast1-74491A 318076 316983
2010-07-03T17:05 BarkerJrCoast1-74491A 189085 187702
2010-07-03T17:20 BarkerJrCoast1-74491A 222455 220275
2010-07-03T17:35 BarkerJrCoast1-74491A 193372 191866
2010-07-03T17:50 BarkerJrCoast1-74491A 200679 199290
2010-07-03T18:05 BarkerJrCoast1-74491A 192368 190510
2010-07-03T18:20 BarkerJrCoast1-74491A 129988 128569
2010-07-03T18:35 BarkerJrCoast1-74491A 156469 154777
2010-07-03T18:50 BarkerJrCoast1-74491A 178097 176289
2010-07-03T19:05 BarkerJrCoast1-74491A 270253 267153
2010-07-03T19:20 BarkerJrCoast1-74491A 305938 302581
2010-07-03T19:35 BarkerJrCoast1-74491A 352717 347021
2010-07-03T19:50 BarkerJrCoast1-74491A 196985 194033
2010-07-03T20:05 BarkerJrCoast1-74491A 146124 144267
2010-07-03T20:20 BarkerJrCoast1-74491A 224633 222019
2010-07-03T20:35 BarkerJrCoast1-74491A 236099 233144
2010-07-03T20:50 BarkerJrCoast1-74491A 135191 133289
2010-07-03T21:05 BarkerJrCoast1-74491A 241720 238956
2010-07-03T21:20 BarkerJrCoast1-74491A 174654 171837
2010-07-03T21:35 BarkerJrCoast1-74491A 254953 251003
2010-07-03T21:50 BarkerJrCoast1-74491A 213792 211015
2010-07-03T22:05 BarkerJrCoast1-74491A 279604 274581
2010-07-03T22:20 BarkerJrCoast1-74491A 314666 310793
2010-07-03T22:35 BarkerJrCoast1-74491A 282074 279013
2010-07-03T22:50 BarkerJrCoast1-74491A 247957 244744
2010-07-03T23:05 BarkerJrCoast1-74491A 125708 123931
2010-07-03T23:20 BarkerJrCoast1-74491A 176455 174067
2010-07-03T23:35 BarkerJrCoast1-74491A 149154 147300
2010-07-03T23:50 BarkerJrCoast1-74491A 164920 162968
2010-07-04T00:05 BarkerJrCoast1-74491A 138491 136981
2010-07-04T00:20 BarkerJrCoast1-74491A 125742 124466
2010-07-04T00:35 BarkerJrCoast1-74491A 181588 180260
2010-07-04T00:50 BarkerJrCoast1-74491A 192362 190324
2010-07-04T01:05 BarkerJrCoast1-74491A 118087 117430
2010-07-04T01:20 BarkerJrCoast1-74491A 119289 117773
2010-07-04T01:35 BarkerJrCoast1-74491A 114354 112356
2010-07-04T01:50 BarkerJrCoast1-74491A 144848 142220
2010-07-04T02:05 BarkerJrCoast1-74491A 195443 192330
2010-07-04T02:20 BarkerJrCoast1-74491A 99668 97883
2010-07-04T02:35 BarkerJrCoast1-74491A 61897 61319
2010-07-04T02:50 BarkerJrCoast1-74491A 135959 134454
2010-07-04T03:05 BarkerJrCoast1-74491A 131177 129180
2010-07-04T03:20 BarkerJrCoast1-74491A 155112 153222
2010-07-04T03:35 BarkerJrCoast1-74491A 270926 267875
2010-07-04T03:50 BarkerJrCoast1-74491A 105938 104767
2010-07-04T04:05 BarkerJrCoast1-74491A 112997 111388
2010-07-04T04:20 BarkerJrCoast1-74491A 190835 188988
2010-07-04T04:35 BarkerJrCoast1-74491A 228064 225151
2010-07-04T04:50 BarkerJrCoast1-74491A 150478 147738
2010-07-04T05:05 BarkerJrCoast1-74491A 200874 197743
2010-07-04T05:20 BarkerJrCoast1-74491A 283138 278475
2010-07-04T05:35 BarkerJrCoast1-74491A 266166 262635
2010-07-04T05:50 BarkerJrCoast1-74491A 224311 220331
2010-07-04T06:05 BarkerJrCoast1-74491A 273210 269714
2010-07-04T06:20 BarkerJrCoast1-74491A 343314 338727
2010-07-04T06:35 BarkerJrCoast1-74491A 346726 342650
2010-07-04T06:50 BarkerJrCoast1-74491A 306979 304624
2010-07-04T07:05 BarkerJrCoast1-74491A 272407 269372
2010-07-04T07:20 BarkerJrCoast1-74491A 280199 276557
2010-07-04T07:35 BarkerJrCoast1-74491A 271189 268511
2010-07-04T07:50 BarkerJrCoast1-74491A 233883 231322
2010-07-04T08:05 BarkerJrCoast1-74491A 258394 256098
2010-07-04T08:20 BarkerJrCoast1-74491A 288318 285377
2010-07-04T08:35 BarkerJrCoast1-74491A 239215 235970
2010-07-04T08:50 BarkerJrCoast1-74491A 323662 321970
2010-07-04T09:05 BarkerJrCoast1-74491A 485129 476558
2010-07-04T09:20 BarkerJrCoast1-74491A 442822 437636
2010-07-04T09:35 BarkerJrCoast1-74491A 328709 324547
2010-07-04T09:50 BarkerJrCoast1-74491A 260216 256706
2010-07-04T10:05 BarkerJrCoast1-74491A 328298 323971
2010-07-04T10:20 BarkerJrCoast1-74491A 301242 297291
2010-07-04T10:35 BarkerJrCoast1-74491A 273615 270112
2010-07-04T10:50 BarkerJrCoast1-74491A 483730 477100
2010-07-04T11:05 BarkerJrCoast1-74491A 566525 559451
2010-07-04T11:20 BarkerJrCoast1-74491A 369939 364758
2010-07-04T11:35 BarkerJrCoast1-74491A 426287 421251
2010-07-04T11:50 BarkerJrCoast1-74491A 372117 367418
2010-07-04T12:05 BarkerJrCoast1-74491A 397401 392110
2010-07-04T12:20 BarkerJrCoast1-74491A 394595 389401
2010-07-04T12:35 BarkerJrCoast1-74491A 345884 340547
2010-07-04T12:50 BarkerJrCoast1-74491A 279901 276276
2010-07-04T13:05 BarkerJrCoast1-74491A 306733 302455
2010-07-04T13:20 BarkerJrCoast1-74491A 303544 300313
2010-07-04T13:35 BarkerJrCoast1-74491A 328377 323380
2010-07-04T13:50 BarkerJrCoast1-74491A 291652 287352
2010-07-04T14:05 BarkerJrCoast1-74491A 195816 193131
2010-07-04T14:20 BarkerJrCoast1-74491A 241594 238329
2010-07-04T14:35 BarkerJrCoast1-74491A 288577 285137
2010-07-04T14:50 BarkerJrCoast1-74491A 339318 334518
2010-07-04T15:05 BarkerJrCoast1-74491A 490157 484242
2010-07-04T15:20 BarkerJrCoast1-74491A 419391 413914
2010-07-04T15:35 BarkerJrCoast1-74491A 408389 403368
2010-07-04T15:50 BarkerJrCoast1-74491A 273548 270857
2010-07-04T16:05 BarkerJrCoast1-74491A 253890 249707
2010-07-04T16:20 BarkerJrCoast1-74491A 352704 348564
2010-07-04T16:35 BarkerJrCoast1-74491A 232406 229341
2010-07-04T16:50 BarkerJrCoast1-74491A 258278 255847
2010-07-04T17:05 BarkerJrCoast1-74491A 244621 241584
2010-07-04T17:20 BarkerJrCoast1-74491A 263953 260149
2010-07-04T17:35 BarkerJrCoast1-74491A 522298 515838
2010-07-04T17:50 BarkerJrCoast1-74491A 268989 265367
2010-07-04T18:05 BarkerJrCoast1-74491A 318071 313797
2010-07-04T18:20 BarkerJrCoast1-74491A 221216 218057
2010-07-04T18:35 BarkerJrCoast1-74491A 214982 212841
2010-07-04T18:50 BarkerJrCoast1-74491A 232561 229216
2010-07-04T19:05 BarkerJrCoast1-74491A 261128 257521
2010-07-04T19:20 BarkerJrCoast1-74491A 304946 300730
2010-07-04T19:35 BarkerJrCoast1-74491A 349407 343919
2010-07-04T19:50 BarkerJrCoast1-74491A 526949 519617
2010-07-04T20:05 BarkerJrCoast1-74491A 408812 403314
2010-07-04T20:20 BarkerJrCoast1-74491A 479579 473813
2010-07-04T20:35 BarkerJrCoast1-74491A 335412 330907
2010-07-04T20:50 BarkerJrCoast1-74491A 524077 517835
2010-07-04T21:05 BarkerJrCoast1-74491A 467253 463117
2010-07-04T21:20 BarkerJrCoast1-74491A 432097 427629
2010-07-04T21:35 BarkerJrCoast1-74491A 778212 769532
2010-07-04T21:50 BarkerJrCoast1-74491A 557370 550366
2010-07-04T22:05 BarkerJrCoast1-74491A 610632 602985
2010-07-04T22:20 BarkerJrCoast1-74491A 875492 865485
2010-07-04T22:35 BarkerJrCoast1-74491A 719988 712165
2010-07-04T22:50 BarkerJrCoast1-74491A 1113510 1100052
2010-07-04T23:05 BarkerJrCoast1-74491A 629936 622766
2010-07-04T23:20 BarkerJrCoast1-74491A 481228 475656
2010-07-04T23:35 BarkerJrCoast1-74491A 519281 513912
2010-07-04T23:50 BarkerJrCoast1-74491A 377618 373585
2010-07-05T00:05 BarkerJrCoast1-74491A 238960 236195
2010-07-05T00:20 BarkerJrCoast1-74491A 270636 267783
2010-07-05T00:35 BarkerJrCoast1-74491A 541677 536339
2010-07-05T00:50 BarkerJrCoast1-74491A 496897 491389
2010-07-05T01:05 BarkerJrCoast1-74491A 383115 378046
2010-07-05T01:20 BarkerJrCoast1-74491A 360327 356575
2010-07-05T01:35 BarkerJrCoast1-74491A 474789 468062
2010-07-05T01:50 BarkerJrCoast1-74491A 412097 405621
2010-07-05T02:05 BarkerJrCoast1-74491A 453523 446863
2010-07-05T02:20 BarkerJrCoast1-74491A 583021 575323
2010-07-05T02:35 BarkerJrCoast1-74491A 664231 657298
2010-07-05T02:50 BarkerJrCoast1-74491A 397756 393531
2010-07-05T03:05 BarkerJrCoast1-74491A 471188 464600
2010-07-05T03:20 BarkerJrCoast1-74491A 606083 597139
2010-07-05T03:35 BarkerJrCoast1-74491A 503920 495169
2010-07-05T03:50 BarkerJrCoast1-74491A 641754 632105
2010-07-05T04:05 BarkerJrCoast1-74491A 461397 455248
2010-07-05T04:20 BarkerJrCoast1-74491A 335084 331236
2010-07-05T04:35 BarkerJrCoast1-74491A 549008 539693
2010-07-05T04:50 BarkerJrCoast1-74491A 609964 601240
2010-07-05T05:05 BarkerJrCoast1-74491A 609472 600959
2010-07-05T05:20 BarkerJrCoast1-74491A 448007 444019
2010-07-05T05:35 BarkerJrCoast1-74491A 420915 414838
2010-07-05T05:50 BarkerJrCoast1-74491A 461162 456142
2010-07-05T06:05 BarkerJrCoast1-74491A 750192 741292
2010-07-05T06:20 BarkerJrCoast1-74491A 547766 539902
2010-07-05T06:35 BarkerJrCoast1-74491A 547832 540047
2010-07-05T06:50 BarkerJrCoast1-74491A 524691 517726
2010-07-05T07:05 BarkerJrCoast1-74491A 578791 571526
2010-07-05T07:20 BarkerJrCoast1-74491A 515710 508932
2010-07-05T07:35 BarkerJrCoast1-74491A 464325 457430
2010-07-05T07:50 BarkerJrCoast1-74491A 596665 588008
2010-07-05T08:05 BarkerJrCoast1-74491A 560400 552715
2010-07-05T08:20 BarkerJrCoast1-74491A 910308 899558
2010-07-05T08:35 BarkerJrCoast1-74491A 712941 703477
2010-07-05T08:50 BarkerJrCoast1-74491A 681455 673321
2010-07-05T09:05 BarkerJrCoast1-74491A 562347 554775
2010-07-05T09:20 BarkerJrCoast1-74491A 525562 519354
2010-07-05T09:35 BarkerJrCoast1-74491A 584594 577021
2010-07-05T09:50 BarkerJrCoast1-74491A 524065 517848
2010-07-05T10:05 BarkerJrCoast1-74491A 568608 561774
2010-07-05T10:20 BarkerJrCoast1-74491A 545645 539021
2010-07-05T10:35 BarkerJrCoast1-74491A 543274 536150
2010-07-05T10:50 BarkerJrCoast1-74491A 473225 468121
2010-07-05T11:05 BarkerJrCoast1-74491A 622038 614413
2010-07-05T11:20 BarkerJrCoast1-74491A 582942 576094
2010-07-05T11:35 BarkerJrCoast1-74491A 680812 672572
2010-07-05T11:50 BarkerJrCoast1-74491A 400490 396318
2010-07-05T12:05 BarkerJrCoast1-74491A 473683 469289
2010-07-05T12:20 BarkerJrCoast1-74491A 454736 447554
2010-07-05T12:35 BarkerJrCoast1-74491A 417774 412087
2010-07-05T12:50 BarkerJrCoast1-74491A 536384 529054
2010-07-05T13:05 BarkerJrCoast1-74491A 642892 634308
2010-07-05T13:20 BarkerJrCoast1-74491A 678859 670559
2010-07-05T13:35 BarkerJrCoast1-74491A 357400 353508
2010-07-05T13:50 BarkerJrCoast1-74491A 485875 480825
2010-07-05T14:05 BarkerJrCoast1-74491A 479984 475736
2010-07-05T14:20 BarkerJrCoast1-74491A 580454 574234
2010-07-05T14:35 BarkerJrCoast1-74491A 602567 595973
2010-07-05T14:50 BarkerJrCoast1-74491A 326223 322414
2010-07-05T15:05 BarkerJrCoast1-74491A 303885 300375
2010-07-05T15:20 BarkerJrCoast1-74491A 341640 337999
2010-07-05T15:35 BarkerJrCoast1-74491A 283406 279558
2010-07-05T15:50 BarkerJrCoast1-74491A 303946 299652
2010-07-05T16:05 BarkerJrCoast1-74491A 293534 289725
2010-07-05T16:20 BarkerJrCoast1-74491A 317857 314496
2010-07-05T16:35 BarkerJrCoast1-74491A 226561 224068
2010-07-05T16:50 BarkerJrCoast1-74491A 337964 333673
2010-07-05T17:05 BarkerJrCoast1-74491A 336197 332167
2010-07-05T17:20 BarkerJrCoast1-74491A 356557 352401
2010-07-05T17:35 BarkerJrCoast1-74491A 378107 373546
2010-07-05T17:50 BarkerJrCoast1-74491A 373905 369829
2010-07-05T18:05 BarkerJrCoast1-74491A 304453 299695
2010-07-05T18:20 BarkerJrCoast1-74491A 302605 297983
2010-07-05T18:35 BarkerJrCoast1-74491A 368373 362433
2010-07-05T18:50 BarkerJrCoast1-74491A 435668 428483
2010-07-05T19:05 BarkerJrCoast1-74491A 385496 380610
2010-07-05T19:20 BarkerJrCoast1-74491A 388986 382892
2010-07-05T19:35 BarkerJrCoast1-74491A 415636 409903
2010-07-05T19:50 BarkerJrCoast1-74491A 459035 453256
2010-07-05T20:05 BarkerJrCoast1-74491A 310285 306509
2010-07-05T20:20 BarkerJrCoast1-74491A 407376 401822
2010-07-05T20:35 BarkerJrCoast1-74491A 555084 546589
2010-07-05T20:50 BarkerJrCoast1-74491A 500739 492669
2010-07-05T21:05 BarkerJrCoast1-74491A 496530 489005
2010-07-05T21:20 BarkerJrCoast1-74491A 611344 601736
2010-07-05T21:35 BarkerJrCoast1-74491A 623012 614516
2010-07-05T21:50 BarkerJrCoast1-74491A 541517 534266
2010-07-05T22:05 BarkerJrCoast1-74491A 525680 517791
2010-07-05T22:20 BarkerJrCoast1-74491A 445186 439405
2010-07-05T22:35 BarkerJrCoast1-74491A 633567 627033
2010-07-05T22:50 BarkerJrCoast1-74491A 648511 641199
2010-07-05T23:05 BarkerJrCoast1-74491A 471128 463529
2010-07-05T23:20 BarkerJrCoast1-74491A 401167 394539
2010-07-05T23:35 BarkerJrCoast1-74491A 349718 344119
2010-07-05T23:50 BarkerJrCoast1-74491A 519266 511431
2010-07-06T00:05 BarkerJrCoast1-74491A 443500 437588
2010-07-06T00:20 BarkerJrCoast1-74491A 433474 427687
2010-07-06T00:35 BarkerJrCoast1-74491A 441347 434667
2010-07-06T00:50 BarkerJrCoast1-74491A 462963 456675
2010-07-06T01:05 BarkerJrCoast1-74491A 394549 388672
2010-07-06T01:20 BarkerJrCoast1-74491A 319863 315116
2010-07-06T01:35 BarkerJrCoast1-74491A 347495 342622
2010-07-06T01:50 BarkerJrCoast1-74491A 251932 248171
2010-07-06T02:05 BarkerJrCoast1-74491A 254869 252395
2010-07-06T02:20 BarkerJrCoast1-74491A 385195 380622
2010-07-06T02:35 BarkerJrCoast1-74491A 346868 344545
2010-07-06T02:50 BarkerJrCoast1-74491A 534922 529831
2010-07-06T03:05 BarkerJrCoast1-74491A 340102 336511
2010-07-06T03:20 BarkerJrCoast1-74491A 357404 351732
2010-07-06T03:35 BarkerJrCoast1-74491A 243019 238725
2010-07-06T03:50 BarkerJrCoast1-74491A 220823 217571
2010-07-06T04:05 BarkerJrCoast1-74491A 331305 328221
2010-07-06T04:20 BarkerJrCoast1-74491A 534550 528750
2010-07-06T04:35 BarkerJrCoast1-74491A 356201 351268
2010-07-06T04:50 BarkerJrCoast1-74491A 388970 384184
2010-07-06T05:05 BarkerJrCoast1-74491A 481898 477170
2010-07-06T05:20 BarkerJrCoast1-74491A 408694 402737
2010-07-06T05:35 BarkerJrCoast1-74491A 342468 338640
2010-07-06T05:50 BarkerJrCoast1-74491A 285261 281639
2010-07-06T06:05 BarkerJrCoast1-74491A 251317 247486
2010-07-06T06:20 BarkerJrCoast1-74491A 212250 209165
2010-07-06T06:35 BarkerJrCoast1-74491A 317023 312908
2010-07-06T06:50 BarkerJrCoast1-74491A 333988 328217
2010-07-06T07:05 BarkerJrCoast1-74491A 289401 285295
2010-07-06T07:20 BarkerJrCoast1-74491A 363349 358952
2010-07-06T07:35 BarkerJrCoast1-74491A 265078 261671
2010-07-06T07:50 BarkerJrCoast1-74491A 325977 322268
2010-07-06T08:05 BarkerJrCoast1-74491A 326931 323846
2010-07-06T08:20 BarkerJrCoast1-74491A 286763 283808
2010-07-06T08:35 BarkerJrCoast1-74491A 305079 301482
2010-07-06T08:50 BarkerJrCoast1-74491A 382169 376743
2010-07-06T09:05 BarkerJrCoast1-74491A 318455 314275
2010-07-06T09:20 BarkerJrCoast1-74491A 314289 309616
2010-07-06T09:35 BarkerJrCoast1-74491A 350603 346850
2010-07-06T09:50 BarkerJrCoast1-74491A 269539 266694
2010-07-06T10:05 BarkerJrCoast1-74491A 277158 273184
2010-07-06T10:20 BarkerJrCoast1-74491A 345082 340592
2010-07-06T10:35 BarkerJrCoast1-74491A 381206 377523
2010-07-06T10:50 BarkerJrCoast1-74491A 294866 291240
2010-07-06T11:05 BarkerJrCoast1-74491A 205641 203457
2010-07-06T11:20 BarkerJrCoast1-74491A 452228 447388
2010-07-06T11:35 BarkerJrCoast1-74491A 360384 355978
2010-07-06T11:50 BarkerJrCoast1-74491A 243445 240886
2010-07-06T12:05 BarkerJrCoast1-74491A 302895 299581
2010-07-06T12:20 BarkerJrCoast1-74491A 330684 326374
2010-07-06T12:35 BarkerJrCoast1-74491A 476614 471280
2010-07-06T12:50 BarkerJrCoast1-74491A 311397 307431
2010-07-06T13:05 BarkerJrCoast1-74491A 240493 237464
2010-07-06T13:20 BarkerJrCoast1-74491A 422082 416934
2010-07-06T13:35 BarkerJrCoast1-74491A 397749 393330
2010-07-06T13:50 BarkerJrCoast1-74491A 371617 366600
2010-07-06T14:05 BarkerJrCoast1-74491A 369456 364365
2010-07-06T14:20 BarkerJrCoast1-74491A 414602 409666
2010-07-06T14:35 BarkerJrCoast1-74491A 347715 343162
2010-07-06T14:50 BarkerJrCoast1-74491A 358735 354610
2010-07-06T15:05 BarkerJrCoast1-74491A 342626 338416
2010-07-06T15:20 BarkerJrCoast1-74491A 375650 370834
2010-07-06T15:35 BarkerJrCoast1-74491A 448997 443648
2010-07-06T15:50 BarkerJrCoast1-74491A 519767 511828
2010-07-06T16:05 BarkerJrCoast1-74491A 533911 526677
2010-07-06T16:20 BarkerJrCoast1-74491A 348330 343921
2010-07-06T16:35 BarkerJrCoast1-74491A 380457 376163
2010-07-06T16:50 BarkerJrCoast1-74491A 460512 454455
2010-07-06T17:05 BarkerJrCoast1-74491A 454117 448193
2010-07-06T17:20 BarkerJrCoast1-74491A 443140 438482
2010-07-06T17:35 BarkerJrCoast1-74491A 498515 493628
2010-07-06T17:50 BarkerJrCoast1-74491A 637316 630412
2010-07-06T18:05 BarkerJrCoast1-74491A 540751 535126
2010-07-06T18:20 BarkerJrCoast1-74491A 349528 345714
2010-07-06T18:35 BarkerJrCoast1-74491A 319780 316383
2010-07-06T18:50 BarkerJrCoast1-74491A 292882 289088
2010-07-06T19:05 BarkerJrCoast1-74491A 262545 259045
2010-07-06T19:20 BarkerJrCoast1-74491A 318570 313989
2010-07-06T19:35 BarkerJrCoast1-74491A 470635 464786
2010-07-06T19:50 BarkerJrCoast1-74491A 383432 378109
2010-07-06T20:05 BarkerJrCoast1-74491A 391568 385685
2010-07-06T20:20 BarkerJrCoast1-74491A 456672 449900
2010-07-06T20:35 BarkerJrCoast1-74491A 500904 493856
2010-07-06T20:50 BarkerJrCoast1-74491A 553766 546272
2010-07-06T21:05 BarkerJrCoast1-74491A 539660 531953
2010-07-06T21:20 BarkerJrCoast1-74491A 414019 407603
2010-07-06T21:35 BarkerJrCoast1-74491A 363194 357316
2010-07-06T21:50 BarkerJrCoast1-74491A 377256 371404
2010-07-06T22:05 BarkerJrCoast1-74491A 375285 369492
2010-07-06T22:20 BarkerJrCoast1-74491A 477082 469302
2010-07-06T22:35 BarkerJrCoast1-74491A 508265 501982
2010-07-06T22:50 BarkerJrCoast1-74491A 417744 412262
2010-07-06T23:05 BarkerJrCoast1-74491A 357731 351910
2010-07-06T23:20 BarkerJrCoast1-74491A 422352 415761
2010-07-06T23:35 BarkerJrCoast1-74491A 338724 333446
2010-07-06T23:50 BarkerJrCoast1-74491A 417096 411610
2010-07-07T00:05 BarkerJrCoast1-74491A 461556 456106
2010-07-07T00:20 BarkerJrCoast1-74491A 348271 344359
2010-07-07T00:35 BarkerJrCoast1-74491A 457607 452199
2010-07-07T00:50 BarkerJrCoast1-74491A 394672 389307
2010-07-07T01:05 BarkerJrCoast1-74491A 331485 325932
2010-07-07T01:20 BarkerJrCoast1-74491A 285007 281481
2010-07-07T01:35 BarkerJrCoast1-74491A 330871 326015
2010-07-07T01:50 BarkerJrCoast1-74491A 536050 528594
2010-07-07T02:05 BarkerJrCoast1-74491A 384036 378193
2010-07-07T02:20 BarkerJrCoast1-74491A 316500 311632
2010-07-07T02:35 BarkerJrCoast1-74491A 297044 292734
2010-07-07T02:50 BarkerJrCoast1-74491A 355374 350223
2010-07-07T03:05 BarkerJrCoast1-74491A 300304 296411
2010-07-07T03:20 BarkerJrCoast1-74491A 315542 310685
2010-07-07T03:35 BarkerJrCoast1-74491A 365704 361072
2010-07-07T03:50 BarkerJrCoast1-74491A 477929 472429
2010-07-07T04:05 BarkerJrCoast1-74491A 447344 440477
2010-07-07T04:20 BarkerJrCoast1-74491A 448364 441318
2010-07-07T04:35 BarkerJrCoast1-74491A 340778 335106
2010-07-07T04:50 BarkerJrCoast1-74491A 362418 355850
2010-07-07T05:05 BarkerJrCoast1-74491A 379028 372306
2010-07-07T05:20 BarkerJrCoast1-74491A 327985 322667
2010-07-07T05:35 BarkerJrCoast1-74491A 403071 395782
2010-07-07T05:50 BarkerJrCoast1-74491A 492487 485693
2010-07-07T06:05 BarkerJrCoast1-74491A 633610 625149
2010-07-07T06:20 BarkerJrCoast1-74491A 423659 416862
2010-07-07T06:35 BarkerJrCoast1-74491A 355851 349884
2010-07-07T06:50 BarkerJrCoast1-74491A 564160 555946
2010-07-07T07:05 BarkerJrCoast1-74491A 453880 447402
2010-07-07T07:20 BarkerJrCoast1-74491A 597997 589654
2010-07-07T07:35 BarkerJrCoast1-74491A 297916 293839
2010-07-07T07:50 BarkerJrCoast1-74491A 262913 259230
2010-07-07T08:05 BarkerJrCoast1-74491A 213298 209973
2010-07-07T08:20 BarkerJrCoast1-74491A 277706 273531
2010-07-07T08:35 BarkerJrCoast1-74491A 264384 260786
2010-07-07T08:50 BarkerJrCoast1-74491A 260112 255523
2010-07-07T09:05 BarkerJrCoast1-74491A 400898 395886
2010-07-07T09:20 BarkerJrCoast1-74491A 370849 365467
2010-07-07T09:35 BarkerJrCoast1-74491A 370661 365056
2010-07-07T09:50 BarkerJrCoast1-74491A 374809 369455
2010-07-07T10:05 BarkerJrCoast1-74491A 569013 561706
2010-07-07T10:20 BarkerJrCoast1-74491A 392265 386772
2010-07-07T10:35 BarkerJrCoast1-74491A 372474 367481
2010-07-07T10:50 BarkerJrCoast1-74491A 405555 400227
2010-07-07T11:05 BarkerJrCoast1-74491A 407856 402436
2010-07-07T11:20 BarkerJrCoast1-74491A 374944 370854
2010-07-07T11:35 BarkerJrCoast1-74491A 429516 424930
2010-07-07T11:50 BarkerJrCoast1-74491A 434146 429646
2010-07-07T12:05 BarkerJrCoast1-74491A 335968 332217
2010-07-07T12:20 BarkerJrCoast1-74491A 238721 235583
2010-07-07T12:35 BarkerJrCoast1-74491A 259437 255657
2010-07-07T12:50 BarkerJrCoast1-74491A 382105 377474
2010-07-07T13:05 BarkerJrCoast1-74491A 538308 530657
2010-07-07T13:20 BarkerJrCoast1-74491A 475800 468267
2010-07-07T13:35 BarkerJrCoast1-74491A 341030 336250
2010-07-07T13:50 BarkerJrCoast1-74491A 348830 342350
2010-07-07T14:05 BarkerJrCoast1-74491A 477169 468087
2010-07-07T14:20 BarkerJrCoast1-74491A 385918 379159
2010-07-07T14:35 BarkerJrCoast1-74491A 474429 466780
2010-07-07T14:50 BarkerJrCoast1-74491A 531745 523957
2010-07-07T15:05 BarkerJrCoast1-74491A 494638 488303
2010-07-07T15:20 BarkerJrCoast1-74491A 461315 455024
2010-07-07T15:35 BarkerJrCoast1-74491A 501140 494087
2010-07-07T15:50 BarkerJrCoast1-74491A 404428 398832
2010-07-07T16:05 BarkerJrCoast1-74491A 467990 461786
2010-07-07T16:20 BarkerJrCoast1-74491A 477365 471515
2010-07-07T16:35 BarkerJrCoast1-74491A 477978 471879
2010-07-07T16:50 BarkerJrCoast1-74491A 471139 464737
2010-07-07T17:05 BarkerJrCoast1-74491A 453035 447146
2010-07-07T17:20 BarkerJrCoast1-74491A 406315 400838
2010-07-07T17:35 BarkerJrCoast1-74491A 469753 464573
2010-07-07T17:50 BarkerJrCoast1-74491A 559922 554102
2010-07-07T18:05 BarkerJrCoast1-74491A 519109 512057
2010-07-07T18:20 BarkerJrCoast1-74491A 426155 420094
2010-07-07T18:35 BarkerJrCoast1-74491A 590590 581650
2010-07-07T18:50 BarkerJrCoast1-74491A 726830 716713
2010-07-07T19:05 BarkerJrCoast1-74491A 646492 638605
2010-07-07T19:20 BarkerJrCoast1-74491A 593081 585343
2010-07-07T19:35 BarkerJrCoast1-74491A 468170 461009
2010-07-07T19:50 BarkerJrCoast1-74491A 324270 320135
2010-07-07T20:05 BarkerJrCoast1-74491A 409970 404959
2010-07-07T20:20 BarkerJrCoast1-74491A 391827 386036
2010-07-07T20:35 BarkerJrCoast1-74491A 453140 446474
2010-07-07T20:50 BarkerJrCoast1-74491A 347502 342714
2010-07-07T21:05 BarkerJrCoast1-74491A 418129 412344
2010-07-07T21:20 BarkerJrCoast1-74491A 427349 421922
2010-07-07T21:35 BarkerJrCoast1-74491A 576016 569448
2010-07-07T21:50 BarkerJrCoast1-74491A 563677 557645
2010-07-07T22:05 BarkerJrCoast1-74491A 712417 702945
2010-07-07T22:20 BarkerJrCoast1-74491A 367398 362544
2010-07-07T22:35 BarkerJrCoast1-74491A 422754 416739
2010-07-07T22:50 BarkerJrCoast1-74491A 435024 428809
2010-07-07T23:05 BarkerJrCoast1-74491A 611896 603257
2010-07-07T23:20 BarkerJrCoast1-74491A 604088 595449
2010-07-07T23:35 BarkerJrCoast1-74491A 444620 440032
2010-07-07T23:50 BarkerJrCoast1-74491A 344520 340352
2010-07-08T00:05 BarkerJrCoast1-74491A 296292 291658
2010-07-08T00:20 BarkerJrCoast1-74491A 340065 334648
2010-07-08T00:35 BarkerJrCoast1-74491A 235209 231573
2010-07-08T00:50 BarkerJrCoast1-74491A 320079 314385
2010-07-08T01:05 BarkerJrCoast1-74491A 440796 433893
2010-07-08T01:20 BarkerJrCoast1-74491A 415972 409907
2010-07-08T01:35 BarkerJrCoast1-74491A 512701 506110
2010-07-08T01:50 BarkerJrCoast1-74491A 468713 460562
2010-07-08T02:05 BarkerJrCoast1-74491A 547002 539088
2010-07-08T02:20 BarkerJrCoast1-74491A 452168 444429
2010-07-08T02:35 BarkerJrCoast1-74491A 331623 325888
2010-07-08T02:50 BarkerJrCoast1-74491A 367433 361087
2010-07-08T03:05 BarkerJrCoast1-74491A 272073 267434
2010-07-08T03:20 BarkerJrCoast1-74491A 369360 364543
2010-07-08T03:35 BarkerJrCoast1-74491A 507863 501438
2010-07-08T03:50 BarkerJrCoast1-74491A 449042 441616
2010-07-08T04:05 BarkerJrCoast1-74491A 403038 395548
2010-07-08T04:20 BarkerJrCoast1-74491A 404558 398568
2010-07-08T04:35 BarkerJrCoast1-74491A 512844 504130
2010-07-08T04:50 BarkerJrCoast1-74491A 742024 731398
2010-07-08T05:05 BarkerJrCoast1-74491A 572944 564362
2010-07-08T05:20 BarkerJrCoast1-74491A 472313 465324
2010-07-08T05:35 BarkerJrCoast1-74491A 441950 435797
2010-07-08T05:50 BarkerJrCoast1-74491A 322013 317333
2010-07-08T06:05 BarkerJrCoast1-74491A 364690 359418
2010-07-08T06:20 BarkerJrCoast1-74491A 406360 400974
2010-07-08T06:35 BarkerJrCoast1-74491A 441784 436102
2010-07-08T06:50 BarkerJrCoast1-74491A 436200 430295
2010-07-08T07:05 BarkerJrCoast1-74491A 438617 431663
2010-07-08T07:20 BarkerJrCoast1-74491A 207593 204524
2010-07-08T07:35 BarkerJrCoast1-74491A 122304 120593
2010-07-08T07:50 BarkerJrCoast1-74491A 102221 100601
2010-07-08T08:05 BarkerJrCoast1-74491A 96158 94818
2010-07-08T08:20 BarkerJrCoast1-74491A 95688 94192
2010-07-08T08:35 BarkerJrCoast1-74491A 58662 57738
2010-07-08T08:50 BarkerJrCoast1-74491A 44089 43548
2010-07-08T09:05 BarkerJrCoast1-74491A 41959 41421
2010-07-08T09:20 BarkerJrCoast1-74491A 42658 42076
2010-07-08T09:35 BarkerJrCoast1-74491A 43249 42683
2010-07-08T09:50 BarkerJrCoast1-74491A 47422 46884
2010-07-08T10:05 BarkerJrCoast1-74491A 52209 51440
2010-07-08T10:20 BarkerJrCoast1-74491A 50561 49813
2010-07-08T10:35 BarkerJrCoast1-74491A 37432 36979
2010-07-08T10:50 BarkerJrCoast1-74491A 33961 33543
2010-07-08T11:05 BarkerJrCoast1-74491A 37839 37359
2010-07-08T11:20 BarkerJrCoast1-74491A 23176 22887
2010-07-08T11:35 BarkerJrCoast1-74491A 19153 19035
2010-07-08T11:50 BarkerJrCoast1-74491A 16475 16265
2010-07-08T12:05 BarkerJrCoast1-74491A 22386 22109
2010-07-08T12:20 BarkerJrCoast1-74491A 16260 16052
2010-07-08T12:35 BarkerJrCoast1-74491A 18118 17902
2010-07-08T12:50 BarkerJrCoast1-74491A 17793 17580
2010-07-08T13:05 BarkerJrCoast1-74491A 20446 20226
2010-07-08T13:20 BarkerJrCoast1-74491A 21726 21512
2010-07-08T13:35 BarkerJrCoast1-74491A 22895 22774
2010-07-08T13:50 BarkerJrCoast1-74491A 21595 21363
2010-07-08T14:05 BarkerJrCoast1-74491A 20044 19791
2010-07-08T14:20 BarkerJrCoast1-74491A 21068 20833
2010-07-08T14:35 BarkerJrCoast1-74491A 21778 21554
2010-07-08T14:50 BarkerJrCoast1-74491A 20727 20484
2010-07-08T15:05 BarkerJrCoast1-74491A 17817 17613
2010-07-08T15:20 BarkerJrCoast1-74491A 18509 18438
2010-07-08T15:35 BarkerJrCoast1-74491A 18440 18218
2010-07-08T15:50 BarkerJrCoast1-74491A 18088 17850
2010-07-08T16:05 BarkerJrCoast1-74491A 18316 18091
2010-07-08T16:20 BarkerJrCoast1-74491A 18289 18081
2010-07-08T16:35 BarkerJrCoast1-74491A 18773 18586
2010-07-08T16:50 BarkerJrCoast1-74491A 16283 16269
2010-07-08T17:05 BarkerJrCoast1-74491A 17787 17619
2010-07-08T17:20 BarkerJrCoast1-74491A 17174 16993
2010-07-08T17:35 BarkerJrCoast1-74491A 14831 14774
2010-07-08T17:50 BarkerJrCoast1-74491A 14523 14340
2010-07-08T18:05 BarkerJrCoast1-74491A 17335 17111
2010-07-08T18:20 BarkerJrCoast1-74491A 15451 15255
2010-07-08T18:35 BarkerJrCoast1-74491A 16358 16143
2010-07-08T18:50 BarkerJrCoast1-74491A 15913 15715
2010-07-08T19:05 BarkerJrCoast1-74491A 19342 19082
2010-07-08T19:20 BarkerJrCoast1-74491A 17040 16814
2010-07-08T19:35 BarkerJrCoast1-74491A 17400 17453
2010-07-08T19:50 BarkerJrCoast1-74491A 20267 19993
2010-07-08T20:05 BarkerJrCoast1-74491A 17948 17707
2010-07-08T20:20 BarkerJrCoast1-74491A 16884 16676
2010-07-08T20:35 BarkerJrCoast1-74491A 17249 17039
2010-07-08T20:50 BarkerJrCoast1-74491A 16151 15947
2010-07-08T21:05 BarkerJrCoast1-74491A 15762 15577
2010-07-08T21:20 BarkerJrCoast1-74491A 19542 19459
2010-07-08T21:35 BarkerJrCoast1-74491A 22260 22010
2010-07-08T21:50 BarkerJrCoast1-74491A 16390 16335
2010-07-08T22:05 BarkerJrCoast1-74491A 18067 17873
2010-07-08T22:20 BarkerJrCoast1-74491A 16600 16404
2010-07-08T22:35 BarkerJrCoast1-74491A 18255 18056
2010-07-08T22:50 BarkerJrCoast1-74491A 15154 14983
2010-07-08T23:05 BarkerJrCoast1-74491A 15303 15118
2010-07-08T23:20 BarkerJrCoast1-74491A 16868 16797
2010-07-08T23:35 BarkerJrCoast1-74491A 6668 6588
2010-07-08T23:50 BarkerJrCoast1-74491A 491 490
2010-07-09T00:05 BarkerJrCoast1-74491A 299 299
2010-07-09T00:20 BarkerJrCoast1-74491A 329 330
2010-07-09T00:35 BarkerJrCoast1-74491A 353 476
2010-07-09T00:50 BarkerJrCoast1-74491A 334 334
2010-07-09T01:05 BarkerJrCoast1-74491A 350 325
2010-07-09T01:20 BarkerJrCoast1-74491A 319 320
2010-07-09T01:35 BarkerJrCoast1-74491A 309 408
2010-07-09T01:50 BarkerJrCoast1-74491A 331 331
2010-07-09T02:05 BarkerJrCoast1-74491A 319 319
2010-07-09T02:20 BarkerJrCoast1-74491A 310 311
2010-07-09T02:35 BarkerJrCoast1-74491A 308 430
2010-07-09T02:50 BarkerJrCoast1-74491A 300 301
2010-07-09T03:05 BarkerJrCoast1-74491A 315 315
2010-07-09T03:20 BarkerJrCoast1-74491A 293 292
2010-07-09T03:35 BarkerJrCoast1-74491A 347 346
2010-07-09T03:50 BarkerJrCoast1-74491A 417 417
2010-07-09T04:05 BarkerJrCoast1-74491A 417 538
2010-07-09T04:20 BarkerJrCoast1-74491A 362 360
2010-07-09T04:35 BarkerJrCoast1-74491A 321 320
2010-07-09T04:50 BarkerJrCoast1-74491A 208 207
2010-07-09T05:05 BarkerJrCoast1-74491A 230 230
2010-07-09T05:20 BarkerJrCoast1-74491A 220 221
2010-07-09T05:35 BarkerJrCoast1-74491A 201 321
2010-07-09T05:50 BarkerJrCoast1-74491A 207 208
2010-07-09T06:05 BarkerJrCoast1-74491A 197 197
2010-07-09T06:20 BarkerJrCoast1-74491A 190 191
2010-07-09T06:35 BarkerJrCoast1-74491A 207 207
2010-07-09T06:50 BarkerJrCoast1-74491A 219 341
2010-07-09T07:05 BarkerJrCoast1-74491A 213 214
2010-07-09T07:20 BarkerJrCoast1-74491A 210 210
2010-07-09T07:35 BarkerJrCoast1-74491A 216 216
2010-07-09T07:50 BarkerJrCoast1-74491A 218 217
2010-07-09T08:05 BarkerJrCoast1-74491A 213 214
2010-07-09T08:20 BarkerJrCoast1-74491A 284 409
2010-07-09T08:35 BarkerJrCoast1-74491A 236 235
2010-07-09T08:50 BarkerJrCoast1-74491A 216 216
2010-07-09T09:05 BarkerJrCoast1-74491A 237 237
2010-07-09T09:20 BarkerJrCoast1-74491A 266 266
2010-07-09T09:35 BarkerJrCoast1-74491A 245 244
2010-07-09T09:50 BarkerJrCoast1-74491A 249 375
2010-07-09T10:05 BarkerJrCoast1-74491A 206 206
2010-07-09T10:20 BarkerJrCoast1-74491A 224 224
2010-07-09T10:35 BarkerJrCoast1-74491A 257 258
2010-07-09T10:50 BarkerJrCoast1-74491A 325 323
2010-07-09T11:05 BarkerJrCoast1-74491A 271 272
2010-07-09T11:20 BarkerJrCoast1-74491A 304 434
2010-07-09T11:35 BarkerJrCoast1-74491A 294 295
2010-07-09T11:50 BarkerJrCoast1-74491A 254 255
2010-07-09T12:05 BarkerJrCoast1-74491A 247 248
2010-07-09T12:20 BarkerJrCoast1-74491A 275 274
2010-07-09T12:35 BarkerJrCoast1-74491A 284 283
2010-07-09T12:50 BarkerJrCoast1-74491A 264 394
2010-07-09T13:05 BarkerJrCoast1-74491A 319 320
2010-07-09T13:20 BarkerJrCoast1-74491A 319 318
2010-07-09T13:35 BarkerJrCoast1-74491A 294 293
2010-07-09T13:50 BarkerJrCoast1-74491A 281 413
2010-07-09T14:05 BarkerJrCoast1-74491A 323 324
2010-07-09T14:20 BarkerJrCoast1-74491A 327 327
2010-07-09T14:35 BarkerJrCoast1-74491A 252 253
2010-07-09T14:50 BarkerJrCoast1-74491A 262 263
2010-07-09T15:05 BarkerJrCoast1-74491A 254 252
2010-07-09T15:20 BarkerJrCoast1-74491A 199 333
2010-07-09T15:35 BarkerJrCoast1-74491A 220 220
2010-07-09T15:50 BarkerJrCoast1-74491A 212 343
2010-07-09T16:05 BarkerJrCoast1-74491A 225 225
2010-07-09T16:20 BarkerJrCoast1-74491A 243 243
2010-07-09T16:35 BarkerJrCoast1-74491A 211 211
2010-07-09T16:50 BarkerJrCoast1-74491A 208 343
2010-07-09T17:05 BarkerJrCoast1-74491A 211 212
2010-07-09T17:20 BarkerJrCoast1-74491A 193 192
2010-07-09T17:35 BarkerJrCoast1-74491A 198 198
2010-07-09T17:50 BarkerJrCoast1-74491A 204 203
2010-07-09T18:05 BarkerJrCoast1-74491A 188 188
2010-07-09T18:20 BarkerJrCoast1-74491A 199 334
2010-07-09T18:35 BarkerJrCoast1-74491A 182 182
2010-07-09T18:50 BarkerJrCoast1-74491A 192 192
2010-07-09T19:05 BarkerJrCoast1-74491A 190 191
2010-07-09T19:20 BarkerJrCoast1-74491A 211 210
2010-07-09T19:35 BarkerJrCoast1-74491A 214 193
2010-07-09T19:50 BarkerJrCoast1-74491A 197 197
2010-07-09T20:05 BarkerJrCoast1-74491A 190 190
2010-07-09T20:20 BarkerJrCoast1-74491A 200 200
2010-07-09T20:35 BarkerJrCoast1-74491A 237 371
2010-07-09T20:50 BarkerJrCoast1-74491A 228 229
2010-07-09T21:05 BarkerJrCoast1-74491A 198 198
2010-07-09T21:20 BarkerJrCoast1-74491A 188 187
2010-07-09T21:35 BarkerJrCoast1-74491A 183 180
2010-07-09T21:50 BarkerJrCoast1-74491A 203 201
2010-07-09T22:05 BarkerJrCoast1-74491A 170 169
2010-07-09T22:20 BarkerJrCoast1-74491A 183 313
2010-07-09T22:35 BarkerJrCoast1-74491A 206 204
2010-07-09T22:50 BarkerJrCoast1-74491A 174 174
2010-07-09T23:05 BarkerJrCoast1-74491A 179 179
2010-07-09T23:20 BarkerJrCoast1-74491A 195 195
2010-07-09T23:35 BarkerJrCoast1-74491A 195 321
2010-07-09T23:50 BarkerJrCoast1-74491A 169 170
2010-07-10T00:05 BarkerJrCoast1-74491A 193 192
2010-07-10T00:20 BarkerJrCoast1-74491A 201 200
2010-07-10T00:35 BarkerJrCoast1-74491A 179 178
2010-07-10T00:50 BarkerJrCoast1-74491A 192 315
2010-07-10T01:05 BarkerJrCoast1-74491A 204 202
2010-07-10T01:20 BarkerJrCoast1-74491A 218 219
2010-07-10T01:35 BarkerJrCoast1-74491A 289 289
2010-07-10T01:50 BarkerJrCoast1-74491A 251 252
2010-07-10T02:05 BarkerJrCoast1-74491A 210 210
2010-07-10T02:20 BarkerJrCoast1-74491A 195 194
2010-07-10T02:35 BarkerJrCoast1-74491A 223 343
2010-07-10T02:50 BarkerJrCoast1-74491A 194 195
2010-07-10T03:05 BarkerJrCoast1-74491A 187 187
2010-07-10T03:20 BarkerJrCoast1-74491A 235 235
2010-07-10T03:35 BarkerJrCoast1-74491A 253 253
2010-07-10T03:50 BarkerJrCoast1-74491A 278 279
2010-07-10T04:05 BarkerJrCoast1-74491A 288 287
2010-07-10T04:20 BarkerJrCoast1-74491A 269 269
2010-07-10T04:35 BarkerJrCoast1-74491A 197 317
2010-07-10T04:50 BarkerJrCoast1-74491A 228 229
2010-07-10T05:05 BarkerJrCoast1-74491A 252 251
2010-07-10T05:20 BarkerJrCoast1-74491A 213 211
2010-07-10T05:35 BarkerJrCoast1-74491A 197 197
2010-07-10T05:50 BarkerJrCoast1-74491A 236 237
2010-07-10T06:05 BarkerJrCoast1-74491A 217 216
2010-07-10T06:20 BarkerJrCoast1-74491A 218 220
2010-07-10T06:35 BarkerJrCoast1-74491A 191 313
2010-07-10T06:50 BarkerJrCoast1-74491A 213 213
2010-07-10T07:05 BarkerJrCoast1-74491A 228 227
2010-07-10T07:20 BarkerJrCoast1-74491A 273 274
2010-07-10T07:35 BarkerJrCoast1-74491A 288 287
2010-07-10T07:50 BarkerJrCoast1-74491A 238 362
2010-07-10T08:05 BarkerJrCoast1-74491A 240 240
2010-07-10T08:20 BarkerJrCoast1-74491A 243 243
2010-07-10T08:35 BarkerJrCoast1-74491A 227 227
2010-07-10T08:50 BarkerJrCoast1-74491A 206 206
2010-07-10T09:05 BarkerJrCoast1-74491A 191 190
2010-07-10T09:20 BarkerJrCoast1-74491A 176 304
2010-07-10T09:35 BarkerJrCoast1-74491A 218 218
2010-07-10T09:50 BarkerJrCoast1-74491A 229 229
2010-07-10T10:05 BarkerJrCoast1-74491A 235 235
2010-07-10T10:20 BarkerJrCoast1-74491A 251 251
2010-07-10T10:35 BarkerJrCoast1-74491A 239 239
2010-07-10T10:50 BarkerJrCoast1-74491A 199 198
2010-07-10T11:05 BarkerJrCoast1-74491A 213 342
2010-07-10T11:20 BarkerJrCoast1-74491A 198 198
2010-07-10T11:35 BarkerJrCoast1-74491A 252 252
2010-07-10T11:50 BarkerJrCoast1-74491A 278 278
2010-07-10T12:05 BarkerJrCoast1-74491A 271 271
2010-07-10T12:20 BarkerJrCoast1-74491A 332 465
2010-07-10T12:35 BarkerJrCoast1-74491A 244 244
2010-07-10T12:50 BarkerJrCoast1-74491A 274 274
2010-07-10T13:05 BarkerJrCoast1-74491A 281 281
2010-07-10T13:20 BarkerJrCoast1-74491A 287 286
2010-07-10T13:35 BarkerJrCoast1-74491A 243 223
2010-07-10T13:50 BarkerJrCoast1-74491A 284 415
2010-07-10T14:05 BarkerJrCoast1-74491A 282 282
2010-07-10T14:20 BarkerJrCoast1-74491A 299 299
2010-07-10T14:35 BarkerJrCoast1-74491A 283 283
2010-07-10T14:50 BarkerJrCoast1-74491A 250 250
2010-07-10T15:05 BarkerJrCoast1-74491A 259 259
2010-07-10T15:20 BarkerJrCoast1-74491A 298 432
2010-07-10T15:35 BarkerJrCoast1-74491A 288 289
2010-07-10T15:50 BarkerJrCoast1-74491A 262 261
2010-07-10T16:05 BarkerJrCoast1-74491A 235 235
2010-07-10T16:20 BarkerJrCoast1-74491A 220 220
2010-07-10T16:35 BarkerJrCoast1-74491A 239 239
2010-07-10T16:50 BarkerJrCoast1-74491A 204 337
2010-07-10T17:05 BarkerJrCoast1-74491A 214 215
2010-07-10T17:20 BarkerJrCoast1-74491A 210 210
2010-07-10T17:35 BarkerJrCoast1-74491A 171 170
2010-07-10T17:50 BarkerJrCoast1-74491A 181 181
2010-07-10T18:05 BarkerJrCoast1-74491A 220 221
2010-07-10T18:20 BarkerJrCoast1-74491A 223 357
2010-07-10T18:35 BarkerJrCoast1-74491A 202 203
2010-07-10T18:50 BarkerJrCoast1-74491A 183 182
2010-07-10T19:05 BarkerJrCoast1-74491A 206 206
2010-07-10T19:20 BarkerJrCoast1-74491A 211 347
2010-07-10T19:35 BarkerJrCoast1-74491A 166 166
2010-07-10T19:50 BarkerJrCoast1-74491A 180 313
2010-07-10T20:05 BarkerJrCoast1-74491A 181 180
2010-07-10T20:20 BarkerJrCoast1-74491A 176 177
2010-07-10T20:35 BarkerJrCoast1-74491A 174 309
2010-07-10T20:50 BarkerJrCoast1-74491A 173 172
2010-07-10T21:05 BarkerJrCoast1-74491A 175 175
2010-07-10T21:20 BarkerJrCoast1-74491A 170 171
2010-07-10T21:35 BarkerJrCoast1-74491A 175 173
2010-07-10T21:50 BarkerJrCoast1-74491A 195 195
2010-07-10T22:05 BarkerJrCoast1-74491A 174 173
2010-07-10T22:20 BarkerJrCoast1-74491A 147 148
2010-07-10T22:35 BarkerJrCoast1-74491A 148 147
2010-07-10T22:50 BarkerJrCoast1-74491A 142 272
2010-07-10T23:05 BarkerJrCoast1-74491A 137 138
2010-07-10T23:20 BarkerJrCoast1-74491A 140 142
2010-07-10T23:35 BarkerJrCoast1-74491A 139 139
2010-07-10T23:50 BarkerJrCoast1-74491A 157 159
2010-07-11T00:05 BarkerJrCoast1-74491A 174 300
2010-07-11T00:20 BarkerJrCoast1-74491A 165 166
2010-07-11T00:35 BarkerJrCoast1-74491A 170 170
2010-07-11T00:50 BarkerJrCoast1-74491A 145 145
2010-07-11T01:05 BarkerJrCoast1-74491A 159 160
2010-07-11T01:20 BarkerJrCoast1-74491A 182 182
2010-07-11T01:35 BarkerJrCoast1-74491A 244 368
2010-07-11T01:50 BarkerJrCoast1-74491A 206 207
2010-07-11T02:05 BarkerJrCoast1-74491A 214 215
2010-07-11T02:20 BarkerJrCoast1-74491A 253 254
2010-07-11T02:35 BarkerJrCoast1-74491A 220 220
2010-07-11T02:50 BarkerJrCoast1-74491A 171 294
2010-07-11T03:05 BarkerJrCoast1-74491A 211 212
2010-07-11T03:20 BarkerJrCoast1-74491A 269 269
2010-07-11T03:35 BarkerJrCoast1-74491A 234 234
2010-07-11T03:50 BarkerJrCoast1-74491A 248 249
2010-07-11T04:05 BarkerJrCoast1-74491A 178 299
2010-07-11T04:20 BarkerJrCoast1-74491A 208 209
2010-07-11T04:35 BarkerJrCoast1-74491A 186 186
2010-07-11T04:50 BarkerJrCoast1-74491A 172 172
2010-07-11T05:05 BarkerJrCoast1-74491A 183 184
2010-07-11T05:20 BarkerJrCoast1-74491A 173 293
2010-07-11T05:35 BarkerJrCoast1-74491A 179 179
2010-07-11T05:50 BarkerJrCoast1-74491A 157 157
2010-07-11T06:05 BarkerJrCoast1-74491A 215 215
2010-07-11T06:20 BarkerJrCoast1-74491A 215 217
2010-07-11T06:35 BarkerJrCoast1-74491A 204 204
2010-07-11T06:50 BarkerJrCoast1-74491A 190 190
2010-07-11T07:05 BarkerJrCoast1-74491A 213 215
2010-07-11T07:20 BarkerJrCoast1-74491A 200 322
2010-07-11T07:35 BarkerJrCoast1-74491A 189 167
2010-07-11T07:50 BarkerJrCoast1-74491A 215 216
2010-07-11T08:05 BarkerJrCoast1-74491A 218 219
2010-07-11T08:20 BarkerJrCoast1-74491A 172 172
2010-07-11T08:35 BarkerJrCoast1-74491A 214 212
2010-07-11T08:50 BarkerJrCoast1-74491A 186 186
2010-07-11T09:05 BarkerJrCoast1-74491A 173 298
2010-07-11T09:20 BarkerJrCoast1-74491A 192 192
2010-07-11T09:35 BarkerJrCoast1-74491A 203 204
2010-07-11T09:50 BarkerJrCoast1-74491A 224 224
2010-07-11T10:05 BarkerJrCoast1-74491A 210 210
2010-07-11T10:20 BarkerJrCoast1-74491A 167 168
2010-07-11T10:35 BarkerJrCoast1-74491A 190 321
2010-07-11T10:50 BarkerJrCoast1-74491A 158 158
2010-07-11T11:05 BarkerJrCoast1-74491A 220 220
2010-07-11T11:20 BarkerJrCoast1-74491A 186 184
2010-07-11T11:35 BarkerJrCoast1-74491A 179 179
2010-07-11T11:50 BarkerJrCoast1-74491A 211 212
2010-07-11T12:05 BarkerJrCoast1-74491A 228 227
2010-07-11T12:20 BarkerJrCoast1-74491A 268 267
2010-07-11T12:35 BarkerJrCoast1-74491A 343 472
2010-07-11T12:50 BarkerJrCoast1-74491A 350 350
2010-07-11T13:05 BarkerJrCoast1-74491A 321 321
2010-07-11T13:20 BarkerJrCoast1-74491A 262 261
2010-07-11T13:35 BarkerJrCoast1-74491A 245 246
2010-07-11T13:50 BarkerJrCoast1-74491A 248 378
2010-07-11T14:05 BarkerJrCoast1-74491A 263 264
2010-07-11T14:20 BarkerJrCoast1-74491A 226 225
2010-07-11T14:35 BarkerJrCoast1-74491A 243 243
2010-07-11T14:50 BarkerJrCoast1-74491A 206 207
2010-07-11T15:05 BarkerJrCoast1-74491A 190 191
2010-07-11T15:20 BarkerJrCoast1-74491A 180 180
2010-07-11T15:35 BarkerJrCoast1-74491A 189 321
2010-07-11T15:50 BarkerJrCoast1-74491A 216 217
2010-07-11T16:05 BarkerJrCoast1-74491A 194 194
2010-07-11T16:20 BarkerJrCoast1-74491A 223 223
2010-07-11T16:35 BarkerJrCoast1-74491A 224 225
2010-07-11T16:50 BarkerJrCoast1-74491A 227 227
2010-07-11T17:05 BarkerJrCoast1-74491A 204 336
2010-07-11T17:20 BarkerJrCoast1-74491A 143 143
2010-07-11T17:35 BarkerJrCoast1-74491A 143 142
2010-07-11T17:50 BarkerJrCoast1-74491A 141 139
2010-07-11T18:05 BarkerJrCoast1-74491A 149 148
2010-07-11T18:20 BarkerJrCoast1-74491A 151 284
2010-07-11T18:35 BarkerJrCoast1-74491A 127 127
2010-07-11T18:50 BarkerJrCoast1-74491A 124 125
2010-07-11T19:05 BarkerJrCoast1-74491A 141 139
2010-07-11T19:20 BarkerJrCoast1-74491A 135 136
2010-07-11T19:35 BarkerJrCoast1-74491A 132 132
2010-07-11T19:50 BarkerJrCoast1-74491A 120 253
2010-07-11T20:05 BarkerJrCoast1-74491A 129 128
2010-07-11T20:20 BarkerJrCoast1-74491A 155 157
2010-07-11T20:35 BarkerJrCoast1-74491A 147 148
2010-07-11T20:50 BarkerJrCoast1-74491A 132 133
2010-07-11T21:05 BarkerJrCoast1-74491A 120 119
2010-07-11T21:20 BarkerJrCoast1-74491A 125 258
2010-07-11T21:35 BarkerJrCoast1-74491A 129 131
2010-07-11T21:50 BarkerJrCoast1-74491A 133 134
2010-07-11T22:05 BarkerJrCoast1-74491A 126 125
2010-07-11T22:20 BarkerJrCoast1-74491A 126 126
2010-07-11T22:35 BarkerJrCoast1-74491A 120 120
2010-07-11T22:50 BarkerJrCoast1-74491A 117 248
2010-07-11T23:05 BarkerJrCoast1-74491A 131 130
2010-07-11T23:20 BarkerJrCoast1-74491A 127 127
2010-07-11T23:35 BarkerJrCoast1-74491A 127 128
2010-07-11T23:50 BarkerJrCoast1-74491A 121 122
2010-07-12T00:05 BarkerJrCoast1-74491A 124 123
2010-07-12T00:20 BarkerJrCoast1-74491A 128 253
2010-07-12T00:35 BarkerJrCoast1-74491A 123 124
2010-07-12T00:50 BarkerJrCoast1-74491A 143 142
2010-07-12T01:05 BarkerJrCoast1-74491A 133 133
2010-07-12T01:20 BarkerJrCoast1-74491A 165 166
2010-07-12T01:35 BarkerJrCoast1-74491A 216 319
2010-07-12T01:50 BarkerJrCoast1-74491A 179 179
2010-07-12T02:05 BarkerJrCoast1-74491A 177 176
2010-07-12T02:20 BarkerJrCoast1-74491A 163 163
2010-07-12T02:35 BarkerJrCoast1-74491A 181 303
2010-07-12T02:50 BarkerJrCoast1-74491A 187 189
2010-07-12T03:05 BarkerJrCoast1-74491A 165 164
2010-07-12T03:20 BarkerJrCoast1-74491A 184 184
2010-07-12T03:35 BarkerJrCoast1-74491A 174 293
2010-07-12T03:50 BarkerJrCoast1-74491A 170 170
2010-07-12T04:05 BarkerJrCoast1-74491A 128 128
2010-07-12T04:20 BarkerJrCoast1-74491A 126 125
2010-07-12T04:35 BarkerJrCoast1-74491A 154 154
2010-07-12T04:50 BarkerJrCoast1-74491A 131 132
2010-07-12T05:05 BarkerJrCoast1-74491A 116 116
2010-07-12T05:20 BarkerJrCoast1-74491A 140 141
2010-07-12T05:35 BarkerJrCoast1-74491A 135 256
2010-07-12T05:50 BarkerJrCoast1-74491A 125 125
2010-07-12T06:05 BarkerJrCoast1-74491A 116 117
2010-07-12T06:20 BarkerJrCoast1-74491A 123 124
2010-07-12T06:35 BarkerJrCoast1-74491A 133 134
2010-07-12T06:50 BarkerJrCoast1-74491A 128 128
2010-07-12T07:05 BarkerJrCoast1-74491A 163 163
2010-07-12T07:20 BarkerJrCoast1-74491A 159 283
2010-07-12T07:35 BarkerJrCoast1-74491A 174 175
2010-07-12T07:50 BarkerJrCoast1-74491A 193 193
2010-07-12T08:05 BarkerJrCoast1-74491A 157 158
2010-07-12T08:20 BarkerJrCoast1-74491A 132 133
2010-07-12T08:35 BarkerJrCoast1-74491A 121 121
2010-07-12T08:50 BarkerJrCoast1-74491A 147 272
2010-07-12T09:05 BarkerJrCoast1-74491A 138 140
2010-07-12T09:20 BarkerJrCoast1-74491A 137 139
2010-07-12T09:35 BarkerJrCoast1-74491A 176 177
2010-07-12T09:50 BarkerJrCoast1-74491A 179 179
2010-07-12T10:05 BarkerJrCoast1-74491A 207 333
2010-07-12T10:20 BarkerJrCoast1-74491A 198 200
2010-07-12T10:35 BarkerJrCoast1-74491A 170 297
2010-07-12T10:50 BarkerJrCoast1-74491A 140 140
2010-07-12T11:05 BarkerJrCoast1-74491A 200 199
2010-07-12T11:20 BarkerJrCoast1-74491A 250 251
2010-07-12T11:35 BarkerJrCoast1-74491A 257 259
2010-07-12T11:50 BarkerJrCoast1-74491A 219 219
2010-07-12T12:05 BarkerJrCoast1-74491A 219 220
2010-07-12T12:20 BarkerJrCoast1-74491A 256 386
2010-07-12T12:35 BarkerJrCoast1-74491A 208 208
2010-07-12T12:50 BarkerJrCoast1-74491A 164 163
2010-07-12T13:05 BarkerJrCoast1-74491A 224 224
2010-07-12T13:20 BarkerJrCoast1-74491A 233 232
2010-07-12T13:35 BarkerJrCoast1-74491A 234 237
2010-07-12T13:50 BarkerJrCoast1-74491A 273 272
2010-07-12T14:05 BarkerJrCoast1-74491A 254 255
2010-07-12T14:20 BarkerJrCoast1-74491A 298 429
2010-07-12T14:35 BarkerJrCoast1-74491A 295 296
2010-07-12T14:50 BarkerJrCoast1-74491A 235 235
2010-07-12T15:05 BarkerJrCoast1-74491A 225 227
2010-07-12T15:20 BarkerJrCoast1-74491A 178 179
2010-07-12T15:35 BarkerJrCoast1-74491A 205 205
2010-07-12T15:50 BarkerJrCoast1-74491A 181 181
2010-07-12T16:05 BarkerJrCoast1-74491A 177 310
2010-07-12T16:20 BarkerJrCoast1-74491A 190 189
2010-07-12T16:35 BarkerJrCoast1-74491A 225 225
2010-07-12T16:50 BarkerJrCoast1-74491A 179 180
2010-07-12T17:05 BarkerJrCoast1-74491A 165 165
2010-07-12T17:20 BarkerJrCoast1-74491A 157 292
2010-07-12T17:35 BarkerJrCoast1-74491A 121 121
2010-07-12T17:50 BarkerJrCoast1-74491A 115 115
2010-07-12T18:05 BarkerJrCoast1-74491A 109 110
2010-07-12T18:20 BarkerJrCoast1-74491A 109 111
2010-07-12T18:35 BarkerJrCoast1-74491A 90 89
2010-07-12T18:50 BarkerJrCoast1-74491A 53 54
2010-07-12T19:05 BarkerJrCoast1-74491A 59 59
2010-07-12T19:20 BarkerJrCoast1-74491A 55 192
2010-07-12T19:35 BarkerJrCoast1-74491A 101 79
2010-07-12T19:50 BarkerJrCoast1-74491A 85 85
2010-07-12T20:05 BarkerJrCoast1-74491A 85 85
2010-07-12T20:20 BarkerJrCoast1-74491A 55 57
2010-07-12T20:35 BarkerJrCoast1-74491A 89 88
2010-07-12T20:50 BarkerJrCoast1-74491A 59 197
2010-07-12T21:05 BarkerJrCoast1-74491A 51 50
2010-07-12T21:20 BarkerJrCoast1-74491A 50 51
2010-07-12T21:35 BarkerJrCoast1-74491A 54 54
2010-07-12T21:50 BarkerJrCoast1-74491A 55 56
2010-07-12T22:05 BarkerJrCoast1-74491A 61 61
2010-07-12T22:20 BarkerJrCoast1-74491A 51 184
2010-07-12T22:35 BarkerJrCoast1-74491A 57 58
2010-07-12T22:50 BarkerJrCoast1-74491A 63 63
2010-07-12T23:05 BarkerJrCoast1-74491A 74 74
2010-07-12T23:20 BarkerJrCoast1-74491A 84 86
2010-07-12T23:35 BarkerJrCoast1-74491A 64 64
2010-07-12T23:50 BarkerJrCoast1-74491A 57 58
2010-07-13T00:05 BarkerJrCoast1-74491A 60 190
2010-07-13T00:20 BarkerJrCoast1-74491A 51 53
2010-07-13T00:35 BarkerJrCoast1-74491A 55 55
2010-07-13T00:50 BarkerJrCoast1-74491A 81 82
2010-07-13T01:05 BarkerJrCoast1-74491A 65 66
2010-07-13T01:20 BarkerJrCoast1-74491A 81 82
2010-07-13T01:35 BarkerJrCoast1-74491A 61 184
2010-07-13T01:50 BarkerJrCoast1-74491A 94 94
2010-07-13T02:05 BarkerJrCoast1-74491A 120 122
2010-07-13T02:20 BarkerJrCoast1-74491A 76 77
2010-07-13T02:35 BarkerJrCoast1-74491A 77 198
2010-07-13T02:50 BarkerJrCoast1-74491A 126 127
2010-07-13T03:05 BarkerJrCoast1-74491A 137 138
2010-07-13T03:20 BarkerJrCoast1-74491A 117 117
2010-07-13T03:35 BarkerJrCoast1-74491A 120 122
2010-07-13T03:50 BarkerJrCoast1-74491A 81 81
2010-07-13T04:05 BarkerJrCoast1-74491A 89 210
2010-07-13T04:20 BarkerJrCoast1-74491A 93 92
2010-07-13T04:35 BarkerJrCoast1-74491A 151 152
2010-07-13T04:50 BarkerJrCoast1-74491A 128 128
2010-07-13T05:05 BarkerJrCoast1-74491A 112 112
2010-07-13T05:20 BarkerJrCoast1-74491A 86 86
2010-07-13T05:35 BarkerJrCoast1-74491A 92 92
2010-07-13T05:50 BarkerJrCoast1-74491A 103 103
2010-07-13T06:05 BarkerJrCoast1-74491A 118 240
2010-07-13T06:20 BarkerJrCoast1-74491A 90 90
2010-07-13T06:35 BarkerJrCoast1-74491A 74 75
2010-07-13T06:50 BarkerJrCoast1-74491A 81 81
2010-07-13T07:05 BarkerJrCoast1-74491A 99 99
2010-07-13T07:20 BarkerJrCoast1-74491A 75 76
2010-07-13T07:35 BarkerJrCoast1-74491A 76 199
2010-07-13T07:50 BarkerJrCoast1-74491A 88 88
2010-07-13T08:05 BarkerJrCoast1-74491A 101 102
2010-07-13T08:20 BarkerJrCoast1-74491A 71 72
2010-07-13T08:35 BarkerJrCoast1-74491A 133 134
2010-07-13T08:50 BarkerJrCoast1-74491A 79 204
2010-07-13T09:05 BarkerJrCoast1-74491A 69 71
2010-07-13T09:20 BarkerJrCoast1-74491A 115 115
2010-07-13T09:35 BarkerJrCoast1-74491A 98 99
2010-07-13T09:50 BarkerJrCoast1-74491A 119 120
2010-07-13T10:05 BarkerJrCoast1-74491A 83 210
2010-07-13T10:20 BarkerJrCoast1-74491A 82 82
2010-07-13T10:35 BarkerJrCoast1-74491A 137 138
2010-07-13T10:50 BarkerJrCoast1-74491A 111 112
2010-07-13T11:05 BarkerJrCoast1-74491A 123 123
2010-07-13T11:20 BarkerJrCoast1-74491A 99 99
2010-07-13T11:35 BarkerJrCoast1-74491A 84 213
2010-07-13T11:50 BarkerJrCoast1-74491A 103 104
2010-07-13T12:05 BarkerJrCoast1-74491A 104 104
2010-07-13T12:20 BarkerJrCoast1-74491A 123 123
2010-07-13T12:35 BarkerJrCoast1-74491A 179 179
2010-07-13T12:50 BarkerJrCoast1-74491A 128 128
2010-07-13T13:05 BarkerJrCoast1-74491A 113 115
2010-07-13T13:20 BarkerJrCoast1-74491A 131 261
2010-07-13T13:35 BarkerJrCoast1-74491A 189 169
2010-07-13T13:50 BarkerJrCoast1-74491A 143 144
2010-07-13T14:05 BarkerJrCoast1-74491A 160 161
2010-07-13T14:20 BarkerJrCoast1-74491A 173 174
2010-07-13T14:35 BarkerJrCoast1-74491A 132 133
2010-07-13T14:50 BarkerJrCoast1-74491A 182 183
2010-07-13T15:05 BarkerJrCoast1-74491A 129 259
2010-07-13T15:20 BarkerJrCoast1-74491A 125 127
2010-07-13T15:35 BarkerJrCoast1-74491A 124 125
2010-07-13T15:50 BarkerJrCoast1-74491A 161 163
2010-07-13T16:05 BarkerJrCoast1-74491A 120 120
2010-07-13T16:20 BarkerJrCoast1-74491A 154 155
2010-07-13T16:35 BarkerJrCoast1-74491A 133 264
2010-07-13T16:50 BarkerJrCoast1-74491A 89 90
2010-07-13T17:05 BarkerJrCoast1-74491A 98 97
2010-07-13T17:20 BarkerJrCoast1-74491A 73 74
2010-07-13T17:35 BarkerJrCoast1-74491A 62 63
2010-07-13T17:50 BarkerJrCoast1-74491A 62 196
2010-07-13T18:05 BarkerJrCoast1-74491A 54 54
2010-07-13T18:20 BarkerJrCoast1-74491A 52 53
2010-07-13T18:35 BarkerJrCoast1-74491A 53 55
2010-07-13T18:50 BarkerJrCoast1-74491A 61 63
2010-07-13T19:05 BarkerJrCoast1-74491A 63 199
2010-07-13T19:20 BarkerJrCoast1-74491A 49 51
2010-07-13T19:35 BarkerJrCoast1-74491A 52 53
2010-07-13T19:50 BarkerJrCoast1-74491A 57 58
2010-07-13T20:05 BarkerJrCoast1-74491A 63 197
2010-07-13T20:20 BarkerJrCoast1-74491A 54 55
2010-07-13T20:35 BarkerJrCoast1-74491A 56 57
2010-07-13T20:50 BarkerJrCoast1-74491A 59 58
2010-07-13T21:05 BarkerJrCoast1-74491A 50 51
2010-07-13T21:20 BarkerJrCoast1-74491A 52 51
2010-07-13T21:35 BarkerJrCoast1-74491A 52 51
2010-07-13T21:50 BarkerJrCoast1-74491A 54 187
2010-07-13T22:05 BarkerJrCoast1-74491A 58 57
2010-07-13T22:20 BarkerJrCoast1-74491A 64 64
2010-07-13T22:35 BarkerJrCoast1-74491A 57 57
2010-07-13T22:50 BarkerJrCoast1-74491A 59 57
2010-07-13T23:05 BarkerJrCoast1-74491A 50 49
2010-07-13T23:20 BarkerJrCoast1-74491A 60 61
2010-07-13T23:35 BarkerJrCoast1-74491A 51 180
2010-07-13T23:50 BarkerJrCoast1-74491A 53 54
2010-07-14T00:05 BarkerJrCoast1-74491A 52 51
2010-07-14T00:20 BarkerJrCoast1-74491A 95 95
2010-07-14T00:35 BarkerJrCoast1-74491A 76 77
2010-07-14T00:50 BarkerJrCoast1-74491A 78 78
2010-07-14T01:05 BarkerJrCoast1-74491A 54 54
2010-07-14T01:20 BarkerJrCoast1-74491A 54 178
2010-07-14T01:35 BarkerJrCoast1-74491A 59 61
2010-07-14T01:50 BarkerJrCoast1-74491A 57 57
2010-07-14T02:05 BarkerJrCoast1-74491A 57 58
2010-07-14T02:20 BarkerJrCoast1-74491A 49 50
2010-07-14T02:35 BarkerJrCoast1-74491A 72 73
2010-07-14T02:50 BarkerJrCoast1-74491A 61 183
2010-07-14T03:05 BarkerJrCoast1-74491A 68 69
2010-07-14T03:20 BarkerJrCoast1-74491A 77 77
2010-07-14T03:35 BarkerJrCoast1-74491A 124 247
2010-07-14T03:50 BarkerJrCoast1-74491A 59 59
2010-07-14T04:05 BarkerJrCoast1-74491A 100 101
2010-07-14T04:20 BarkerJrCoast1-74491A 101 101
2010-07-14T04:35 BarkerJrCoast1-74491A 85 86
2010-07-14T04:50 BarkerJrCoast1-74491A 67 188
2010-07-14T05:05 BarkerJrCoast1-74491A 85 86
2010-07-14T05:20 BarkerJrCoast1-74491A 60 61
2010-07-14T05:35 BarkerJrCoast1-74491A 80 81
2010-07-14T05:50 BarkerJrCoast1-74491A 68 67
2010-07-14T06:05 BarkerJrCoast1-74491A 64 66
2010-07-14T06:20 BarkerJrCoast1-74491A 65 65
2010-07-14T06:35 BarkerJrCoast1-74491A 64 188
2010-07-14T06:50 BarkerJrCoast1-74491A 107 106
2010-07-14T07:05 BarkerJrCoast1-74491A 92 93
2010-07-14T07:20 BarkerJrCoast1-74491A 59 60

comment:13 Changed 9 years ago by mwenge

So to sum up so far:

  • BarkerJrCoast1 hibernated on the 7th July when it reached its soft-limit. This was somewhere around the 190GB mark, so it looks like its AccountingMax was indeed 200GB.
  • It stopped sending descriptors around the 14th.
  • Its logs report shortly after the 14th that it went into hibernation again. So something happened to the machine or network on that date that caused that to happen and also caused it to stop reporting descriptors. Maybe it changed identity or something?

comment:14 Changed 9 years ago by arma

My first thought is that our calculation for when to wake up is wrong. Notice how we never go into hard hibernation, just soft. Also notice how we aren't waking up at exactly midnight on day 1 -- we start a few hours in, like we think we're leaving 99% of the period available because last month we only used 99% of our bandwidth, which means this month we should wake up way earlier to make sure to use all of it.

comment:15 in reply to:  14 Changed 9 years ago by mwenge

Replying to arma:

My first thought is that our calculation for when to wake up is wrong. Notice how we never go into hard hibernation, just soft. Also notice how we aren't waking up at exactly midnight on day 1 -- we start a few hours in, like we think we're leaving 99% of the period available because last month we only used 99% of our bandwidth, which means this month we should wake up way earlier to make sure to use all of it.

When you have a AccountingMax of 200GB your soft limit is 190GB, and the last set of spam stats show this is the threshold the router hit around the 7th. However it never reached its hard limit, and its hard to think of a router with such an AccountingMax that could, because hitting the soft limit means you don't open new connections - just maintain old ones. The router's traffic dropped off and it read another 2GB or so. At that rate it would not have reached its AccountingMax for the month.

So if you're correct that the next accounting interval is based on the rate at which you reached your hard limit rather than your soft one then all large capacity routers must have this problem. My reading of the code is that you are right: n_seconds_active_in_interval appears to be the total time for which we were reading/writing bytes and that, together with the bytes read/written, is used to calculate our expected bandwidth usage.

So it looks like Tor needs to tune the 'soft limit is 95% of AccountingMax' rule to tolerate servers that specify a high capacity.

comment:16 Changed 9 years ago by Sebastian

Yeah, we should do that. But also we should just consider how long it took us to reach that new soft limit. Never quite reaching the hard limit is a good thing in my world, because it means we didn't break any long-running client connections.

comment:17 Changed 9 years ago by nickm

Okay, so two new rules that might be good:

  • Set SoftLimit to 95% of accountingmax, OR to accountingmax - 500 MB, OR to AccountingMax * (expected_uptime - 3 hours)/expected_uptime, whichever is *highest*.
  • When we hit the SoftLimit, record the number of bytes-per-minute that we, on average, logging up to that time. Use _that_ to determine our usage for the next interval.

Sounds plausible?

comment:18 Changed 9 years ago by Sebastian

Yes, that sounds smart.

comment:19 Changed 9 years ago by nickm

Status: newneeds_review

See branch bug1789 in my public. Apparently I did this last week and forgot to tell anybody about it. :p

comment:20 Changed 9 years ago by Sebastian

+    n_seconds_to_hit_soft_limit = state->AccountingSoftLimitHitAt;
}}
This looks wrong. I think we want
{{{
state->AccountingSoftLimitHitAt - state->AccountingIntervalStart
}}}

comment:21 Changed 9 years ago by Sebastian

Well, I didn't want that at all, especially not with crappy formatting. New fix is in branch bug1789 in my repository. Looks good to me otherwise

comment:22 Changed 9 years ago by Sebastian

You might want to take out the "Just take the " comment though, or complete it. I think it is clear enough what happens though.

comment:23 Changed 9 years ago by nickm

Merged your branch onto mine; fixed the comment. thanks!

comment:24 Changed 9 years ago by arma

+  uint64_t soft_limit = DBL_TO_U64(U64_TO_DBL(acct_max) * SOFT_LIM_PCT);
+  if (acct_max > SOFT_LIM_BYTES && acct_max - SOFT_LIM_BYTES > soft_limit) {
+    soft_limit = acct_max - SOFT_LIM_BYTES;
+  }

I'm not certain of my C anymore, but it seems like the "acct_max > SOFT_LIM_BYTES" part here is redundant, yes? If acct_max - SOFT_LIM_BYTES is negative, it's hardly going to find itself > soft_limit, which is a uint64, right?

I mean, I can see the underflow worry if you were assigning acct_max - SOFT_LIM_BYTES to a uint64_t before the comparison. But you're not. Are compilers really so stupid? Or to say it more nicely, are compilers really so fickle?

Same deal with the later stanza:

+    if (acct_max > expected_usage && acct_max - expected_usage > soft_limit)
+      soft_limit = acct_max - expected_usage;

comment:25 Changed 9 years ago by arma

   uint64_t max_configured = (get_options()->BandwidthRate * 60);

We probably want to consider RelayBandwidthRate here when it's non-zero.

So e.g.

  or_options_t *options = get_options();
  uint64_t max_configured = (options->RelayBandwidthRate > 0 ?
                               options->RelayBandwidthRate :
                               options->BandwidthRate) * 60;

comment:26 Changed 9 years ago by arma

+    n_bytes_at_soft_limit = MAX(n_bytes_read_in_interval,
+                               n_bytes_written_in_interval);

formatting error (missing a space)

comment:27 Changed 9 years ago by arma

+  /* Older versions of Tor (before 0.2.2.16-alpha) didn't generate these

Version number needs to be bumped. :( Maybe bumped twice, depending on whether we're planning to call the one that compiles on Windows 16b or 17.

comment:28 Changed 9 years ago by arma

+      Now, we use up 95% of our allotment, AND make sure that we have
+      no more than 500MB/3 hours of traffic remaining before we enter
+      soft hibernation.

We should break out the "500MB/3" to make the logic clearer.

If I understand it right, it means we make sure we've used 95% of our allotment, AND make sure we have no more than 500MB of traffic left, AND make sure we have no more than 3 hours of traffic left at the current pace, before we enter soft hibernation?

comment:29 Changed 9 years ago by arma

Looks good other than those. Green light to merge whenever you want.

comment:30 in reply to:  24 Changed 9 years ago by nickm

Resolution: fixed
Status: needs_reviewclosed

Replying to arma:

I'm not certain of my C anymore, but it seems like the "acct_max > SOFT_LIM_BYTES" part here is redundant, yes? If acct_max - SOFT_LIM_BYTES is negative, it's hardly going to find itself > soft_limit, which is a uint64, right?

acct_max is unsigned, so acct_max - SOFT_LIM_BYTES can't be negative. If you take a uint64_t and subtract a number from it, the result is a uint64_t, even if it wraps around.

To check how C works, write a test program:

#include <stdio.h>
int main(int argc, char **argv)
{
  unsigned a;
  a = 99;
  printf("a - 100 = %u\n", a-100);
  printf("(a - 100 > 1) == %d\n", (a-100)>1);
  return 0;
}

I mean, I can see the underflow worry if you were assigning acct_max - SOFT_LIM_BYTES to a uint64_t before the comparison. But you're not. Are compilers really so stupid? Or to say it more nicely, are compilers really so fickle?

Neither. This is how unsigned arithmetic works in C. When you take a signed integer and an unsigned integer and do a binary operation with them, the signed int is first converted to an unsigned[*] before the operation happens. Unsigned overflow/underflow is defined to wrap around.

Fixed the other stuff, and merging.

[*] Unless the signed type is larger than the unsigned type, such that it can hold every legal value of the unsigned type. (K&R 2nd ed Section A6)

comment:31 Changed 7 years ago by nickm

Keywords: tor-relay added

comment:32 Changed 7 years ago by nickm

Component: Tor RelayTor
Note: See TracTickets for help on using tickets.