Opened 9 years ago

Closed 8 years ago

#2392 closed enhancement (fixed)

Support PostGre/mysql backend

Reported by: mikeperry Owned by: mikeperry
Priority: Medium Milestone:
Component: Core Tor/Torflow Version:
Severity: Keywords:
Cc: aagbsn@…, arma Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We should finish the work I started in git branch mikeperry/postgre-bwscanner to rid us of the possibly memleaky and poorly performing sqlite backend. The problem is that for some reason the postgre backend produces incorrect results, and mysql doesn't support the queries that sqlalchemy generates for us. Perhaps this will improve in 0.6.x?

Child Tickets

Change History (7)

comment:1 Changed 9 years ago by aagbsn

Cc: aagbsn@… added

comment:2 Changed 8 years ago by aagbsn

This fix lived here https://github.com/aagbsn/torflow/tree/ticket2392. It has since been merged into master.
The changes consist of:

  1. rewriting queries so that UPDATE target and source tables are not the same (this is achieved by using nested SELECT which cause MySQL to generate temporary tables)
  2. rewriting queries to avoid divide-by-zeros.
  1. adding a new configuration parameter, db_url, to bwauthority.cfg which specifies the database type:

# if db_url is unset bwauthority will default to sqlite
#db_url =
#db_url = mysql+mysqldb://bwscanner:password@127.0.0.1/BwScan1
db_url = postgresql://bwscanner:password@127.0.0.1/BwScan1

comment:3 Changed 8 years ago by aagbsn

Status: newneeds_review

comment:4 Changed 8 years ago by mikeperry

Cc: arma added

Roger, how is your postgres scanner working?

comment:5 Changed 8 years ago by aagbsn

note: Roger ran into a table error with older (sqlalchemy 0.5.x, elixir 0.6.x) using the Postgres backend. Upgrading to (latest at time of this comment) sqlalchemy 0.7.2 and elixir 0.7.1 resolved this issue.

comment:6 Changed 8 years ago by arma

I ran it for a while. The bwauth parents are still going. I mailed the bw.log files to Mike. They have some tracebacks in them. They each stop at different times:

$ date
Thu Aug 25 00:46:25 EDT 2011
$ ls -la scanner.?/bw.log
-rw-r--r-- 1 bwauth bwauth  47047146 2011-08-21 17:18 scanner.1/bw.log
-rw-r--r-- 1 bwauth bwauth   2007759 2011-08-19 22:15 scanner.2/bw.log
-rw-r--r-- 1 bwauth bwauth 175908437 2011-08-24 09:37 scanner.3/bw.log
-rw-r--r-- 1 bwauth bwauth 192325325 2011-08-24 15:03 scanner.4/bw.log

Scanner number 2 looks particularly sad.

comment:7 Changed 8 years ago by aagbsn

Resolution: fixed
Status: needs_reviewclosed

resolved in #3834

Note: See TracTickets for help on using tickets.