Opened 4 years ago

Closed 4 years ago

#9668 closed defect (fixed)

restructure flashproxy source tree

Reported by: infinity0 Owned by: dcf
Priority: Medium Milestone:
Component: Archived/Flashproxy Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

The source tree has quite a messy structure that makes it hard to write a coherent install script, e.g. for debian packaging. I propose something like the following:

# Top-level
# renamed: doc/design.txt -> design.txt

# client
# renamed: Makefile -> client/Makefile
# renamed: doc/flashproxy-$$ -> client/doc/flashproxy-$$
# renamed: flashproxy-$$ -> client/flashproxy-$$
# renamed: setup.py -> client/setup.py
# renamed: torrc -> client/torrc

# facilitator
# renamed: doc/facilitator-howto.txt -> facilitator/doc/facilitator-howto.txt
# renamed: doc/appengine-howto.txt -> facilitator/doc/appengine-howto.txt
# renamed: doc/gmail-setup.txt -> facilitator/doc/gmail-setup.txt
# renamed: appengine/$$ -> facilitator/appengine/$$

# websocket-transport
# renamed: doc/websocket-transport.txt -> websocket-transport/doc

# proxy
# renamed: modules/$$ -> proxy/modules/$$
/websocket-transport.txt

plus the necessary adjustments (paths etc) to make everything work again.

Child Tickets

Change History (13)

comment:1 Changed 4 years ago by infinity0

Hey, let me know if you disagree in principle with this idea - otherwise I will work on it as my immediate next task.

comment:2 Changed 4 years ago by dcf

What parts are you packaging? Did you try make dist and make install DESTDIR=...? Those package the things you need for flashproxy-client. (What ends up in the PT TBB.)

Yes to:

  • Moving facilitator docs to /facilitator/doc.
  • Moving appengine to /facilitator.

No to:

  • Putting flashproxy-client stuff in a client subdirectory. I would sooner put facilitator and websocket-transport in their own git repos, than make flashproxy-client a second-class citizen in this one.
  • Moving modules to /proxy/modules.
  • Moving websocket-transport.txt out of /doc. flashproxy-client implements this spec too; it's not something that belongs only to websocket-server and websocket-client.
  • Moving design.txt to /. I don't want it and similar documents in the root. design.txt is partially out of date so I don't want it emphasized too much.

Fight the urge to compartmentalize... Eschew deep nesting...

comment:3 Changed 4 years ago by infinity0

I am going to package everything, and my current thinking (very fluid atm) is to have two binary packages:

  • flashproxy-client - client transport plugin plus registration methods, so either similar to, or the same as, the currently-distributed tarball
  • flashproxy-server - server transport plugin, facilitator, plus browser proxy, in the same package just to simplify things

"modules" is vague and ambiguous; if you don't want deep nesting, how about renaming this to "proxy-modules"?

WRT flashproxy-client being a "second-class citizen", well everything else is already a "second-class citizen". My main concern is that the top-level Makefile/setup.py has a bit of an identity crisis. Currently they build the client/client-reg code but Makefile also tries to be a parent for other things, such as running facilitator/proxy tests.

Either it should *only* be responsible for the client, or it should *only* act as an aggregator for other sub-components. (And it's important to have some sort of aggregator to make it easy to "build/dist/test everything".) I don't see a way to resolve this, other than by nesting the client - since if the build scripts are top-level then the temptation will always be there to use it to aggregate other things from subdirectories.

Why keep client stuff at top-level? We can retain a way to make a client-only tarball, if that is your main concern.

(I'm also thinking through the idea of a naming convention like tor-pt-flashproxy, tor-pt-websocket, but that is another concern; I've brought this up on the tor-dev mailing list.)

comment:4 in reply to:  3 Changed 4 years ago by dcf

Replying to infinity0:

  • flashproxy-server - server transport plugin, facilitator, plus browser proxy, in the same package just to simplify things

Don't put facilitator and websocket-server in the same package. Those don't run on the same host. People will want to run websocket-server without running all the junk that comes with the facilitator. websocket-transport is conceptually separate from all the flashproxy stuff; it should actually be in its own repo and should for sure be packaged separately. (BTW please don't package websocket-client; it is alpha code with known bugs that I only undeleted for the purpose of testing in a specific PT bundle.)

comment:5 Changed 4 years ago by infinity0

I've added autotools packaging for the facilitator component. Please take a look:

https://github.com/infinity0/flashproxy/compare/master...src-tree

You can find the new instructions in facilitator/INSTALL. doc/facilitator-howto.txt now only contains post-installation instructions on setting up your httpd.

comment:6 Changed 4 years ago by infinity0

Continuing from 8:ticket:6810:

Let me also explain the packaging model. Currently, the top-level Makefile has two *binary* distribution targets, a python-source and a py2exe zipball. (We call the python-source a "binary" package even though it's source-code python, because the intention is to run it directly, there is no "install" process, and you don't provide any Makefiles or tests.)

OTOH for Debian (and good FOSS practise) I am building source packages, which include tests and build scripts. Everything is in one source repo at the moment, but I am doing source packaging on subfolders to make it easier to split the repo later, which you were vaguely talking about doing. The top-level Makefile will tie everything together for an all-in-one package, but there are two inconsistencies that need sorting:

  1. the top-level Makefile also acts as the sub-Makefile for the client programs (mentioned in 3)
  2. make dist by convention builds a source package, but the current target builds a binary package. I can keep the binary target but will need to rename it.

comment:7 Changed 4 years ago by infinity0

I've merged your latest changes (on flashproxy master) into my src-tree branch, which atm only contains facilitator packaging (plus renaming modules -> proxy-modules). You can have a look here:

https://github.com/infinity0/flashproxy/compare/master...src-tree

It is ready to be reviewed an merged. It ought to be done before #6810, my changes there build on top of this.

Last edited 4 years ago by infinity0 (previous) (diff)

comment:8 in reply to:  7 ; Changed 4 years ago by infinity0

Replying to infinity0:

It ought to be done before #6810, my changes there build on top of this.

Actually, thinking about it, the majority of those changes are independent of this work, I will try to split it into a different branch and update you when I've done that.

comment:9 in reply to:  8 Changed 4 years ago by dcf

Replying to infinity0:

Replying to infinity0:

It ought to be done before #6810, my changes there build on top of this.

Actually, thinking about it, the majority of those changes are independent of this work, I will try to split it into a different branch and update you when I've done that.

Yeah, I was about to say, I think I want to merge #6810 first, so that's great.

comment:10 in reply to:  6 ; Changed 4 years ago by dcf

Replying to infinity0:

Continuing from 8:ticket:6810:

Let me also explain the packaging model. Currently, the top-level Makefile has two *binary* distribution targets, a python-source and a py2exe zipball. (We call the python-source a "binary" package even though it's source-code python, because the intention is to run it directly, there is no "install" process, and you don't provide any Makefiles or tests.)

You are correct, make dist is totally a binary package. It's what ends up in the PT TBB, for example. (Just as with obfsproxy we do setup.py build before copying into the PT TBB, though we could probably get away with just copying straight from the source directory.)

OTOH for Debian (and good FOSS practise) I am building source packages, which include tests and build scripts. Everything is in one source repo at the moment, but I am doing source packaging on subfolders to make it easier to split the repo later, which you were vaguely talking about doing.

Maybe I misunderstand you. Isn't it typical that many binary packages are built from one source package? For example, when I apt-get source openssh-server, it informs me that it is actually downloading the source package "openssh".

$ apt-get source openssh-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
Picking 'openssh' as source package instead of 'openssh-server'
NOTICE: 'openssh' packaging is maintained in the 'Bzr' version control system at:
http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk

I expected that flashproxy packaging would work the same way: binary packages "flashproxy-client" and "flashproxy-facilitator", say, built from a single source package "flashproxy". I don't understand the need to have multiple source packages, each made from a subdirectory.

As for splitting the repo, let's assume that flashproxy-client and facilitator will continue to be together, because they will share libraries. websocket-transport will be completely separate--I don't want it even referred to in the top-level makefile.

comment:11 in reply to:  10 Changed 4 years ago by infinity0

Replying to dcf:

Maybe I misunderstand you. Isn't it typical that many binary packages are built from one source package? For example, when I apt-get source openssh-server, it informs me that it is actually downloading the source package "openssh".

[..]

I expected that flashproxy packaging would work the same way: binary packages "flashproxy-client" and "flashproxy-facilitator", say, built from a single source package "flashproxy". I don't understand the need to have multiple source packages, each made from a subdirectory.

Sorry, by "source package" I simply meant a build script that handles a distinct sub-component. (This makes the packaging process simpler and enforces good separation of concerns.) For example, the setup.py for -common and Makefile for -facilitator. The infrastructure for these build scripts typically also includes "for free" the ability to build a source tarball, but you are correct that we don't need to actually do this and distribute them. The important thing is a coherent and well-supported install process.

comment:12 Changed 4 years ago by dcf

I cherry-picked the changes to do facilitator/appengine, facilitator/doc, and proxy/modules. websocket-transport is gone already.

The rest of the comments in this ticket, and commits in the former src-tree branch, seem to be about facilitator packaging--should we rather open a new ticket about that?

comment:13 Changed 4 years ago by infinity0

Resolution: fixed
Status: newclosed

OK - will file a new ticket.

Note: See TracTickets for help on using tickets.