Opened 5 years ago

Closed 5 years ago

#13089 closed enhancement (implemented)

use an embedded web server/servlet engine for Onionoo

Reported by: iwakeh Owned by:
Priority: Low Milestone:
Component: Metrics/Onionoo Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Something I meant to ask for a while:
Why not use an embedded web server/servlet engine for Onionoo?
Like embedded tomcat, jetty, or netty?

Deploying would become much easier.

Child Tickets

Attachments (8)

0001-Jetty-embedded.-Redundant-etc-logback.xml-removed.patch (15.1 KB) - added by iwakeh 5 years ago.
embedded jetty, some ant task, some logging
0001-Nicer-logging-configuration.-Version-independent-war.patch (3.2 KB) - added by iwakeh 5 years ago.
based on ​your task-13089 branch
0002-Suggestion-for-INSTALL.patch (4.4 KB) - added by iwakeh 5 years ago.
based on your ​task-13089 branch
onionoo-jetty-switch.png (193.6 KB) - added by karsten 5 years ago.
Performance statistics before/after switching from Tomcat to Jetty
0001-added-compression-to-jetty-config.patch (1.4 KB) - added by iwakeh 5 years ago.
jetty.xml with compression
onionoo-jetty-switch.2.png (189.0 KB) - added by karsten 5 years ago.
Performance statistics before/after switching from Tomcat to Jetty, updated
0001-set-compression-filter-in-web-app.patch (2.2 KB) - added by iwakeh 5 years ago.
web-app compression config based on task-13089-4
onionoo-jetty-switch.3.png (188.5 KB) - added by karsten 5 years ago.
Performance statistics before/after switching from Tomcat to Jetty, updated once more

Download all attachments as: .zip

Change History (36)

comment:1 Changed 5 years ago by karsten

I thought about switching to Jetty a while ago, but frustration about Tomcat was not big enough to proceed with that. Embedded Tomcat or Netty sound interesting, too. And from a very quick look it seems that there are packages available for Debian stable for all three of them. I'm not opposed to the idea.

What other advantages are there over using the installed Tomcat than easier deployment? Are they worth the effort? And is the performance as good or maybe even better than with the installed Tomcat?

comment:2 Changed 5 years ago by iwakeh

Well, easier deployment and an easier configuration, of course.

The performance ought to be fine for all, Jetty was used all along for Hudson/Jenkins installations,
and I could witness that this setup could handle many, many projects and users.
Embedded tomcat is used by SonarQube. Projects like Solr think about switching to Jetty or Netty.

The performance of the older versions available in wheezy should be fine, too,
but of course newer ones will be better. (Btw, wheezy only contains older Sonar, not SonarQube.)

It's sort of state-of-the-art to provide the all in one jar for an application without server
configuration/deployment. Is it a goal for the Onionoo project to have many servers running?
If so, ease of configuration and deployment should be a high priority.

comment:3 in reply to:  2 ; Changed 5 years ago by karsten

Replying to iwakeh:

Well, easier deployment and an easier configuration, of course.

Okay.

The performance ought to be fine for all, Jetty was used all along for Hudson/Jenkins installations,
and I could witness that this setup could handle many, many projects and users.
Embedded tomcat is used by SonarQube. Projects like Solr think about switching to Jetty or Netty.

The performance of the older versions available in wheezy should be fine, too,
but of course newer ones will be better. (Btw, wheezy only contains older Sonar, not SonarQube.)

Okay.

It's sort of state-of-the-art to provide the all in one jar for an application without server
configuration/deployment.

We could provide a jar that, by default, listens on 8080. People could then decide to put it behind an Apache or make it listen on port 80 directly. We cannot do the latter, because running a service on a Tor machine must not depend on being root, not even when it drops privileges after opening the socket. Plausible?

Is it a goal for the Onionoo project to have many servers running?
If so, ease of configuration and deployment should be a high priority.

I surely wouldn't mind other people to run their own instances of Onionoo. For example, I'd love to see you running one and have Onionoo clients list that as fallback alternative if onionoo.torproject.org is down. Though I'm not sure if this ticket is the main barrier, because the real trick is to initialize a new Onionoo instance with years of Tor network archives. But that shouldn't stop us here. Let's make it easier to configure and deploy Onionoo, regardless of how many instances there will be.

Which of the three (embedded Tomcat, Jetty, Netty) should we try out, and why? And would you be willing to submit a patch?

comment:4 in reply to:  3 ; Changed 5 years ago by iwakeh

Replying to karsten:

I surely wouldn't mind other people to run their own instances of Onionoo. For example, I'd love to see you running one and have Onionoo clients list that as fallback alternative if onionoo.torproject.org is down.

Running my own Onionoo server? Hmm, never occured to me ...
The only big hindrance is that I don't have the means for hosting it.
But, maybe I can find some free hosting.

Though I'm not sure if this ticket is the main barrier, because the real trick is to initialize a new Onionoo instance with years of Tor network archives. But that shouldn't stop us here.

How many GB/TB approximately?

... Let's make it easier to configure and deploy Onionoo, regardless of how many instances there will be.

Yeah, that'll be helpful for testing, too.

Which of the three (embedded Tomcat, Jetty, Netty) should we try out, and why? And would you be willing to submit a patch?

Sure, I'd like to submit a patch.
I would aim at making the change independent of the actual choice of server, well, as far as possible.
Switching the (embedded or not) server should always be possible, just in case something
better comes along ...

I mentioned testing above. Do you have any kind of benchmark for Onionoo?

Criteria for choosing the embedded server could be

  • easy integration (i.e. none or hardly any coding overhead)
  • performance (according the to-be-determined benchmark)
  • ease of configuration

comment:5 in reply to:  4 Changed 5 years ago by karsten

Replying to iwakeh:

Replying to karsten:

I surely wouldn't mind other people to run their own instances of Onionoo. For example, I'd love to see you running one and have Onionoo clients list that as fallback alternative if onionoo.torproject.org is down.

Running my own Onionoo server? Hmm, never occured to me ...
The only big hindrance is that I don't have the means for hosting it.
But, maybe I can find some free hosting.

Okay. Maybe we can figure something out.

Though I'm not sure if this ticket is the main barrier, because the real trick is to initialize a new Onionoo instance with years of Tor network archives. But that shouldn't stop us here.

How many GB/TB approximately?

The current system uses 33G of its 242G, but you'll need even more for initialization. 500G maybe?

... Let's make it easier to configure and deploy Onionoo, regardless of how many instances there will be.

Yeah, that'll be helpful for testing, too.

Which of the three (embedded Tomcat, Jetty, Netty) should we try out, and why? And would you be willing to submit a patch?

Sure, I'd like to submit a patch.
I would aim at making the change independent of the actual choice of server, well, as far as possible.
Switching the (embedded or not) server should always be possible, just in case something
better comes along ...

Sure, the fewer changes are necessary to switch back or to a different embedded server solution, the better.

I mentioned testing above. Do you have any kind of benchmark for Onionoo?

Criteria for choosing the embedded server could be

  • easy integration (i.e. none or hardly any coding overhead)
  • performance (according the to-be-determined benchmark)
  • ease of configuration

There's no benchmark yet, but we might be able to come up with something. I could try to extract distributions of requested document types and parameters from current access logs, and then we measure how many requests each of the variants can handle in a given amount of time.

Is there a tool for this, or are we building the tool while performing the measurements?

comment:6 Changed 5 years ago by iwakeh

There's no benchmark yet, but we might be able to come up with something. I could try to extract distributions of requested document types and parameters from current access logs, and then we measure how many requests each of the variants can handle in a given amount of time.

Is there a tool for this, or are we building the tool while performing the measurements?

What first comes to mind is Apache's JMeter, which is available in wheezy stable.
As there is no web-gui in Onionoo, writing the test plans should be strait-forward using the statistics you intend to retrieve from the access logs.

comment:7 Changed 5 years ago by karsten

Priority: minormajor

Let's ignore performance measurements for now and assume that embedded Tomcat will be fast enough.

The reason I'm pushing this is that I just broke the Onionoo service for over an hour. I *think* the problem was that the Tomcat on the production system still uses Java 6 and didn't like my Java 7 .war. And it didn't like it to the extent that it didn't even let me deploy an older .war compiled for Java 6. I had to ask sysadmins to delete the extracted .war and restart Tomcat. I'm now feeling uncomfortable deploying new Onionoo versions on the server. Let's try to get rid of Tomcat really soon if we can, so that we have more control over the service, even without being root. (The alternative is to uninstall Java 7 and switch back to Java 6 on the production system, but that seems backward.)

Speaking of, I just discussed the embedded Tomcat idea with our sysadmin, and they'd like us to run the two java processes as two different users for added security; the hourly cronjob would have permissions to write files to /srv/onionoo.torproject.org/onionoo/, and the serving process would only have read permissions to those files.

I'm going to start the switch to embedded Tomcat tomorrow, unless you already have a patch or partial patch.

comment:8 in reply to:  7 Changed 5 years ago by iwakeh

Such incidents with the productive system are annoying.
But there is no need to hurry. I looked at the tomcat startup script
/etc/init/tomcat6 and I think it's the culprit.

I'm not a shell hacker, but to me it looks like java 7 cannot win there,
unless it is the only java on the system.

Maybe the admins could hard code jdk 7 for tomcat startup?
Or fix the script?
Or, uninstall java 6 like with Vagrant setup?

(I began looking at the embedded jetty8 so far; not really a patch yet, but
will hardly find time this week to continue there.)

Last edited 5 years ago by iwakeh (previous) (diff)

comment:9 Changed 5 years ago by karsten

You're right, let's not rush this. I'm talking to sysadmins today. Maybe we can fix this problem with installed Tomcat before switching to something else. Uninstalling Java 6 and editing the startup script (in /etc/default/tomcat6) both sound like promising approaches.

comment:10 Changed 5 years ago by karsten

Priority: majorminor

And it's fixed. Java 6 is now uninstalled, and Tomcat seems to work just fine with Java 7. Phew. Changing priority back to minor.

comment:11 Changed 5 years ago by iwakeh

A first embedded jetty and an all-in-one jar for the backend.

The command lines (prerequisite is having the jar/war in the current directory):

  • java -Xmx4096M -jar onionoo-1.1.1.jar starts the backend.
  • java -Xmx4096M -jar dist/onionoo-1.1.1.war starts the web server (port 8080).

(Memory settings should be adjusted appropriately.)

The jetty configuration is etc/onionoo-jetty.xml.

ant releasejar and ant war will create versioned archives in directory 'dist'.

Please review the attached patch.
(No hurry; I won't have much time during the next days anyway.)

Changed 5 years ago by iwakeh

embedded jetty, some ant task, some logging

comment:12 Changed 5 years ago by iwakeh

PS: The funny versions of all the jetty8 jars are provided by wheezy package 'jetty8'.

comment:13 Changed 5 years ago by karsten

Status: newneeds_revision

Neat! Thanks for writing and submitting this patch!

I made some tweaks in my task-13089 branch that are supposed to be squashed with your commit.

Here are some more comments for things I didn't fix/change yet:

  • Your first command line for the backend above should be java -Xmx4096M -jar dist/onionoo-1.1.1.jar.
  • I didn't manage to start the server with your second command line. Logs contain this line: "Web application not found onionoo-1.1.1.war". Hmm. Maybe we're not supposed to create a .war for the front-end, but also a .jar?!...
  • Do you mind updating vagrant/bootstrap.sh by removing Tomcat specifics and installing libjetty8-java? Anything else that needs changing?
  • Would you also want to update INSTALL by replacing the last section with new instructions?
  • It seems that the run Ant target is not required anymore, and bin/update.sh needs updating, too.
  • Are really all jetty8-*.jar files required? Did you try taking out some of them?
  • I wonder if copying logback.xml can be avoided in build.xml. Or if possible, is it possible to avoid copying it to directories which otherwise contain files under version control (etc/)?
  • Can you explain why you switched from <lib> to <zipgroupfileset> and from <classes> to <fileset>? I assume this is related to all-in-one jars?
  • Should we exclude test classes from generated .jar and .war files? I think I found test classes in both of them.
  • Should both back-end and front-end really log to the same files, or should we separate those?

Thanks, again!

comment:14 in reply to:  13 ; Changed 5 years ago by iwakeh

Thanks, for all your suggestions!

I made some tweaks in my task-13089 branch that are supposed to be squashed with your commit.

This branch looks fine. I added two patches based on this branch.

Here are some more comments for things I didn't fix/change yet:

  • Your first command line for the backend above should be java -Xmx4096M -jar dist/onionoo-1.1.1.jar.

This is correct; I changed it in the INSTALL suggestions.

  • I didn't manage to start the server with your second command line. Logs contain this line: "Web application not found onionoo-1.1.1.war". Hmm. Maybe we're not supposed to create a .war for the front-end, but also a .jar?!...

We don't need a jar here. I changed jetty.xml in such a way that the war
can be started and (this wasn't the case initially) the starting is independent of the actual path, name, and version of the war.

  • Do you mind updating vagrant/bootstrap.sh by removing Tomcat specifics and installing libjetty8-java? Anything else that needs changing?

The package is named 'jetty8' here.

I think, this is not only an update. With the embedded-one-jar-one-war setting
we are free to choose a new setup.
It might also be a good idea (with the upcoming mirrors in mind)
to make path setting for 'in', 'out', etc. configurable? => new task?
Should the vagrant-setup first of all reflect the main-onionoo server
or a good test setup or a general setup?
Maybe, the documentation should be extended to describe the possible options.
Another new task?

  • Would you also want to update INSTALL by replacing the last section with new instructions?

Patch with suggestions is attached.

  • It seems that the run Ant target is not required anymore, and bin/update.sh needs updating, too.

I removed the 'run' task. And I think update.sh could be removed, too? You decide.

  • Are really all jetty8-*.jar files required? Did you try taking out some of them?

I think so, as I only added them incrementally; but no guarantee ;-)

  • I wonder if copying logback.xml can be avoided in build.xml. Or if possible, is it possible to avoid copying it to directories which otherwise contain files under version control (etc/)?

Removed the copying.

  • Can you explain why you switched from <lib> to <zipgroupfileset> and from <classes> to <fileset>? I assume this is related to all-in-one jars?

Yes, it's the classpath: we need some classes/jars in the initial startup
(e.g. jetty-stuff and logging) and inside the web-app. Adding them to the
initial classpath seemed the best way to have all classes and dependencies in one place.
As we have an embedded server and only one web-app, this should be ok.

  • Should we exclude test classes from generated .jar and .war files? I think I found test classes in both of them.

I only found some junit test classes in the war and added the necessary exclusions.

  • Should both back-end and front-end really log to the same files, or should we separate those?

Of course, they shouldn't ;-) Initially I thought they would be started in different folders and hence have separate logs.
Now, I made the logbase path configurable on the command line (cf. INSTALL patch).

Changed 5 years ago by iwakeh

based on ​your task-13089 branch

Changed 5 years ago by iwakeh

based on your ​task-13089 branch

comment:15 in reply to:  14 Changed 5 years ago by karsten

Replying to iwakeh:

Thanks, for all your suggestions!

I made some tweaks in my task-13089 branch that are supposed to be squashed with your commit.

This branch looks fine. I added two patches based on this branch.

Cool! Added them to my branch and added two more. (Weren't you planning to create a Git repository somewhere?)

Here are some more comments for things I didn't fix/change yet:

  • Your first command line for the backend above should be java -Xmx4096M -jar dist/onionoo-1.1.1.jar.

This is correct; I changed it in the INSTALL suggestions.

Okay.

  • I didn't manage to start the server with your second command line. Logs contain this line: "Web application not found onionoo-1.1.1.war". Hmm. Maybe we're not supposed to create a .war for the front-end, but also a .jar?!...

We don't need a jar here. I changed jetty.xml in such a way that the war
can be started and (this wasn't the case initially) the starting is independent of the actual path, name, and version of the war.

Okay. (I didn't test this, but decided to give it another try when setting up the Onionoo mirror.)

  • Do you mind updating vagrant/bootstrap.sh by removing Tomcat specifics and installing libjetty8-java? Anything else that needs changing?

The package is named 'jetty8' here.

IIRC, that package attempts to start an actual server, which we don't need, because we're starting our own server. I could be wrong though. But in theory, the fewer/smaller packages we need, the better.

I think, this is not only an update. With the embedded-one-jar-one-war setting
we are free to choose a new setup.
It might also be a good idea (with the upcoming mirrors in mind)
to make path setting for 'in', 'out', etc. configurable? => new task?
Should the vagrant-setup first of all reflect the main-onionoo server
or a good test setup or a general setup?
Maybe, the documentation should be extended to describe the possible options.
Another new task?

Fine questions. How about we update the Vagrant setup in this branch to be as similar to the Jetty-based production environment as possible, regardless of the setup on the main Onionoo instance? I'd like us to try out Jetty on the mirrors, and if that goes well, switch the main instance to Jetty, too.

  • Would you also want to update INSTALL by replacing the last section with new instructions?

Patch with suggestions is attached.

Thanks, applied to my branch.

  • It seems that the run Ant target is not required anymore, and bin/update.sh needs updating, too.

I removed the 'run' task. And I think update.sh could be removed, too? You decide.

Sure, I'm also fine with removing it.

  • Are really all jetty8-*.jar files required? Did you try taking out some of them?

I think so, as I only added them incrementally; but no guarantee ;-)

Okay.

  • I wonder if copying logback.xml can be avoided in build.xml. Or if possible, is it possible to avoid copying it to directories which otherwise contain files under version control (etc/)?

Removed the copying.

Thanks.

  • Can you explain why you switched from <lib> to <zipgroupfileset> and from <classes> to <fileset>? I assume this is related to all-in-one jars?

Yes, it's the classpath: we need some classes/jars in the initial startup
(e.g. jetty-stuff and logging) and inside the web-app. Adding them to the
initial classpath seemed the best way to have all classes and dependencies in one place.
As we have an embedded server and only one web-app, this should be ok.

Okay.

  • Should we exclude test classes from generated .jar and .war files? I think I found test classes in both of them.

I only found some junit test classes in the war and added the necessary exclusions.

Thanks.

  • Should both back-end and front-end really log to the same files, or should we separate those?

Of course, they shouldn't ;-) Initially I thought they would be started in different folders and hence have separate logs.
Now, I made the logbase path configurable on the command line (cf. INSTALL patch).

Looks good. Should we also suggest paths for running/logging back-end and front-end?

I might have more feedback when trying to deploy this branch on a mirror. Similarly, maybe you'll have more tweaks when trying the same. I'd say let's merge when both of us successfully got this running.

Thanks!

comment:16 Changed 5 years ago by karsten

Status: needs_revisionneeds_review

Okay, I got the Jetty branch successfully running on a mirror. I made more tweaks in my ​task-13089 branch.

Things to do before merging into master:

  • Agree that today's commits should be merged, and if not, tweak them.
  • Decide whether to update the Vagrant setup to make it as similar to the Jetty-based production environment as possible, and then update it.
  • Decide whether we should suggest logging paths in INSTALL or not.
  • Consider configuring a request log.
  • Ideally squash all changes since branching off master into a single commit with a nice new commit message.

comment:17 Changed 5 years ago by karsten

I cleaned up my task-13089 branch a bit (rebased to current master, extracted parts unrelated to Jetty switch to their own commit, squashed remaining commits, reworded commit message, etc.). Please review my task-13089-2 branch. I'm running it on my Onionoo mirror now to give it some testing. If you like it and it doesn't break horribly in the next few days, I'd like to merge it to master and deploy it on the official Onionoo, too. Thoughts?

comment:18 Changed 5 years ago by iwakeh

I agree, things look fine and we should just switch to jetty, assuming the test phase on the mirror is ok.

Some minor things:

  • 'bootstrap.sh' still contains the obsolete war linking:
    ln -s /srv/onionoo.torproject.org/onionoo/onionoo.war \
     /var/lib/tomcat6/webapps/onionoo.war
    
  • What about 'update.sh'? It doesn't seem necessary anymore.
  • In 'CONTRIB.md' the paragraph Vagrant-based development environment (line 44 ff) needs adaption. I'd suggest removing lines 90 to 114.

Do you intend to perform a minor stress test just using repeated wget calls?
Or, does the mirror receive some significant traffic already?

comment:19 in reply to:  18 Changed 5 years ago by karsten

Replying to iwakeh:

I agree, things look fine and we should just switch to jetty, assuming the test phase on the mirror is ok.

Some minor things:

  • 'bootstrap.sh' still contains the obsolete war linking:
    ln -s /srv/onionoo.torproject.org/onionoo/onionoo.war \
     /var/lib/tomcat6/webapps/onionoo.war
    

Good catch. Removed.

  • What about 'update.sh'? It doesn't seem necessary anymore.

That one's already gone.

  • In 'CONTRIB.md' the paragraph Vagrant-based development environment (line 44 ff) needs adaption. I'd suggest removing lines 90 to 114.

Removed.

See my branch task-13089-2 for the changes and task-13089-3 for a newly rebased branch with a single squashed commit. I plan to merge task-13089-3 into master tomorrow.

Do you intend to perform a minor stress test just using repeated wget calls?
Or, does the mirror receive some significant traffic already?

I just put the Jetty version online on the main Onionoo. It didn't break so far and switching back is easy. I'll take a quick look at the logs tomorrow. If all looks good, I'll switch permanently.

Changed 5 years ago by karsten

Attachment: onionoo-jetty-switch.png added

Performance statistics before/after switching from Tomcat to Jetty

comment:20 Changed 5 years ago by karsten

Here are some usage/performance statistics:

Number of log entries per day indicating that processing took longer than expected (* means numbers are extrapolated from partial days):

2014-11-02 14501 Tomcat
2014-11-03 11798 Tomcat
2014-11-04 9734 Tomcat
2014-11-05 10348 Tomcat
2014-11-06 11574 Tomcat
2014-11-07 10442 Tomcat
2014-11-08 9775 Tomcat
2014-11-09 11225 Tomcat
2014-11-10 12652 Tomcat
2014-11-11 11973 Tomcat
2014-11-12* 10779 Tomcat
2014-11-12* 59420 Jetty
2014-11-13* 48179 Jetty

So, five times as many requests taking longer than expected with Jetty as with Tomcat. Not the end of the world, but worth looking into.

See the attached graph with some usage/performance statistics. Some remarks:

  • The vertical line is when we switched from Tomcat to Jetty.
  • All graphs besides the middle two (time_build_resp and time_handle_req) show service usage, not performance. They basically show that usage hasn't changed after switching to Jetty. So far so good.
  • The time_handle_req plot looks okay. At some point we'll want to investigate those spikes of the 0.999 line. But that's not an issue right now. Let's ignore that.
  • The time_build_resp plot is what we should be worried about. The 0.99 line shouldn't go up that much. This tells us that Jetty is systematically slower in building responses than Tomcat. This is something we should fix.

Here's Tomcat's server.xml. Maybe there's something we can tweak in the Jetty setup?

<Server port="8005" shutdown="SHUTDOWN">
  <Service name="Catalina">
    <Connector port="8080" maxHttpHeaderSize="8192"
               maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
               enableLookups="false" redirectPort="8443" acceptCount="100"
               connectionTimeout="20000" disableUploadTimeout="true"
               compression="on" compressionMinSize="2048"
               noCompressionUserAgents="gozilla, traviata"
               compressableMimeType="text/html,text/xml,text/plain,application/json" />
    <Engine name="Catalina" defaultHost="sewerzowi.torproject.org">
      <Host name="ooni.torproject.org" appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">
        <Alias>sewerzowi.torproject.org</Alias>
        <Valve className="org.apache.catalina.valves.AccessLogValve"
               directory="logs" prefix="ooni_access_log." suffix=".txt"
               pattern="%l %u %t %r %s %b" resolveHosts="false"/>
      </Host>
    </Engine>
  </Service>
</Server>

comment:21 Changed 5 years ago by iwakeh

Which 'jetty.xml' is currently used?

comment:22 Changed 5 years ago by karsten

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN"
  "http://www.eclipse.org/jetty/configure.dtd">

<Configure id="server" class="org.eclipse.jetty.server.Server" >
  <Set name="dumpAfterStart">false</Set>

  <Set name="threadPool">
    <New class="org.eclipse.jetty.util.thread.QueuedThreadPool">
      <Set name="minThreads">10</Set>
      <Set name="maxThreads">250</Set>
      <Set name="detailedDump">false</Set>
    </New>
  </Set>

  <New id="webAppContext" class="org.eclipse.jetty.webapp.WebAppContext">
    <Set name="logUrlOnStart">true</Set>
    <Set name="war">
      <Call class="java.lang.System" name="getProperty">
        <Arg>java.class.path</Arg>
      </Call>
    </Set>
  </New>

  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
        <Set name="port">8080</Set>
      </New>
    </Arg>
  </Call>

  <Set name="handler">
    <!-- maybe add more handlers (statistics, logging, etc.) later -->
    <New class="org.eclipse.jetty.server.handler.ContextHandlerCollection">
      <Call name="addHandler">
        <Arg>
          <Ref id="webAppContext"/>
        </Arg>
      </Call>
    </New>
  </Set>

</Configure>

(I switched back to Tomcat a few hours ago in the context of deploying #13135, but I can rebase and re-deploy this branch at any time.)

comment:23 Changed 5 years ago by iwakeh

Comparing the two configurations I noticed that jetty is lacking compression.
A patch is attached.
I also increased the min-thread-number.

Maybe the reduced delivery time for compressed content will enable quicker processing.

In order to really measure before having code deployed productively, I'll set #13616 to a higher priority.

Changed 5 years ago by iwakeh

jetty.xml with compression

comment:24 Changed 5 years ago by karsten

Oh hey. I noticed that compression was missing in the Jetty setup, but didn't think that this could have any (negative) effect on performance. But this is plausible, because we're taking the last timestamp after flushing the connection, and if we're limited by client bandwidth then this could show up as higher response times.

I applied your patch, squashed it into your earlier patch, rebased that to master, updated the protocol version in build.xml to 2.0, pushed it to my repository as task-13089-4 branch, and deployed that on the server. I'll wait a few hours before making a new performance graph. I'm optimistic.

As always, thanks!

Changed 5 years ago by karsten

Attachment: onionoo-jetty-switch.2.png added

Performance statistics before/after switching from Tomcat to Jetty, updated

comment:25 Changed 5 years ago by karsten

Here's the updated graph. Looks like it didn't work. I also checked whether compression works using curl, and I don't get a compressed response. Any idea how I can further investigate why compression doesn't work?

comment:26 in reply to:  25 Changed 5 years ago by iwakeh

Indeed, the server compression configuration does not work.
I couldn't identify the reason yet.

So, I tried web-app compression configuration. And, it works.
(I use 'w3m -dump_extra ...' for testing this)

Patch attached. It concerns build.xml and etc/web.xml.template)
The jetty.xml changes can be removed or kept.

Changed 5 years ago by iwakeh

web-app compression config based on task-13089-4

comment:27 Changed 5 years ago by karsten

Great, I just deployed that on the server and will post another graph in a couple of hours. Thanks!

Changed 5 years ago by karsten

Attachment: onionoo-jetty-switch.3.png added

Performance statistics before/after switching from Tomcat to Jetty, updated once more

comment:28 Changed 5 years ago by karsten

Resolution: implemented
Status: needs_reviewclosed

Yay, looks better now. See the attached graph with four vertical lines:

  1. November 12: switch from Tomcat to Jetty
  2. November 15: switch back to Tomcat
  3. November 16: switch to Jetty again
  4. November 17: switch to Jetty with working compression

The relevant part is that the 0.99 and 0.999 percentiles of time_build_resp go down to values before November 12. Great!

I went ahead and merged the patch into master. That concludes this ticket. Thanks, much appreciated! Closing.

Note: See TracTickets for help on using tickets.