Opened 5 years ago

Closed 16 months ago

#15719 closed enhancement (wontfix)

Handle scaling issues with OONI pipeline

Reported by: hellais Owned by: hellais
Priority: Medium Milestone:
Component: Archived/Ooni Version:
Severity: Normal Keywords: archived-closed-2018-07-04
Cc: joelanders@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

It looks like the new machine I acquired for hosting the OONI pipeline just 6 months ago has now run out of disk space.

585G	mongodb
68G	pipeline

It appears that since we have started compressing the reports now the majority of the disk usage is sucked up by mongodb.

I think this is a good opportunity to consider the migration over to elasticsearch as was suggested by @robertkeizer:
https://github.com/TheTorProject/ooni-pipeline/commit/c20ad24e96736eb8dbebda6b23fe6ccde944b066#commitcomment-10450852

Support for elastic search in ooni-pipeline has been implemented inside of https://github.com/TheTorProject/ooni-pipeline/compare/feature/elasticsearch.

Once that is merged we need to consider the possibility of starting building an elastic search cluster for all the data.

I propose we use this venue to discuss.

Child Tickets

Change History (3)

comment:1 Changed 4 years ago by joelanders

Cc: joelanders@… added

cc'ing myself for Elasticsearch related things.

comment:2 Changed 23 months ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

comment:3 Changed 16 months ago by teor

Keywords: archived-closed-2018-07-04 added
Resolution: wontfix
Status: newclosed

Close all tickets in archived components

Note: See TracTickets for help on using tickets.