Handle scaling issues with OONI pipeline
It looks like the new machine I acquired for hosting the OONI pipeline just 6 months ago has now run out of disk space.
585G mongodb
68G pipeline
It appears that since we have started compressing the reports now the majority of the disk usage is sucked up by mongodb.
I think this is a good opportunity to consider the migration over to elasticsearch as was suggested by @robertkeizer: https://github.com/TheTorProject/ooni-pipeline/commit/c20ad24e96736eb8dbebda6b23fe6ccde944b066#commitcomment-10450852
Support for elastic search in ooni-pipeline has been implemented inside of https://github.com/TheTorProject/ooni-pipeline/compare/feature/elasticsearch.
Once that is merged we need to consider the possibility of starting building an elastic search cluster for all the data.
I propose we use this venue to discuss.