Opened 5 years ago

Closed 15 months ago

#13720 closed enhancement (wontfix)

Investigate possible performance improvements to the ooni-pipeline

Reported by: hellais Owned by: hellais
Priority: Medium Milestone:
Component: Archived/Ooni Version:
Severity: Normal Keywords: archived-closed-2018-07-04
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Currently the ooni-pipeline is *very* slow to process the collected data. It takes about 3-4 days to process 16 GB of data.

Here is some profiling information for the import task:

         55705786497 function calls (55522495904 primitive calls) in 150444.996 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 <string>:1(ParseResult)
        1    0.000    0.000    0.000    0.000 <string>:1(SplitResult)
        9    0.000    0.000    0.000    0.000 UserDict.py:18(__getitem__)
        1    0.000    0.000    0.000    0.000 UserDict.py:4(__init__)
        9    0.000    0.000    0.000    0.000 UserDict.py:58(get)
        9    0.000    0.000    0.000    0.000 UserDict.py:70(__contains__)
        1    0.000    0.000    0.000    0.000 __future__.py:48(<module>)
        1    0.000    0.000    0.000    0.000 __future__.py:74(_Feature)
        7    0.000    0.000    0.000    0.000 __future__.py:75(__init__)
        4    0.101    0.025    2.397    0.599 __init__.py:1(<module>)
        3    0.000    0.000    0.000    0.000 __init__.py:104(CFunctionType)
        1    0.000    0.000    0.000    0.000 __init__.py:1093(Logger)
        1    0.000    0.000    0.000    0.000 __init__.py:1108(__init__)
        1    0.000    0.000    0.000    0.000 __init__.py:1368(RootLogger)
        1    0.000    0.000    0.000    0.000 __init__.py:1374(__init__)
        1    0.000    0.000    0.000    0.000 __init__.py:1382(LoggerAdapter)
  1050904   10.463    0.000   23.302    0.000 __init__.py:142(run)
       14    0.000    0.000    0.000    0.000 __init__.py:147(_check_size)
        1    0.082    0.082    0.480    0.480 __init__.py:15(<module>)
        1    0.000    0.000    0.000    0.000 __init__.py:159(py_object)
        1    0.019    0.019    0.207    0.207 __init__.py:16(<module>)
        1    0.000    0.000    0.000    0.000 __init__.py:1670(NullHandler)
        1    0.000    0.000    0.000    0.000 __init__.py:168(c_short)
  1050904    3.528    0.000   26.829    0.000 __init__.py:169(run_process)
        1    0.000    0.000    0.000    0.000 __init__.py:172(c_ushort)
        1    0.000    0.000    0.000    0.000 __init__.py:176(c_long)
        1    0.000    0.000 150443.669 150443.669 __init__.py:18(run)
        1    0.000    0.000    0.000    0.000 __init__.py:180(c_ulong)
        1    0.000    0.000    0.000    0.000 __init__.py:182(_checkLevel)
        1    0.000    0.000    0.000    0.000 __init__.py:189(c_int)
        1    0.000    0.000    0.000    0.000 __init__.py:193(c_uint)
        1    0.000    0.000    0.000    0.000 __init__.py:197(c_float)
        1    0.079    0.079    0.181    0.181 __init__.py:2(<module>)
        1    0.000    0.000    0.000    0.000 __init__.py:201(c_double)
        1    0.000    0.000    0.000    0.000 __init__.py:205(c_longdouble)
        1    0.000    0.000    0.000    0.000 __init__.py:226(c_ubyte)
        1    0.000    0.000    0.000    0.000 __init__.py:230(LogRecord)
        1    0.000    0.000    0.000    0.000 __init__.py:233(c_byte)
        1    0.000    0.000    0.000    0.000 __init__.py:238(c_char)
        1    0.001    0.001    0.001    0.001 __init__.py:24(<module>)
        1    0.000    0.000    0.000    0.000 __init__.py:243(c_char_p)
        1    0.000    0.000    0.011    0.011 __init__.py:245(load)
        1    0.000    0.000    0.000    0.000 __init__.py:255(c_void_p)
        1    0.000    0.000    0.000    0.000 __init__.py:260(c_bool)
        1    0.000    0.000    0.000    0.000 __init__.py:265(_reset_cache)
        1    0.000    0.000    0.000    0.000 __init__.py:276(YAMLObjectMetaclass)
        1    0.000    0.000    0.000    0.000 __init__.py:280(__init__)
        1    0.000    0.000    0.000    0.000 __init__.py:281(loads)
        1    0.000    0.000    0.000    0.000 __init__.py:286(YAMLObject)
        1    0.000    0.000    0.000    0.000 __init__.py:291(c_wchar_p)
        1    0.000    0.000    0.000    0.000 __init__.py:294(c_wchar)
        1    0.000    0.000    0.000    0.000 __init__.py:332(CDLL)
        1    0.000    0.000    0.000    0.000 __init__.py:346(Formatter)
        3    0.000    0.000    0.001    0.000 __init__.py:349(__init__)
        3    0.000    0.000    0.000    0.000 __init__.py:359(_FuncPtr)
        4    0.000    0.000    0.000    0.000 __init__.py:375(__getattr__)
        4    0.000    0.000    0.000    0.000 __init__.py:382(__getitem__)
        1    0.000    0.000    0.000    0.000 __init__.py:388(PyDLL)
        1    0.000    0.000    0.000    0.000 __init__.py:390(__init__)
        1    0.058    0.058    0.059    0.059 __init__.py:4(<module>)
        1    0.000    0.000    0.000    0.000 __init__.py:428(LibraryLoader)
        2    0.000    0.000    0.000    0.000 __init__.py:429(__init__)
        3    0.000    0.000    0.000    0.000 __init__.py:49(normalize_encoding)
        3    0.000    0.000    0.000    0.000 __init__.py:493(PYFUNCTYPE)
        1    0.000    0.000    0.000    0.000 __init__.py:494(BufferingFormatter)
        3    0.000    0.000    0.000    0.000 __init__.py:494(CFunctionType)
        1    0.000    0.000    0.000    0.000 __init__.py:536(Filter)
        1    0.000    0.000    0.000    0.000 __init__.py:566(BSON)
        1    0.000    0.000    0.000    0.000 __init__.py:573(Filterer)
        1    0.000    0.000    0.000    0.000 __init__.py:578(__init__)
        1    0.000    0.000    0.000    0.000 __init__.py:645(Handler)
        3    0.000    0.000    0.038    0.013 __init__.py:71(search_function)
  1061845    6.150    0.000 148475.239    0.140 __init__.py:75(load_all)
        3    0.000    0.000    0.000    0.000 __init__.py:78(CFUNCTYPE)
        1    0.000    0.000    0.000    0.000 __init__.py:809(StreamHandler)
        1    0.000    0.000    0.000    0.000 __init__.py:82(get_version_string)
        1    0.000    0.000    0.000    0.000 __init__.py:880(FileHandler)
        1    0.000    0.000    0.000    0.000 __init__.py:944(PlaceHolder)
     3753    0.008    0.000    0.008    0.000 __init__.py:95(safe_load_all)
        1    0.019    0.019    0.150    0.150 __init__.py:99(<module>)
        1    0.000    0.000    0.000    0.000 __init__.py:991(Manager)
        1    0.000    0.000    0.000    0.000 __init__.py:996(__init__)
        1    0.000    0.000    0.000    0.000 _endian.py:26(_swapped_meta)
        1    0.000    0.000    0.000    0.000 _endian.py:4(<module>)
        1    0.000    0.000    0.000    0.000 _endian.py:49(BigEndianStructure)
        1    0.000    0.000    0.000    0.000 ascii.py:13(Codec)
        1    0.000    0.000    0.000    0.000 ascii.py:20(IncrementalEncoder)
        1    0.000    0.000    0.000    0.000 ascii.py:24(IncrementalDecoder)
        1    0.000    0.000    0.000    0.000 ascii.py:28(StreamWriter)
        1    0.000    0.000    0.000    0.000 ascii.py:31(StreamReader)
        1    0.000    0.000    0.000    0.000 ascii.py:34(StreamConverter)
        1    0.000    0.000    0.000    0.000 ascii.py:41(getregentry)
        1    0.000    0.000    0.000    0.000 ascii.py:8(<module>)
        2    0.000    0.000    0.000    0.000 atexit.py:37(register)
        1    0.000    0.000    0.000    0.000 atexit.py:6(<module>)
        1    0.020    0.020    0.021    0.021 auth.py:15(<module>)
        1    0.000    0.000    0.001    0.001 base64.py:3(<module>)
   201782    0.486    0.000   15.786    0.000 base64.py:319(decodestring)
   201782    0.782    0.000   16.761    0.000 base64_codec.py:27(base64_decode)
        1    0.000    0.000    0.000    0.000 base64_codec.py:45(Codec)
        1    0.000    0.000    0.000    0.000 base64_codec.py:52(IncrementalEncoder)
        1    0.000    0.000    0.000    0.000 base64_codec.py:57(IncrementalDecoder)
        1    0.000    0.000    0.000    0.000 base64_codec.py:62(StreamWriter)
        1    0.000    0.000    0.000    0.000 base64_codec.py:65(StreamReader)
        1    0.000    0.000    0.000    0.000 base64_codec.py:70(getregentry)
        1    0.000    0.000    0.000    0.000 base64_codec.py:8(<module>)
        1    0.000    0.000    0.000    0.000 binary.py:107(Binary)
        1    0.004    0.004    0.174    0.174 binary.py:15(<module>)
        1    0.000    0.000    0.000    0.000 binary.py:173(UUIDLegacy)
        1    0.000    0.000    0.000    0.000 bisect.py:1(<module>)
        1    0.001    0.001    0.001    0.001 bulk.py:18(<module>)
        1    0.000    0.000    0.000    0.000 bulk.py:181(_Bulk)
        1    0.000    0.000    0.000    0.000 bulk.py:41(_Run)
        1    0.000    0.000    0.000    0.000 bulk.py:438(BulkUpsertOperation)
        1    0.000    0.000    0.000    0.000 bulk.py:475(BulkWriteOperation)
        1    0.000    0.000    0.000    0.000 bulk.py:530(BulkOperationBuilder)
        1    0.000    0.000    0.000    0.000 calendar.py:126(Calendar)
        1    0.000    0.000    0.000    0.000 calendar.py:132(__init__)
        1    0.000    0.000    0.000    0.000 calendar.py:138(setfirstweekday)
        1    0.000    0.000    0.000    0.000 calendar.py:21(IllegalMonthError)
        1    0.000    0.000    0.000    0.000 calendar.py:255(TextCalendar)
        1    0.000    0.000    0.000    0.000 calendar.py:28(IllegalWeekdayError)
        1    0.000    0.000    0.000    0.000 calendar.py:372(HTMLCalendar)
        1    0.000    0.000    0.000    0.000 calendar.py:47(_localized_month)
        1    0.000    0.000    0.000    0.000 calendar.py:484(TimeEncoding)
        1    0.000    0.000    0.000    0.000 calendar.py:496(LocaleTextCalendar)
        2    0.000    0.000    0.000    0.000 calendar.py:52(__init__)
        1    0.000    0.000    0.000    0.000 calendar.py:531(LocaleHTMLCalendar)
        1    0.000    0.000    0.000    0.000 calendar.py:6(<module>)
        1    0.000    0.000    0.000    0.000 calendar.py:66(_localized_day)
        2    0.000    0.000    0.000    0.000 calendar.py:71(__init__)
        1    0.000    0.000    0.000    0.000 code.py:16(<module>)
        1    0.000    0.000    0.000    0.000 code.py:18(Code)
        3    0.000    0.000    0.000    0.000 codecs.py:77(__new__)
        1    0.005    0.005    0.006    0.006 collection.py:15(<module>)
  1054657    1.238    0.000    1.238    0.000 collection.py:173(name)
  2109314    2.496    0.000    2.496    0.000 collection.py:182(database)
  1054657   40.496    0.000 1761.871    0.002 collection.py:291(insert)
  2109314   13.536    0.000   56.410    0.000 collection.py:378(gen)
        1    0.000    0.000    0.000    0.000 collection.py:47(Collection)
  1054657   40.971    0.000  139.474    0.000 collection.py:51(__init__)
        1    0.001    0.001    0.002    0.002 collections.py:1(<module>)
        1    0.000    0.000    0.000    0.000 collections.py:25(OrderedDict)
        2    0.001    0.001    0.002    0.001 collections.py:287(namedtuple)
       99    0.000    0.000    0.000    0.000 collections.py:328(<genexpr>)
       13    0.000    0.000    0.000    0.000 collections.py:352(<genexpr>)
       13    0.000    0.000    0.000    0.000 collections.py:354(<genexpr>)
        1    0.000    0.000    0.000    0.000 collections.py:386(Counter)
        1    0.000    0.000    0.000    0.000 command_cursor.py:15(<module>)
        1    0.000    0.000    0.000    0.000 command_cursor.py:23(CommandCursor)
        1    0.000    0.000    0.000    0.000 common.py:136(validate_positive_integer_or_none)
        1    0.020    0.020    0.337    0.337 common.py:16(<module>)
  1054658    7.056    0.000    7.056    0.000 common.py:166(validate_positive_float)
  1054658    2.463    0.000    2.463    0.000 common.py:193(validate_read_preference)
  1054658    6.361    0.000    9.398    0.000 common.py:206(validate_tag_sets)
  1054658    3.683    0.000    5.386    0.000 common.py:250(validate_uuid_subtype)
        1    0.000    0.000    0.000    0.000 common.py:328(WriteConcern)
  1054659    3.739    0.000    3.739    0.000 common.py:330(__init__)
        1    0.000    0.000    0.000    0.000 common.py:344(BaseObject)
  1054659   25.149    0.000   84.755    0.000 common.py:351(__init__)
  1054659   20.336    0.000   49.668    0.000 common.py:393(__set_options)
  1054658    1.464    0.000    1.464    0.000 common.py:431(__get_write_concern)
  1054659    1.705    0.000    1.705    0.000 common.py:493(__get_slave_okay)
  1054658    1.524    0.000    1.524    0.000 common.py:511(__get_read_pref)
  1054658    1.586    0.000    1.586    0.000 common.py:527(__get_acceptable_latency)
  1054658    1.740    0.000    1.740    0.000 common.py:552(__get_tag_sets)
  2109315    2.815    0.000    2.815    0.000 common.py:574(__get_uuid_subtype)
  2109315    2.463    0.000    2.463    0.000 common.py:589(__get_safe)
  1054657    9.417    0.000   16.567    0.000 common.py:678(_get_write_mode)
  1054657    2.630    0.000    3.654    0.000 common.py:692(pop1)
  2109318    4.867    0.000    6.786    0.000 common.py:73(validate_boolean)
        1    0.000    0.000    0.000    0.000 common.py:86(validate_integer)
        1    0.000    0.000    0.000    0.000 common.py:99(validate_positive_integer)
        1    0.000    0.000    0.000    0.000 composer.py:11(Composer)
10216397/1054657  416.899    0.000 144221.349    0.137 composer.py:117(compose_mapping_node)
     3753    0.006    0.000    0.006    0.000 composer.py:13(__init__)
  2119842    8.896    0.000  260.499    0.000 composer.py:16(check_node)
        1    0.000    0.000    0.000    0.000 composer.py:2(<module>)
  1058094    4.860    0.000 144797.659    0.137 composer.py:24(get_node)
  1058094    9.367    0.000 144789.151    0.137 composer.py:50(compose_document)
134369987/1058094 1414.987    0.000 144723.179    0.137 composer.py:63(compose_node)
        1    0.000    0.000    0.000    0.000 composer.py:8(ComposerError)
100165785  711.662    0.000 2152.531    0.000 composer.py:88(compose_scalar_node)
23707649/1017384  538.730    0.000 135636.380    0.133 composer.py:99(compose_sequence_node)
        1    0.002    0.002    0.060    0.060 connection.py:38(<module>)
        1    0.000    0.000    0.000    0.000 connection.py:43(Connection)
100165785  213.180    0.000  300.982    0.000 constructor.py:105(construct_scalar)
 23707649  140.701    0.000 1003.688    0.000 constructor.py:112(construct_sequence)
        1    0.000    0.000    0.000    0.000 constructor.py:12(ConstructorError)
 10216397  196.595    0.000 1755.491    0.000 constructor.py:120(construct_mapping)
       24    0.000    0.000    0.000    0.000 constructor.py:149(add_constructor)
        1    0.000    0.000    0.000    0.000 constructor.py:15(BaseConstructor)
        5    0.000    0.000    0.000    0.000 constructor.py:155(add_multi_constructor)
        1    0.000    0.000    0.009    0.009 constructor.py:161(SafeConstructor)
100165785  341.062    0.000  750.006    0.000 constructor.py:163(construct_scalar)
 10216397   98.158    0.000  142.798    0.000 constructor.py:170(flatten_mapping)
        1    0.001    0.001    0.011    0.011 constructor.py:2(<module>)
     3753    0.011    0.000    0.011    0.000 constructor.py:20(__init__)
 10216397   61.343    0.000 1968.865    0.000 constructor.py:205(construct_mapping)
  3256523    7.213    0.000   31.596    0.000 constructor.py:210(construct_yaml_null)
  3948671   16.697    0.000   51.955    0.000 constructor.py:223(construct_yaml_bool)
  4696782   47.762    0.000  101.156    0.000 constructor.py:227(construct_yaml_int)
  1061748    3.921    0.000  253.251    0.000 constructor.py:26(check_data)
   718607    8.202    0.000   15.718    0.000 constructor.py:260(construct_yaml_float)
   201782    3.823    0.000   22.734    0.000 constructor.py:284(construct_yaml_binary)
  1058094  153.870    0.000 148188.377    0.140 constructor.py:30(get_data)
   526316    1.936    0.000   13.874    0.000 constructor.py:377(construct_yaml_set)
 87343420  292.915    0.000 1133.273    0.000 constructor.py:383(construct_yaml_str)
47415298/33067942  112.558    0.000 1127.398    0.000 constructor.py:390(construct_yaml_seq)
19906478/16131125   52.538    0.000 2023.643    0.000 constructor.py:395(construct_yaml_map)
  1058092   62.355    0.000 3225.680    0.003 constructor.py:42(construct_document)
        1    0.000    0.000    0.000    0.000 constructor.py:467(Constructor)
        1    0.000    0.000    0.000    0.000 constructor.py:532(classobj)
134369985  797.700    0.000 2372.299    0.000 constructor.py:55(construct_object)
        1    0.000    0.000    0.000    0.000 cursor.py:15(<module>)
        1    0.000    0.000    0.000    0.000 cursor.py:40(_SocketManager)
        1    0.000    0.000    0.000    0.000 cursor.py:72(Cursor)
        1    0.000    0.000    0.000    0.000 cursor_manager.py:24(<module>)
        1    0.000    0.000    0.000    0.000 cursor_manager.py:29(CursorManager)
        1    0.000    0.000    0.000    0.000 cursor_manager.py:35(__init__)
        1    0.000    0.000    0.000    0.000 cursor_manager.py:57(BatchCursorManager)
        1    0.000    0.000    0.000    0.000 cyaml.py:14(CBaseLoader)
        1    0.002    0.002    0.002    0.002 cyaml.py:2(<module>)
        1    0.000    0.000    0.000    0.000 cyaml.py:21(CSafeLoader)
        1    0.000    0.000    0.000    0.000 cyaml.py:28(CLoader)
        1    0.000    0.000    0.000    0.000 cyaml.py:35(CBaseDumper)
        1    0.000    0.000    0.000    0.000 cyaml.py:52(CSafeDumper)
        1    0.000    0.000    0.000    0.000 cyaml.py:69(CDumper)
  1054657    1.713    0.000    1.713    0.000 database.py:122(connection)
  2109314    2.886    0.000    2.886    0.000 database.py:131(name)
        1    0.003    0.003    0.009    0.009 database.py:15(<module>)
  1054657    5.262    0.000  144.735    0.000 database.py:193(__getattr__)
  1054657    1.915    0.000    1.915    0.000 database.py:250(_apply_incoming_manipulators)
  1054657    1.595    0.000    1.595    0.000 database.py:255(_apply_incoming_copying_manipulators)
        1    0.000    0.000    0.000    0.000 database.py:34(_check_name)
        1    0.000    0.000    0.000    0.000 database.py:46(Database)
        1    0.000    0.000    0.000    0.000 database.py:50(__init__)
        1    0.000    0.000    0.000    0.000 database.py:974(SystemJS)
        1    0.001    0.001    0.001    0.001 dbref.py:15(<module>)
        1    0.000    0.000    0.000    0.000 dbref.py:22(DBRef)
        1    0.000    0.000    0.026    0.026 decoder.py:17(_floatconstants)
        1    0.098    0.098    0.128    0.128 decoder.py:2(<module>)
        1    0.000    0.000    0.000    0.000 decoder.py:273(JSONDecoder)
        1    0.000    0.000    0.000    0.000 decoder.py:303(__init__)
        1    0.000    0.000    0.000    0.000 decoder.py:360(decode)
        1    0.000    0.000    0.000    0.000 decoder.py:371(raw_decode)
        1    0.005    0.005    0.006    0.006 dumper.py:2(<module>)
        1    0.000    0.000    0.000    0.000 dumper.py:27(SafeDumper)
        1    0.000    0.000    0.000    0.000 dumper.py:45(Dumper)
        1    0.000    0.000    0.000    0.000 dumper.py:9(BaseDumper)
        1    0.000    0.000    0.000    0.000 emitter.py:14(EmitterError)
        1    0.000    0.000    0.000    0.000 emitter.py:17(ScalarAnalysis)
        1    0.000    0.000    0.000    0.000 emitter.py:31(Emitter)
        1    0.000    0.000    0.000    0.000 emitter.py:9(<module>)
        1    0.000    0.000    0.000    0.000 encoder.py:101(__init__)
        1    0.000    0.000    0.002    0.002 encoder.py:2(<module>)
        1    0.000    0.000    0.000    0.000 encoder.py:70(JSONEncoder)
        1    0.000    0.000    0.000    0.000 error.py:2(<module>)
        1    0.000    0.000    0.000    0.000 error.py:4(Mark)
        1    0.000    0.000    0.000    0.000 error.py:45(YAMLError)
        1    0.000    0.000    0.000    0.000 error.py:48(MarkedYAMLError)
570464357 1246.534    0.000 1246.534    0.000 error.py:6(__init__)
        1    0.000    0.000    0.000    0.000 errors.py:109(TimeoutError)
        1    0.000    0.000    0.000    0.000 errors.py:116(WTimeoutError)
        1    0.000    0.000    0.000    0.000 errors.py:127(DuplicateKeyError)
        1    0.000    0.000    0.000    0.000 errors.py:136(BulkWriteError)
        1    0.000    0.000    0.000    0.000 errors.py:146(InvalidOperation)
        2    0.032    0.016    0.242    0.121 errors.py:15(<module>)
        1    0.000    0.000    0.000    0.000 errors.py:151(InvalidName)
        1    0.000    0.000    0.000    0.000 errors.py:156(CollectionInvalid)
        1    0.000    0.000    0.000    0.000 errors.py:161(InvalidURI)
        1    0.000    0.000    0.000    0.000 errors.py:168(UnsupportedOption)
        1    0.000    0.000    0.000    0.000 errors.py:175(ExceededMaxWaiters)
        1    0.000    0.000    0.000    0.000 errors.py:18(BSONError)
        1    0.000    0.000    0.000    0.000 errors.py:184(DocumentTooLarge)
        1    0.000    0.000    0.000    0.000 errors.py:23(InvalidBSON)
        1    0.000    0.000    0.000    0.000 errors.py:25(PyMongoError)
        1    0.000    0.000    0.000    0.000 errors.py:28(InvalidStringData)
        1    0.000    0.000    0.000    0.000 errors.py:32(ConnectionFailure)
        1    0.000    0.000    0.000    0.000 errors.py:33(InvalidDocument)
        1    0.000    0.000    0.000    0.000 errors.py:37(AutoReconnect)
        1    0.000    0.000    0.000    0.000 errors.py:38(InvalidId)
        1    0.000    0.000    0.000    0.000 errors.py:52(ConfigurationError)
        1    0.000    0.000    0.000    0.000 errors.py:57(OperationFailure)
        1    0.000    0.000    0.000    0.000 errors.py:91(CursorNotFound)
        1    0.000    0.000    0.000    0.000 errors.py:99(ExecutionTimeout)
        1    0.000    0.000    0.000    0.000 events.py:15(NodeEvent)
   280154    0.498    0.000    0.498    0.000 events.py:16(__init__)
        1    0.000    0.000    0.000    0.000 events.py:21(CollectionStartEvent)
 33924046   96.749    0.000   96.749    0.000 events.py:22(__init__)
        1    0.000    0.000    0.000    0.000 events.py:31(CollectionEndEvent)
        1    0.000    0.000    0.000    0.000 events.py:36(StreamStartEvent)
     3753    0.009    0.000    0.009    0.000 events.py:37(__init__)
        1    0.000    0.000    0.000    0.000 events.py:4(<module>)
        1    0.000    0.000    0.000    0.000 events.py:4(Event)
        1    0.000    0.000    0.000    0.000 events.py:42(StreamEndEvent)
        1    0.000    0.000    0.000    0.000 events.py:45(DocumentStartEvent)
  1058094    2.579    0.000    2.579    0.000 events.py:46(__init__)
 33927700   54.305    0.000   54.305    0.000 events.py:5(__init__)
        1    0.000    0.000    0.000    0.000 events.py:54(DocumentEndEvent)
  1058092    2.340    0.000    2.340    0.000 events.py:55(__init__)
        1    0.000    0.000    0.000    0.000 events.py:61(AliasEvent)
        1    0.000    0.000    0.000    0.000 events.py:64(ScalarEvent)
100165785  303.055    0.000  303.055    0.000 events.py:65(__init__)
        1    0.000    0.000    0.000    0.000 events.py:75(SequenceStartEvent)
        1    0.000    0.000    0.000    0.000 events.py:78(SequenceEndEvent)
        1    0.000    0.000    0.000    0.000 events.py:81(MappingStartEvent)
        1    0.000    0.000    0.000    0.000 events.py:84(MappingEndEvent)
        1    0.002    0.002    0.004    0.004 export.py:1(<module>)
        1    0.000    0.000    0.000    0.000 fnmatch.py:11(<module>)
     3698    0.018    0.000    0.081    0.000 genericpath.py:15(exists)
     3753    0.013    0.000    0.044    0.000 genericpath.py:38(isdir)
        1    0.003    0.003    0.003    0.003 hashlib.py:55(<module>)
        6    0.000    0.000    0.000    0.000 hashlib.py:94(__get_openssl_constructor)
        1    0.001    0.001    0.001    0.001 heapq.py:31(<module>)
  1054561    5.390    0.000    6.572    0.000 helpers.py:122(_check_command_response)
        1    0.000    0.000    0.000    0.000 helpers.py:15(<module>)
  1054560    3.000    0.000    3.000    0.000 helpers.py:181(_check_write_command_response)
  1054561   14.289    0.000   27.218    0.000 helpers.py:76(_unpack_response)
        1    0.000    0.000    0.000    0.000 hex_codec.py:27(hex_decode)
        1    0.000    0.000    0.000    0.000 hex_codec.py:45(Codec)
        1    0.000    0.000    0.000    0.000 hex_codec.py:52(IncrementalEncoder)
        1    0.000    0.000    0.000    0.000 hex_codec.py:57(IncrementalDecoder)
        1    0.000    0.000    0.000    0.000 hex_codec.py:62(StreamWriter)
        1    0.000    0.000    0.000    0.000 hex_codec.py:65(StreamReader)
        1    0.000    0.000    0.000    0.000 hex_codec.py:70(getregentry)
        1    0.000    0.000    0.000    0.000 hex_codec.py:8(<module>)
        1    0.000    0.000    0.000    0.000 hmac.py:20(HMAC)
        1    0.001    0.001    0.001    0.001 hmac.py:4(<module>)
        1    0.000    0.000    0.000    0.000 keyword.py:11(<module>)
        1    0.000    0.000    0.000    0.000 loader.py:11(BaseLoader)
        1    0.042    0.042    0.094    0.094 loader.py:2(<module>)
        1    0.000    0.000    0.000    0.000 loader.py:21(SafeLoader)
     3753    0.076    0.000   27.455    0.007 loader.py:23(__init__)
        1    0.000    0.000    0.000    0.000 loader.py:31(Loader)
        1    0.000    0.000    0.000    0.000 max_key.py:16(<module>)
        1    0.000    0.000    0.000    0.000 max_key.py:19(MaxKey)
        1    0.001    0.001    0.002    0.002 measurements.py:1(<module>)
        1    0.000    0.000    0.000    0.000 measurements.py:139(Measurements)
        1    0.000    0.000    0.000    0.000 measurements.py:29(Measurement)
        1    0.000    0.000    0.000    0.000 member.py:100(is_arbiter)
        1    0.000    0.000    0.000    0.000 member.py:15(<module>)
        1    0.000    0.000    0.000    0.000 member.py:29(Member)
        1    0.000    0.000    0.000    0.000 member.py:41(__init__)
        1    0.000    0.000    0.000    0.000 message.py:23(<module>)
        1    0.000    0.000    0.000    0.000 min_key.py:16(<module>)
        1    0.000    0.000    0.000    0.000 min_key.py:19(MinKey)
  1054560    6.873    0.000   40.662    0.000 mongo_client.py:1027(__check_response_to_last_error)
  1054560    2.840    0.000    3.915    0.000 mongo_client.py:1068(__check_bson_size)
  1054560   23.295    0.000 1405.804    0.001 mongo_client.py:1089(_send_message)
  2109122   11.540    0.000  611.302    0.000 mongo_client.py:1138(__receive_data_on_socket)
  1054561   11.762    0.000  627.035    0.001 mongo_client.py:1153(__receive_message_on_socket)
        1    0.000    0.000    0.000    0.000 mongo_client.py:1300(__getattr__)
        1    0.000    0.000    0.000    0.000 mongo_client.py:346(<lambda>)
        1    0.019    0.019    0.058    0.058 mongo_client.py:35(<module>)
        1    0.000    0.000    0.000    0.000 mongo_client.py:472(__create_pool)
  1054560    2.004    0.000    2.004    0.000 mongo_client.py:487(__check_auth)
  3163971    7.977    0.000   12.176    0.000 mongo_client.py:505(__member_property)
  1054560    1.402    0.000    1.402    0.000 mongo_client.py:593(auto_start_request)
  1054657    3.334    0.000    7.222    0.000 mongo_client.py:620(max_bson_size)
  1054657    3.732    0.000    8.287    0.000 mongo_client.py:652(max_wire_version)
  1054657    2.837    0.000    6.571    0.000 mongo_client.py:663(max_write_batch_size)
        1    0.000    0.000    0.097    0.097 mongo_client.py:675(__simple_command)
        1    0.000    0.000    0.098    0.098 mongo_client.py:696(__try_node)
  2109218   10.932    0.000   16.852    0.000 mongo_client.py:767(__ensure_member)
        1    0.000    0.000    0.098    0.098 mongo_client.py:830(__find_node)
        1    0.000    0.000    0.000    0.000 mongo_client.py:84(MongoClient)
  1054560   10.329    0.000   81.488    0.000 mongo_client.py:898(__socket)
        1    0.000    0.000    0.099    0.099 mongo_client.py:91(__init__)
  1054658    3.678    0.000   13.073    0.000 mongo_client.py:927(_ensure_connected)
        1    0.000    0.000    0.000    0.000 mongo_replica_set_client.py:131(RSState)
        1    0.000    0.000    0.000    0.000 mongo_replica_set_client.py:301(Monitor)
        1    0.001    0.001    0.001    0.001 mongo_replica_set_client.py:32(<module>)
        1    0.000    0.000    0.000    0.000 mongo_replica_set_client.py:371(MonitorThread)
        1    0.000    0.000    0.000    0.000 mongo_replica_set_client.py:428(MongoReplicaSetClient)
        1    0.000    0.000    0.000    0.000 nodes.py:2(<module>)
        1    0.000    0.000    0.000    0.000 nodes.py:2(Node)
        1    0.000    0.000    0.000    0.000 nodes.py:25(ScalarNode)
100165785  182.376    0.000  182.376    0.000 nodes.py:27(__init__)
        1    0.000    0.000    0.000    0.000 nodes.py:35(CollectionNode)
 33924046   62.153    0.000   62.153    0.000 nodes.py:36(__init__)
        1    0.000    0.000    0.000    0.000 nodes.py:44(SequenceNode)
        1    0.000    0.000    0.000    0.000 nodes.py:47(MappingNode)
  1054657   22.528    0.000   34.859    0.000 objectid.py:152(__generate)
        1    0.003    0.003    0.011    0.011 objectid.py:17(<module>)
        1    0.000    0.000    0.000    0.000 objectid.py:43(_machine_bytes)
        1    0.000    0.000    0.000    0.000 objectid.py:57(ObjectId)
  1054657    3.260    0.000   38.118    0.000 objectid.py:70(__init__)
        1    0.129    0.129 150444.996 150444.996 oonipipeline:2(<module>)
       44    0.000    0.000    0.005    0.000 os.py:136(makedirs)
     3654    0.011    0.000    0.060    0.000 os.py:159(removedirs)
     3654    0.070    0.000    0.619    0.000 os.py:181(renames)
        2    0.014    0.007    0.085    0.042 os.py:209(walk)
        1    0.000    0.000    0.000    0.000 os.py:35(_get_exports_list)
134089831  157.263    0.000  157.263    0.000 parser.py:107(peek_event)
170413970  233.594    0.000  284.814    0.000 parser.py:114(get_event)
     3753    0.027    0.000    0.107    0.000 parser.py:127(parse_stream_start)
     3753    0.021    0.000    0.818    0.000 parser.py:139(parse_implicit_document_start)
  1061748   19.890    0.000  231.621    0.000 parser.py:159(parse_document_start)
  1058092   10.453    0.000   51.220    0.000 parser.py:190(parse_document_end)
  1058094    5.111    0.000  463.971    0.000 parser.py:208(parse_document_content)
  1058094    7.048    0.000   20.697    0.000 parser.py:217(process_directives)
 52879541  151.073    0.000 5901.951    0.000 parser.py:264(parse_block_node)
    10652    0.030    0.000    1.040    0.000 parser.py:267(parse_flow_node)
 81479792  245.582    0.000 20317.619    0.000 parser.py:270(parse_block_node_or_indentless_sequence)
134369985 2213.589    0.000 25823.924    0.000 parser.py:273(parse_node)
 14347188   92.269    0.000 7921.270    0.001 parser.py:376(parse_block_sequence_first_entry)
 36917786  345.101    0.000 12853.680    0.000 parser.py:381(parse_block_sequence_entry)
 37489272  328.860    0.000 12318.202    0.000 parser.py:402(parse_indentless_sequence_entry)
  9987161   57.042    0.000 1814.894    0.000 parser.py:422(parse_block_mapping_first_key)
 50727057  453.853    0.000 13961.679    0.000 parser.py:427(parse_block_mapping_key)
 40739896  352.033    0.000 96992.671    0.002 parser.py:446(parse_block_mapping_value)
  1122040    7.080    0.000  140.391    0.000 parser.py:471(parse_flow_sequence_first_entry)
  1132530   10.722    0.000  119.551    0.000 parser.py:476(parse_flow_sequence_entry)
   229236    1.566    0.000   28.825    0.000 parser.py:537(parse_flow_mapping_first_key)
   229317    2.337    0.000   23.902    0.000 parser.py:542(parse_flow_mapping_key)
       81    0.001    0.000    0.029    0.000 parser.py:569(parse_flow_mapping_value)
        1    0.000    0.000    0.000    0.000 parser.py:62(<module>)
        1    0.000    0.000    0.000    0.000 parser.py:69(ParserError)
        1    0.000    0.000    0.000    0.000 parser.py:72(Parser)
     3753    0.018    0.000    0.018    0.000 parser.py:81(__init__)
     3753    0.007    0.000    0.007    0.000 parser.py:89(dispose)
444394001 1660.088    0.000 139665.555    0.000 parser.py:94(check_event)
        1    0.000    0.000    0.000    0.000 pool.py:100(__init__)
        1    0.001    0.001    0.002    0.002 pool.py:15(<module>)
        1    0.000    0.000    0.001    0.001 pool.py:207(create_connection)
        1    0.000    0.000    0.001    0.001 pool.py:260(connect)
  1054561   19.752    0.000   67.754    0.000 pool.py:286(get_socket)
  1054561   17.456    0.000  123.715    0.000 pool.py:392(maybe_return_socket)
  1054561   14.549    0.000   58.623    0.000 pool.py:413(_return_socket)
  1054560    5.753    0.000    8.608    0.000 pool.py:433(_check)
    30299    0.186    0.000    0.867    0.000 pool.py:44(_closed)
  2109122   11.207    0.000   48.999    0.000 pool.py:516(_get_request_state)
        1    0.000    0.000    0.000    0.000 pool.py:536(Request)
        1    0.000    0.000    0.000    0.000 pool.py:55(SocketInfo)
        1    0.000    0.000    0.000    0.000 pool.py:58(__init__)
  3163683    8.575    0.000   17.552    0.000 pool.py:78(__eq__)
  1054561    4.307    0.000   10.981    0.000 pool.py:83(__ne__)
  1054561    3.747    0.000    5.361    0.000 pool.py:86(__hash__)
        1    0.000    0.000    0.000    0.000 pool.py:99(Pool)
     3753    0.014    0.000    0.024    0.000 posixpath.py:119(basename)
       17    0.000    0.000    0.000    0.000 posixpath.py:321(normpath)
       17    0.000    0.000    0.001    0.000 posixpath.py:350(abspath)
        1    0.000    0.000    0.000    0.000 posixpath.py:51(normcase)
       17    0.000    0.000    0.000    0.000 posixpath.py:59(isabs)
    11262    0.070    0.000    0.106    0.000 posixpath.py:68(join)
     7352    0.063    0.000    0.101    0.000 posixpath.py:89(split)
        1    0.000    0.000    0.000    0.000 process.py:1(<module>)
   649699    4.994    0.000    4.994    0.000 process.py:1(http_template)
   377252    1.446    0.000    1.446    0.000 process.py:12(http_requests)
     1693    0.063    0.000    0.063    0.000 process.py:17(scapy_template)
   144518    0.157    0.000    0.157    0.000 process.py:30(dns_template)
   144518    0.641    0.000    2.170    0.000 process.py:34(dns_consistency)
      131    0.001    0.000    0.001    0.000 process.py:39(captive_portal)
    45930    0.053    0.000    0.053    0.000 process.py:45(null)
    10697    0.012    0.000    0.012    0.000 process.py:53(bridge_reachability)
   198236    0.205    0.000    0.205    0.000 process.py:57(tcp_connect)
  1050904    1.513    0.000    1.513    0.000 process.py:61(default)
     3754    0.025    0.000    0.148    0.000 publish.py:11(list_report_files)
        1    0.000    0.000    0.000    0.000 publish.py:18(ReportInserter)
     3753   28.703    0.008 150443.383   40.086 publish.py:19(__init__)
        1    0.000    0.000    0.000    0.000 publish.py:3(<module>)
     3753    0.004    0.000    0.004    0.000 publish.py:46(__iter__)
1057995/1054560    5.035    0.000 148398.734    0.141 publish.py:49(next)
        1    0.138    0.138 150443.669 150443.669 publish.py:60(main)
        1    0.000    0.000    0.000    0.000 py3compat.py:15(<module>)
       37    0.000    0.000    0.000    0.000 py3compat.py:49(b)
        1    0.000    0.000    0.000    0.000 random.py:100(seed)
        1    0.000    0.000    0.000    0.000 random.py:173(randrange)
        1    0.000    0.000    0.000    0.000 random.py:237(randint)
        1    0.018    0.018    0.050    0.050 random.py:40(<module>)
        1    0.000    0.000    0.000    0.000 random.py:651(WichmannHill)
        1    0.000    0.000    0.000    0.000 random.py:72(Random)
        1    0.000    0.000    0.000    0.000 random.py:801(SystemRandom)
        1    0.000    0.000    0.000    0.000 random.py:91(__init__)
     3753    0.018    0.000    0.066    0.000 re.py:134(match)
        2    0.000    0.000    0.005    0.003 re.py:139(search)
       20    0.000    0.000    0.055    0.003 re.py:188(compile)
        2    0.000    0.000    0.000    0.000 re.py:204(escape)
     3775    0.020    0.000    0.087    0.000 re.py:226(_compile)
        1    0.004    0.004    0.296    0.296 read_preferences.py:15(<module>)
        1    0.000    0.000    0.000    0.000 read_preferences.py:192(MovingAverage)
        1    0.000    0.000    0.000    0.000 read_preferences.py:22(ReadPreference)
570464359 1932.378    0.000 3178.912    0.000 reader.py:114(get_mark)
     3753    0.065    0.000   27.174    0.007 reader.py:122(determine_encoding)
 14213942   41.286    0.000  660.778    0.000 reader.py:138(check_printable)
 18933075  215.663    0.000 1209.008    0.000 reader.py:146(update)
 14217695   75.937    0.000  220.102    0.000 reader.py:177(update_raw)
        1    0.000    0.000    0.015    0.015 reader.py:18(<module>)
        1    0.000    0.000    0.000    0.000 reader.py:24(ReaderError)
        1    0.000    0.000    0.015    0.015 reader.py:45(Reader)
     3753    0.046    0.000   27.237    0.007 reader.py:59(__init__)
19629977522 20983.947    0.000 21923.467    0.000 reader.py:87(peek)
1923998066 5019.320    0.000 6738.774    0.000 reader.py:94(prefix)
4554085195 22534.513    0.000 26641.085    0.000 reader.py:99(forward)
        1    0.000    0.000    0.000    0.000 regex.py:16(<module>)
        1    0.000    0.000    0.000    0.000 regex.py:42(Regex)
        1    0.000    0.000    0.000    0.000 replica_set_connection.py:35(<module>)
        1    0.000    0.000    0.000    0.000 replica_set_connection.py:40(ReplicaSetConnection)
        1    0.015    0.015    0.035    0.035 report.py:1(<module>)
        1    0.000    0.000    0.000    0.000 report.py:6(Report)
        1    0.000    0.000    0.000    0.000 representer.py:12(RepresenterError)
        1    0.000    0.000    0.000    0.000 representer.py:139(SafeRepresenter)
        1    0.000    0.000    0.000    0.000 representer.py:15(BaseRepresenter)
        1    0.000    0.000    0.001    0.001 representer.py:2(<module>)
        1    0.000    0.000    0.000    0.000 representer.py:291(Representer)
       24    0.000    0.000    0.000    0.000 representer.py:74(add_representer)
        2    0.000    0.000    0.000    0.000 representer.py:80(add_multi_representer)
134089831  159.567    0.000  159.567    0.000 resolver.py:111(ascend_resolver)
        1    0.000    0.000    0.000    0.000 resolver.py:12(BaseResolver)
133624891  715.185    0.000 1178.441    0.000 resolver.py:140(resolve)
        1    0.000    0.000    0.000    0.000 resolver.py:164(Resolver)
        1    0.000    0.000    0.026    0.026 resolver.py:2(<module>)
     3753    0.007    0.000    0.007    0.000 resolver.py:21(__init__)
        8    0.000    0.000    0.000    0.000 resolver.py:25(add_implicit_resolver)
134089831  166.303    0.000  166.303    0.000 resolver.py:88(descend_resolver)
        1    0.000    0.000    0.000    0.000 resolver.py:9(ResolverError)
        1    0.000    0.000    0.000    0.000 sanitise.py:1(<module>)
        1    0.083    0.083    1.112    1.112 sanitise.py:21(<module>)
   201782    0.828    0.000    1.286    0.000 scanner.py:1059(scan_block_scalar_indicators)
   201782    1.245    0.000    4.852    0.000 scanner.py:1099(scan_block_scalar_ignored_line)
   201782    5.439    0.000   14.494    0.000 scanner.py:1113(scan_block_scalar_indentation)
 34140527  642.371    0.000 1933.404    0.000 scanner.py:1128(scan_block_scalar_breaks)
1025898704 4733.200    0.000 126846.740    0.000 scanner.py:113(check_token)
  5756191 2516.658    0.000 72156.222    0.013 scanner.py:1141(scan_flow_scalar)
397883862 20264.915    0.000 58373.938    0.000 scanner.py:1191(scan_flow_scalar_non_spaces)
392127671 3989.462    0.000 9828.381    0.000 scanner.py:1234(scan_flow_scalar_spaces)
177738300  422.192    0.000 1799.794    0.000 scanner.py:125(peek_token)
103649325 2007.848    0.000 5786.946    0.000 scanner.py:1258(scan_flow_scalar_breaks)
 94207814 3906.488    0.000 14046.982    0.000 scanner.py:1276(scan_plain)
287878713 1129.961    0.000 3840.744    0.000 scanner.py:132(get_token)
121812308 1770.001    0.000 4913.405    0.000 scanner.py:1323(scan_plain_spaces)
   464940    3.843    0.000    9.210    0.000 scanner.py:1360(scan_tag_handle)
   464940    7.880    0.000   16.797    0.000 scanner.py:1386(scan_tag_uri)
1689982002 5762.602    0.000 12600.034    0.000 scanner.py:142(need_more_tokens)
413351707 1547.830    0.000 3761.361    0.000 scanner.py:1431(scan_line_break)
198466287 2091.144    0.000 111695.939    0.001 scanner.py:153(fetch_more_tokens)
        1    0.001    0.001    0.003    0.003 scanner.py:2(<module>)
1533013057 1823.252    0.000 1823.252    0.000 scanner.py:261(next_possible_simple_key)
        1    0.000    0.000    0.000    0.000 scanner.py:27(<module>)
1731479344 4087.439    0.000 5707.817    0.000 scanner.py:276(stale_possible_simple_keys)
102225619  667.303    0.000 1408.306    0.000 scanner.py:292(save_possible_simple_key)
128605819  171.031    0.000  171.031    0.000 scanner.py:313(remove_possible_simple_key)
        1    0.000    0.000    0.000    0.000 scanner.py:32(ScannerError)
200586127  411.804    0.000  640.185    0.000 scanner.py:326(unwind_indent)
        1    0.000    0.000    0.000    0.000 scanner.py:35(SimpleKey)
 92561345  163.056    0.000  186.107    0.000 scanner.py:350(add_indent)
     3753    0.031    0.000    0.074    0.000 scanner.py:360(fetch_stream_start)
     3654    0.040    0.000    0.088    0.000 scanner.py:372(fetch_stream_end)
 73105128  155.077    0.000  155.077    0.000 scanner.py:38(__init__)
  1058094    3.325    0.000   42.027    0.000 scanner.py:403(fetch_document_start)
  1058092    3.165    0.000   55.703    0.000 scanner.py:406(fetch_document_end)
  2116186   24.450    0.000   91.240    0.000 scanner.py:409(fetch_document_indicator)
  1122040    3.178    0.000   38.084    0.000 scanner.py:425(fetch_flow_sequence_start)
   229236    0.767    0.000    7.765    0.000 scanner.py:428(fetch_flow_mapping_start)
  1351276   14.157    0.000   41.904    0.000 scanner.py:431(fetch_flow_collection_start)
  1122040    3.160    0.000   37.232    0.000 scanner.py:448(fetch_flow_sequence_end)
   229236    0.813    0.000    7.796    0.000 scanner.py:451(fetch_flow_mapping_end)
  1351276   13.970    0.000   41.055    0.000 scanner.py:454(fetch_flow_collection_end)
        1    0.000    0.000    0.000    0.000 scanner.py:46(Scanner)
     6344    0.069    0.000    0.195    0.000 scanner.py:471(fetch_flow_entry)
     3753    0.026    0.000    0.099    0.000 scanner.py:48(__init__)
 51821449  647.886    0.000 1877.131    0.000 scanner.py:485(fetch_block_entry)
 40739977  630.635    0.000 1631.647    0.000 scanner.py:546(fetch_value)
   280154    1.686    0.000   20.017    0.000 scanner.py:602(fetch_alias)
   165244    1.039    0.000   12.113    0.000 scanner.py:613(fetch_anchor)
   464940    2.881    0.000   44.828    0.000 scanner.py:624(fetch_tag)
   201782    0.593    0.000 11281.375    0.056 scanner.py:635(fetch_literal)
   201782    3.673    0.000 11280.782    0.056 scanner.py:641(fetch_block_scalar)
  5289551   15.525    0.000 3589.368    0.001 scanner.py:652(fetch_single)
   466640    1.554    0.000 68674.094    0.147 scanner.py:655(fetch_double)
  5756191   61.507    0.000 72246.383    0.013 scanner.py:658(fetch_flow_scalar)
 94207814  529.422    0.000 16057.790    0.000 scanner.py:669(fetch_plain)
 52932279   85.302    0.000  144.844    0.000 scanner.py:691(check_document_start)
  1059342    4.693    0.000   10.006    0.000 scanner.py:699(check_document_end)
 51874185  121.911    0.000  184.161    0.000 scanner.py:707(check_block_entry)
        8    0.000    0.000    0.000    0.000 scanner.py:712(check_key)
 40740016  100.487    0.000  145.128    0.000 scanner.py:722(check_value)
 94207814  228.769    0.000  331.720    0.000 scanner.py:732(check_plain)
198466287 1539.055    0.000 3864.798    0.000 scanner.py:753(scan_to_next_token)
   445398   10.368    0.000   28.044    0.000 scanner.py:905(scan_anchor)
   464940    6.925    0.000   40.682    0.000 scanner.py:941(scan_tag)
   201782 3479.712    0.017 11276.601    0.056 scanner.py:983(scan_block_scalar)
        1    0.000    0.000    0.000    0.000 serializer.py:11(Serializer)
        1    0.000    0.000    0.000    0.000 serializer.py:2(<module>)
        1    0.000    0.000    0.000    0.000 serializer.py:8(SerializerError)
        1    0.054    0.054    0.794    0.794 settings.py:1(<module>)
        1    0.000    0.000    0.000    0.000 shutil.py:31(Error)
        1    0.000    0.000    0.000    0.000 shutil.py:34(SpecialFileError)
        1    0.000    0.000    0.000    0.000 shutil.py:38(ExecError)
        1    0.003    0.003    0.004    0.004 shutil.py:5(<module>)
        1    0.000    0.000    0.000    0.000 socket.py:167(_closedsocket)
        1    0.000    0.000    0.000    0.000 socket.py:179(_socketobject)
        1    0.000    0.000    0.000    0.000 socket.py:185(__init__)
  1084864    6.068    0.000  498.577    0.000 socket.py:223(meth)
        1    0.000    0.000    0.000    0.000 socket.py:235(_fileobject)
        1    0.004    0.004    0.004    0.004 socket.py:45(<module>)
  2109314   10.734    0.000   22.514    0.000 son.py:102(__setitem__)
  3163971    5.910    0.000    5.910    0.000 son.py:111(keys)
  3163971    5.734    0.000    7.743    0.000 son.py:122(__iter__)
  2109314    5.621    0.000    9.523    0.000 son.py:129(__contains__)
  2109314   13.750    0.000   43.330    0.000 son.py:183(update)
        1    0.000    0.000    0.000    0.000 son.py:19(<module>)
        1    0.000    0.000    0.000    0.000 son.py:29(SON)
  1054657    6.606    0.000   49.936    0.000 son.py:85(__init__)
  1054657    5.523    0.000    7.900    0.000 son.py:91(__new__)
      105    0.001    0.000    0.021    0.000 sre_compile.py:178(_compile_charset)
      105    0.006    0.000    0.019    0.000 sre_compile.py:207(_optimize_charset)
      466    0.000    0.000    0.000    0.000 sre_compile.py:24(_identityfunction)
        6    0.000    0.000    0.000    0.000 sre_compile.py:258(_mk_bitmap)
        2    0.009    0.004    0.009    0.004 sre_compile.py:301(_optimize_unicode)
   186/22    0.004    0.000    0.020    0.001 sre_compile.py:32(_compile)
       63    0.000    0.000    0.001    0.000 sre_compile.py:354(_simple)
       22    0.000    0.000    0.012    0.001 sre_compile.py:361(_compile_info)
       44    0.000    0.000    0.000    0.000 sre_compile.py:474(isstring)
       22    0.000    0.000    0.032    0.001 sre_compile.py:480(_code)
       22    0.000    0.000    0.060    0.003 sre_compile.py:495(compile)
      323    0.001    0.000    0.001    0.000 sre_parse.py:126(__len__)
       12    0.000    0.000    0.000    0.000 sre_parse.py:128(__delitem__)
      795    0.002    0.000    0.003    0.000 sre_parse.py:130(__getitem__)
       63    0.000    0.000    0.000    0.000 sre_parse.py:134(__setitem__)
      379    0.001    0.000    0.001    0.000 sre_parse.py:138(append)
   249/85    0.002    0.000    0.003    0.000 sre_parse.py:140(getwidth)
       22    0.000    0.000    0.000    0.000 sre_parse.py:178(__init__)
     1660    0.006    0.000    0.009    0.000 sre_parse.py:182(__next)
      642    0.001    0.000    0.002    0.000 sre_parse.py:195(match)
     1378    0.003    0.000    0.010    0.000 sre_parse.py:201(get)
        1    0.000    0.000    0.000    0.000 sre_parse.py:205(tell)
       61    0.000    0.000    0.000    0.000 sre_parse.py:210(isident)
       11    0.000    0.000    0.000    0.000 sre_parse.py:216(isname)
       24    0.000    0.000    0.000    0.000 sre_parse.py:225(_class_escape)
       20    0.000    0.000    0.000    0.000 sre_parse.py:257(_escape)
    68/22    0.001    0.000    0.026    0.001 sre_parse.py:301(_parse_sub)
   111/22    0.007    0.000    0.026    0.001 sre_parse.py:379(_parse)
       22    0.000    0.000    0.027    0.001 sre_parse.py:663(parse)
       22    0.000    0.000    0.000    0.000 sre_parse.py:67(__init__)
       19    0.000    0.000    0.000    0.000 sre_parse.py:72(opengroup)
       19    0.000    0.000    0.000    0.000 sre_parse.py:83(closegroup)
      189    0.000    0.000    0.000    0.000 sre_parse.py:90(__init__)
        1    0.000    0.000    0.000    0.000 ssl.py:100(SSLSocket)
        1    0.002    0.002    0.003    0.003 ssl.py:56(<module>)
        1    0.000    0.000    0.000    0.000 ssl_match_hostname.py:4(<module>)
        1    0.000    0.000    0.000    0.000 ssl_match_hostname.py:7(CertificateError)
     3753    0.004    0.000    0.004    0.000 stat.py:24(S_IFMT)
     3753    0.008    0.000    0.011    0.000 stat.py:40(S_ISDIR)
        1    0.000    0.000    0.000    0.000 sync.py:26(Report)
        1    0.000    0.000    0.000    0.000 sync.py:7(<module>)
        1    0.000    0.000    0.000    0.000 tarfile.py:1475(TarFile)
        1    0.000    0.000    0.000    0.000 tarfile.py:2442(TarIter)
        1    0.000    0.000    0.000    0.000 tarfile.py:2479(_section)
        1    0.000    0.000    0.000    0.000 tarfile.py:2488(_data)
        1    0.000    0.000    0.000    0.000 tarfile.py:2495(_hole)
        1    0.000    0.000    0.000    0.000 tarfile.py:2500(_ringbuffer)
        1    0.000    0.000    0.000    0.000 tarfile.py:2526(TarFileCompat)
        1    0.014    0.014    0.019    0.019 tarfile.py:31(<module>)
        1    0.000    0.000    0.000    0.000 tarfile.py:326(TarError)
        1    0.000    0.000    0.000    0.000 tarfile.py:329(ExtractError)
        1    0.000    0.000    0.000    0.000 tarfile.py:332(ReadError)
        1    0.000    0.000    0.000    0.000 tarfile.py:335(CompressionError)
        1    0.000    0.000    0.000    0.000 tarfile.py:338(StreamError)
        1    0.000    0.000    0.000    0.000 tarfile.py:341(HeaderError)
        1    0.000    0.000    0.000    0.000 tarfile.py:344(EmptyHeaderError)
        1    0.000    0.000    0.000    0.000 tarfile.py:347(TruncatedHeaderError)
        1    0.000    0.000    0.000    0.000 tarfile.py:350(EOFHeaderError)
        1    0.000    0.000    0.000    0.000 tarfile.py:353(InvalidHeaderError)
        1    0.000    0.000    0.000    0.000 tarfile.py:356(SubsequentHeaderError)
        1    0.000    0.000    0.000    0.000 tarfile.py:363(_LowLevelFile)
        1    0.000    0.000    0.000    0.000 tarfile.py:387(_Stream)
        1    0.000    0.000    0.000    0.000 tarfile.py:616(_StreamProxy)
        1    0.000    0.000    0.000    0.000 tarfile.py:640(_BZ2Proxy)
        1    0.000    0.000    0.000    0.000 tarfile.py:705(_FileInFile)
        1    0.000    0.000    0.000    0.000 tarfile.py:781(ExFileObject)
        1    0.000    0.000    0.000    0.000 tarfile.py:912(TarInfo)
        1    0.000    0.000    0.000    0.000 tempfile.py:107(_RandomNameSequence)
        1    0.000    0.000    0.000    0.000 tempfile.py:18(<module>)
        1    0.000    0.000    0.000    0.000 tempfile.py:364(_TemporaryFileWrapper)
        1    0.000    0.000    0.000    0.000 tempfile.py:493(SpooledTemporaryFile)
  2109122    8.978    0.000   35.150    0.000 thread_util.py:106(get)
        1    0.000    0.000    0.000    0.000 thread_util.py:114(GreenletIdent)
        2    0.000    0.000    0.000    0.000 thread_util.py:144(create_ident)
        1    0.001    0.001    0.001    0.001 thread_util.py:15(<module>)
        1    0.000    0.000    0.000    0.000 thread_util.py:151(Counter)
        1    0.000    0.000    0.000    0.000 thread_util.py:154(__init__)
        1    0.000    0.000    0.000    0.000 thread_util.py:189(Future)
        1    0.000    0.000    0.000    0.000 thread_util.py:194(__init__)
        1    0.000    0.000    0.000    0.000 thread_util.py:199(set_result)
        1    0.000    0.000    0.000    0.000 thread_util.py:220(Semaphore)
        1    0.000    0.000    0.000    0.000 thread_util.py:224(__init__)
  1054561    5.959    0.000    8.454    0.000 thread_util.py:230(acquire)
  1054561    8.371    0.000   23.442    0.000 thread_util.py:255(release)
        1    0.000    0.000    0.000    0.000 thread_util.py:269(BoundedSemaphore)
        1    0.000    0.000    0.000    0.000 thread_util.py:271(__init__)
  1054561    4.890    0.000   28.331    0.000 thread_util.py:275(release)
        1    0.000    0.000    0.000    0.000 thread_util.py:282(DummySemaphore)
        1    0.000    0.000    0.000    0.000 thread_util.py:293(MaxWaitersBoundedSemaphore)
        1    0.000    0.000    0.000    0.000 thread_util.py:310(MaxWaitersBoundedSemaphoreThread)
        1    0.000    0.000    0.000    0.000 thread_util.py:323(create_semaphore)
        1    0.000    0.000    0.000    0.000 thread_util.py:338(create_event)
        1    0.000    0.000    0.000    0.000 thread_util.py:48(DummyLock)
  2109122    2.448    0.000    2.448    0.000 thread_util.py:49(acquire)
  2109122    2.241    0.000    2.241    0.000 thread_util.py:52(release)
        1    0.000    0.000    0.000    0.000 thread_util.py:56(Ident)
        2    0.000    0.000    0.000    0.000 thread_util.py:57(__init__)
        1    0.000    0.000    0.000    0.000 thread_util.py:78(ThreadIdent)
        2    0.000    0.000    0.000    0.000 thread_util.py:79(__init__)
        1    0.000    0.000    0.000    0.000 thread_util.py:89(ThreadVigil)
  2109122   15.256    0.000   23.834    0.000 thread_util.py:92(_make_vigil)
        1    0.001    0.001    0.004    0.004 threading.py:1(<module>)
        1    0.000    0.000    0.000    0.000 threading.py:101(RLock)
        1    0.000    0.000    0.000    0.000 threading.py:104(_RLock)
        1    0.000    0.000    0.000    0.000 threading.py:106(__init__)
        4    0.000    0.000    0.000    0.000 threading.py:181(Condition)
        1    0.000    0.000    0.000    0.000 threading.py:184(_Condition)
        4    0.000    0.000    0.000    0.000 threading.py:186(__init__)
  1054563    3.249    0.000    4.710    0.000 threading.py:226(_is_owned)
  1054563    6.356    0.000   12.530    0.000 threading.py:277(notify)
        2    0.000    0.000    0.000    0.000 threading.py:295(notifyAll)
        1    0.000    0.000    0.000    0.000 threading.py:304(_Semaphore)
        1    0.000    0.000    0.000    0.000 threading.py:352(_BoundedSemaphore)
        2    0.000    0.000    0.000    0.000 threading.py:364(Event)
        1    0.000    0.000    0.000    0.000 threading.py:367(_Event)
        2    0.000    0.000    0.000    0.000 threading.py:371(__init__)
        2    0.000    0.000    0.000    0.000 threading.py:385(set)
        1    0.000    0.000    0.000    0.000 threading.py:424(Thread)
        1    0.000    0.000    0.000    0.000 threading.py:436(__init__)
        1    0.000    0.000    0.000    0.000 threading.py:531(_set_ident)
        1    0.000    0.000    0.000    0.000 threading.py:56(_Verbose)
        8    0.000    0.000    0.000    0.000 threading.py:58(__init__)
  1054563    1.465    0.000    1.465    0.000 threading.py:63(_note)
        1    0.000    0.000    0.000    0.000 threading.py:737(_Timer)
        1    0.000    0.000    0.000    0.000 threading.py:766(_MainThread)
        1    0.000    0.000    0.000    0.000 threading.py:768(__init__)
        1    0.000    0.000    0.000    0.000 threading.py:775(_set_daemon)
        1    0.000    0.000    0.000    0.000 threading.py:806(_DummyThread)
        1    0.000    0.000    0.000    0.000 timestamp.py:16(<module>)
        1    0.000    0.000    0.000    0.000 timestamp.py:25(Timestamp)
        1    0.000    0.000    0.000    0.000 tokens.py:17(DirectiveToken)
        1    0.001    0.001    0.001    0.001 tokens.py:2(<module>)
        1    0.000    0.000    0.000    0.000 tokens.py:2(Token)
        1    0.000    0.000    0.000    0.000 tokens.py:25(DocumentStartToken)
        1    0.000    0.000    0.000    0.000 tokens.py:28(DocumentEndToken)
186798837  294.322    0.000  294.322    0.000 tokens.py:3(__init__)
        1    0.000    0.000    0.000    0.000 tokens.py:31(StreamStartToken)
     3753    0.009    0.000    0.009    0.000 tokens.py:33(__init__)
        1    0.000    0.000    0.000    0.000 tokens.py:39(StreamEndToken)
        1    0.000    0.000    0.000    0.000 tokens.py:42(BlockSequenceStartToken)
        1    0.000    0.000    0.000    0.000 tokens.py:45(BlockMappingStartToken)
        1    0.000    0.000    0.000    0.000 tokens.py:48(BlockEndToken)
        1    0.000    0.000    0.000    0.000 tokens.py:51(FlowSequenceStartToken)
        1    0.000    0.000    0.000    0.000 tokens.py:54(FlowMappingStartToken)
        1    0.000    0.000    0.000    0.000 tokens.py:57(FlowSequenceEndToken)
        1    0.000    0.000    0.000    0.000 tokens.py:60(FlowMappingEndToken)
        1    0.000    0.000    0.000    0.000 tokens.py:63(KeyToken)
        1    0.000    0.000    0.000    0.000 tokens.py:66(ValueToken)
        1    0.000    0.000    0.000    0.000 tokens.py:69(BlockEntryToken)
        1    0.000    0.000    0.000    0.000 tokens.py:72(FlowEntryToken)
        1    0.000    0.000    0.000    0.000 tokens.py:75(AliasToken)
   280154    0.438    0.000    0.438    0.000 tokens.py:77(__init__)
        1    0.000    0.000    0.000    0.000 tokens.py:82(AnchorToken)
   165244    0.346    0.000    0.346    0.000 tokens.py:84(__init__)
        1    0.000    0.000    0.000    0.000 tokens.py:89(TagToken)
   464940    0.909    0.000    0.909    0.000 tokens.py:91(__init__)
        1    0.000    0.000    0.000    0.000 tokens.py:96(ScalarToken)
100165785  194.997    0.000  194.997    0.000 tokens.py:98(__init__)
        1    0.002    0.002    0.002    0.002 traceback.py:1(<module>)
        1    0.000    0.000    0.000    0.000 tz_util.py:15(<module>)
        1    0.000    0.000    0.000    0.000 tz_util.py:23(FixedOffset)
        1    0.000    0.000    0.000    0.000 tz_util.py:31(__init__)
        1    0.000    0.000    0.000    0.000 tz_util.py:41(utcoffset)
        2    0.000    0.000    0.000    0.000 tz_util.py:47(dst)
        1    0.000    0.000    0.000    0.000 uri_parser.py:113(parse_host)
        1    0.019    0.019    0.027    0.027 uri_parser.py:16(<module>)
        1    0.000    0.000    0.000    0.000 uri_parser.py:214(split_hosts)
        1    0.000    0.000    0.000    0.000 urllib.py:109(ContentTooShortError)
        1    0.000    0.000    0.000    0.000 urllib.py:115(URLopener)
      485    0.001    0.000    0.002    0.000 urllib.py:1199(<genexpr>)
        1    0.001    0.001    0.008    0.008 urllib.py:23(<module>)
        1    0.000    0.000    0.000    0.000 urllib.py:609(FancyURLopener)
        1    0.000    0.000    0.000    0.000 urllib.py:850(ftpwrapper)
        1    0.000    0.000    0.000    0.000 urllib.py:945(addbase)
        1    0.000    0.000    0.000    0.000 urllib.py:974(addclosehook)
        1    0.000    0.000    0.000    0.000 urllib.py:989(addinfo)
        1    0.000    0.000    0.000    0.000 urllib.py:999(addinfourl)
        1    0.000    0.000    0.000    0.000 urlparse.py:118(SplitResult)
        1    0.000    0.000    0.000    0.000 urlparse.py:126(ParseResult)
        1    0.001    0.001    0.004    0.004 urlparse.py:29(<module>)
      485    0.001    0.000    0.002    0.000 urlparse.py:320(<genexpr>)
        1    0.000    0.000    0.000    0.000 urlparse.py:71(ResultMixin)
        2    0.001    0.000    0.089    0.045 util.py:185(_findSoname_ldconfig)
        2    0.000    0.000    0.089    0.045 util.py:223(find_library)
        1    0.004    0.004    0.004    0.004 util.py:4(<module>)
        4    0.000    0.000    0.000    0.000 uuid.py:101(__init__)
        1    0.016    0.016    0.169    0.169 uuid.py:45(<module>)
        1    0.000    0.000    0.000    0.000 uuid.py:53(UUID)
        1    0.000    0.000    0.001    0.001 warnings.py:45(filterwarnings)
        1    0.000    0.000    0.000    0.000 weakref.py:47(__init__)
        3    0.037    0.012    0.038    0.013 {__import__}
 14213942  108.702    0.000  108.702    0.000 {_codecs.utf_8_decode}
        2    0.000    0.000    0.000    0.000 {_ctypes.POINTER}
        3    0.000    0.000    0.000    0.000 {_ctypes.dlopen}
        1    0.000    0.000    0.000    0.000 {_ctypes.set_conversion_mode}
       38    0.000    0.000    0.000    0.000 {_ctypes.sizeof}
        2    0.000    0.000    0.000    0.000 {_hashlib.openssl_md5}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha1}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha224}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha256}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha384}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha512}
        1    0.000    0.000    0.000    0.000 {_socket.getaddrinfo}
        1    0.000    0.000    0.000    0.000 {_socket.gethostname}
       22    0.000    0.000    0.000    0.000 {_sre.compile}
       12    0.000    0.000    0.000    0.000 {_sre.getlower}
       20    0.000    0.000    0.000    0.000 {_struct.calcsize}
  3163971    5.947    0.000    5.947    0.000 {_struct.pack}
  7381928    8.946    0.000    8.946    0.000 {_struct.unpack}
       13    0.000    0.000    0.000    0.000 {all}
   201782   15.300    0.000   15.300    0.000 {binascii.a2b_base64}
        1    0.000    0.000    0.000    0.000 {binascii.a2b_hex}
        1    0.000    0.000    0.000    0.000 {binascii.hexlify}
  1054561    6.924    0.000    6.924    0.000 {bson._cbson.decode_all}
  1054660    2.377    0.000    2.377    0.000 {built-in method __new__ of type object at 0x887160}
        1    0.000    0.000    0.000    0.000 {built-in method fromtimestamp}
        1    0.000    0.000    0.000    0.000 {built-in method utcfromtimestamp}
     1768    0.002    0.000    0.002    0.000 {chr}
        1    0.000    0.000    0.000    0.000 {dir}
        1    0.000    0.000    0.000    0.000 {function seed at 0x1a3e9b0}
  6361740    9.466    0.000    9.466    0.000 {getattr}
        6    0.000    0.000    0.000    0.000 {globals}
  6855110   15.662    0.000   15.662    0.000 {hasattr}
 41794538   39.436    0.000   39.436    0.000 {hash}
  2109122    2.338    0.000    2.338    0.000 {id}
2132706585 2330.915    0.000 2330.915    0.000 {isinstance}
        1    0.000    0.000    0.000    0.000 {issubclass}
6656076886/6656076798 5724.072    0.000 5724.072    0.000 {len}
        4    0.000    0.000    0.001    0.000 {map}
        1    0.000    0.000    0.000    0.000 {math.exp}
        2    0.000    0.000    0.000    0.000 {math.log}
        1    0.028    0.028    0.028    0.028 {math.sqrt}
   201837    0.345    0.000    0.345    0.000 {max}
       13    0.000    0.000    0.000    0.000 {method '__contains__' of 'frozenset' objects}
  8436685   12.790    0.000   12.790    0.000 {method 'acquire' of 'thread.lock' objects}
  1054572    5.331    0.000   10.692    0.000 {method 'add' of 'set' objects}
2745961008 2696.551    0.000 2696.551    0.000 {method 'append' of 'list' objects}
        2    0.000    0.000    0.000    0.000 {method 'clear' of 'dict' objects}
     3656    0.126    0.000    0.126    0.000 {method 'close' of 'file' objects}
        1    0.001    0.001    0.001    0.001 {method 'connect' of '_socket.socket' objects}
  1054664    1.245    0.000    1.245    0.000 {method 'copy' of 'dict' objects}
        4    0.000    0.000    0.000    0.000 {method 'count' of 'list' objects}
   201783    0.639    0.000   17.436    0.000 {method 'decode' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'digest' of '_hashlib.HASH' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
 87343420  186.131    0.000  186.134    0.000 {method 'encode' of 'unicode' objects}
        2    0.000    0.000    0.000    0.000 {method 'end' of '_sre.SRE_Match' objects}
    18770    0.025    0.000    0.025    0.000 {method 'endswith' of 'str' objects}
1126405857 1100.405    0.000 1100.405    0.000 {method 'extend' of 'list' objects}
    30299    0.045    0.000    0.045    0.000 {method 'fileno' of '_socket.socket' objects}
        1    0.000    0.000    0.000    0.000 {method 'find' of 'str' objects}
      247    0.000    0.000    0.000    0.000 {method 'format' of 'str' objects}
196856481  211.397    0.000  211.397    0.000 {method 'get' of 'dict' objects}
        2    0.000    0.000    0.000    0.000 {method 'group' of '_sre.SRE_Match' objects}
 50727141   75.486    0.000   75.486    0.000 {method 'insert' of 'list' objects}
       86    0.000    0.000    0.000    0.000 {method 'isalnum' of 'str' objects}
       13    0.000    0.000    0.000    0.000 {method 'isdigit' of 'str' objects}
   144542    1.529    0.000    1.529    0.000 {method 'items' of 'dict' objects}
  2109317    2.518    0.000    2.518    0.000 {method 'iteritems' of 'dict' objects}
       29    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
100630725  204.560    0.000  204.560    0.000 {method 'join' of 'unicode' objects}
1731479345 1620.378    0.000 1620.378    0.000 {method 'keys' of 'dict' objects}
   718609    0.891    0.000    0.891    0.000 {method 'lower' of 'str' objects}
  3948671    5.619    0.000    5.619    0.000 {method 'lower' of 'unicode' objects}
 80570401  260.895    0.000  260.895    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
        8    0.000    0.000    0.000    0.000 {method 'pop' of 'dict' objects}
472268672  647.211    0.000  647.211    0.000 {method 'pop' of 'list' objects}
  1054561    1.350    0.000    1.350    0.000 {method 'pop' of 'set' objects}
        1    0.000    0.000    0.000    0.000 {method 'random' of '_random.Random' objects}
 14217698  130.483    0.000  130.483    0.000 {method 'read' of 'file' objects}
  2109122  597.624    0.000  597.624    0.000 {method 'recv' of '_socket.socket' objects}
  7382122    8.202    0.000    8.202    0.000 {method 'release' of 'thread.lock' objects}
       19    0.000    0.000    0.000    0.000 {method 'remove' of 'list' objects}
  5415406    7.921    0.000    7.921    0.000 {method 'replace' of 'str' objects}
    11105    0.028    0.000    0.028    0.000 {method 'rfind' of 'str' objects}
     7352    0.013    0.000    0.013    0.000 {method 'rstrip' of 'str' objects}
 14213944  619.493    0.000  619.493    0.000 {method 'search' of '_sre.SRE_Pattern' objects}
  1054561  491.091    0.000  491.091    0.000 {method 'sendall' of '_socket.socket' objects}
       86    0.000    0.000    0.000    0.000 {method 'setdefault' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'setsockopt' of '_socket.socket' objects}
        2    0.000    0.000    0.000    0.000 {method 'setter' of 'property' objects}
        2    0.000    0.000    0.000    0.000 {method 'settimeout' of '_socket.socket' objects}
        1    0.000    0.000    0.000    0.000 {method 'sort' of 'list' objects}
       23    0.000    0.000    0.000    0.000 {method 'split' of 'str' objects}
  9409503   11.873    0.000   11.873    0.000 {method 'startswith' of 'str' objects}
        4    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'toordinal' of 'datetime.date' objects}
        3    0.000    0.000    0.000    0.000 {method 'translate' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'update' of '_hashlib.HASH' objects}
  9953240   18.843    0.000   18.843    0.000 {method 'update' of 'dict' objects}
   263159    0.514    0.000    0.514    0.000 {method 'update' of 'set' objects}
  1054658    1.703    0.000    1.703    0.000 {method 'values' of 'dict' objects}
      427    0.000    0.000    0.000    0.000 {min}
     3754    0.070    0.000    0.070    0.000 {open}
      394    0.000    0.000    0.000    0.000 {ord}
        3    0.000    0.000    0.000    0.000 {posix.getcwd}
  3163780    3.799    0.000    3.799    0.000 {posix.getpid}
        1    0.002    0.002    0.002    0.002 {posix.listdir}
       44    0.004    0.000    0.004    0.000 {posix.mkdir}
        2    0.007    0.004    0.007    0.004 {posix.popen}
     3654    0.302    0.000    0.302    0.000 {posix.rename}
     3654    0.049    0.000    0.049    0.000 {posix.rmdir}
     7451    0.082    0.000    0.082    0.000 {posix.stat}
        2    0.000    0.000    0.000    0.000 {posix.uname}
        1    0.000    0.000    0.000    0.000 {posix.urandom}
  1054657  129.816    0.000 1613.565    0.002 {pymongo._cmessage._do_batched_write_command}
        1    0.000    0.000    0.000    0.000 {pymongo._cmessage._query_message}
123231986  165.172    0.000  165.172    0.000 {range}
        6    0.000    0.000    0.000    0.000 {repr}
    30299    0.407    0.000    0.650    0.000 {select.select}
       22    0.000    0.000    0.000    0.000 {setattr}
        2    0.000    0.000    0.000    0.000 {sys._getframe}
        1    0.000    0.000    0.000    0.000 {sys.getfilesystemencoding}
       10    0.000    0.000    0.000    0.000 {thread.allocate_lock}
        2    0.000    0.000    0.000    0.000 {thread.get_ident}
  3163782    5.951    0.000    5.951    0.000 {time.time}
123231905  131.173    0.000  131.173    0.000 {unichr}
        1    0.000    0.000    0.000    0.000 {zip}


82213.52user 64491.05system 41:47:25elapsed 97%CPU (0avgtext+0avgdata 93764maxresident)k
28355272inputs+0outputs (27major+410693minor)pagefaults 0swaps

Child Tickets

Change History (5)

comment:1 Changed 5 years ago by otr

I investigated the pyyaml issue using the tests mentioned here [1] as the output above also suggests that there is an issue in the yaml library. I changed the load invokatin in report.py to be yaml.load_all(y, Loader=yaml.CSafeLoader)' instead of yaml.safe_load_all(y).

However I did _not_ observe any difference in performance. The time it took to import is the same down to the second and still most of the processing is done in parser.py, scanner.py and reader.py.

[1] https://stackoverflow.com/questions/16864132/is-there-an-up-to-date-fast-yaml-parser-with-python-bindings

comment:2 Changed 4 years ago by dcf

For what it's worth, I was also struggling with the slowness of the Python yaml module (in the context of this project) to find server-side blocking of Tor in OONI reports). For me, yaml.CSafeLoader is way faster, like over 30×.

These are the times to parse 1.5 GB of gzip files, consisting of http_requests reports between 2015-06-16 and 2015-06-24:

yaml.safe_load_all(f)
real    138m29.467s
user    138m27.808s
sys     0m6.356s

yaml.load_all(f, Loader=yaml.CSafeLoader)
real    4m40.021s
user    5m21.960s
sys     0m7.428s

I had tried optimizing the HTML parsing and gzip decompression; the YAML decoding was the bottleneck by far.

comment:3 Changed 4 years ago by dcf

Here is another data point using many more reports. All the http_requests reports until 2015-07-17, 2,758 of them, 33 GB compressed.

yaml.load_all(f, Loader=yaml.CSafeLoader)
real    133m23.210s
user    147m24.860s
sys     2m8.624s

This is faster than processing 1.5 GB of files using yaml.safe_load_all (and the processing program is even doing more work now). The processing program grabs the bodies of all responses, runs a bunch of regexes against them, and optionally saves the bodies to files.

comment:4 Changed 22 months ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

comment:5 Changed 15 months ago by teor

Keywords: archived-closed-2018-07-04 added
Resolution: wontfix
Status: newclosed

Close all tickets in archived components

Note: See TracTickets for help on using tickets.