Opened 7 months ago

Closed 7 months ago

Last modified 3 months ago

#26207 closed defect (fixed)

Stem 1.6.0 incompatible with PyPy 5.6.0

Reported by: mikeperry Owned by: atagar
Priority: Medium Milestone:
Component: Core Tor/Stem Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Steps to reproduce:

sudo apt-get install pypy
virtualenv -p pypy stemtest
source stemtest/bin/activate
pip install stem==1.6.0
pypy -c "import stem.util.system"

This yields:

Traceback (most recent call last):
  File "<module>", line 1, in <module>
  File "/home/user/Source/vanguards.git/stemtest/site-packages/stem/util/system.py", line 97, in <module>
    DEFAULT_SIZE = sys.getsizeof(0)  # estimate if object lacks a __sizeof__
TypeError: sys.getsizeof() is not implemented on PyPy.

There is then a whole blob of explanatory text after that error from PyPy:

A memory profiler using this function is most likely to give results
inconsistent with reality on PyPy.  It would be possible to have
sys.getsizeof() return a number (with enough work), but that may or
may not represent how much memory the object uses.  It doesn't even
make really sense to ask how much *one* object uses, in isolation
with the rest of the system.  For example, instances have maps,
which are often shared across many instances; in this case the maps
would probably be ignored by an implementation of sys.getsizeof(),
but their overhead is important in some cases if they are many
instances with unique maps.  Conversely, equal strings may share
their internal string data even if they are different objects---or
empty containers may share parts of their internals as long as they
are empty.  Even stranger, some lists create objects as you read
them; if you try to estimate the size in memory of range(10**6) as
the sum of all items' size, that operation will by itself create one
million integer objects that never existed in the first place.

Stem 1.5.4 does not have this issue.

Child Tickets

Change History (6)

comment:1 Changed 7 months ago by atagar

Resolution: fixed
Status: newclosed

Thanks Mike! PyPy compatibility fixed, and added checking it to our release instructions.

https://gitweb.torproject.org/stem.git/commit/?id=c52db04

comment:2 Changed 7 months ago by teor

Pypy also comes in python 2.7 and python 3.5 compatible flavours. I'm not sure if you want to test both.
http://pypy.org/download.html

comment:3 Changed 3 months ago by traumschule

also happens with PyPy 6.0.0 and Python 2.7.13 packaged in debian buster:

$ pypy --version
Python 2.7.13 (6.0.0+dfsg-1, Apr 27 2018, 22:00:35)
[PyPy 6.0.0 with GCC 7.3.0]
Last edited 3 months ago by traumschule (previous) (diff)

comment:4 Changed 3 months ago by traumschule

is resource.getrusage(resource.RUSAGE_SELF).ru_maxrss an appropriate alternative?

comment:5 Changed 3 months ago by atagar

Thank traumschule. On first glance that looks to be a replacement for 'size of our process in memory' rather than 'size of individual object X'. That said, this is a very, very unimportant helper function. If there's a PyPy drop in replacement then neat, but unimportant.

As for testing multiple versions of pypy, pre-release I'll only be testing against the copy I got from apt-get. If folks would like specific versions checked we should probably set up a jenkins job.

comment:6 Changed 3 months ago by traumschule

Maybe then pypy could just return 0 for the size instead of bailing out on stem 1.6.0. I assume you put some work into the new version .)
(or there's less need for running high performance stuff on the latest version)

Note: See TracTickets for help on using tickets.