Skip to main content

Pycon UK, Javascript and the GIL

Just got back from Pycon UK 2008 - here are some impressions.

Both the keynote speakers Mark Shuttleworth (Canonical) and Ted Leung (Sun Microsystems) expressed their concerns about Javascript becoming so fast and prominent that it could displace Python in the future. They also highlighted the fact that Multi-core systems get cheaper and more popular also on desktop computers or notebooks. They challenged the community to advance Python implementations to exploit it. Question was up what PyPy can do here. As it stands, PyPy still uses the good old Global Interpreter Lock (GIL) but our approaches should indeed lend itself well to do experimentation with free threading.

During the 2-day conference we met many interesting people, most notably the guys from Resolver, among them William Reade who is working on IronClad -- which implements a fake python25.dll on top of IronPython. He presented some good results for Numpy in his lightning talk. This approach is surely something to follow closely and potentially use for PyPy.

We also had lunch and a couple of chats with Jacob Kaplan-Moss from Django fame - he is apparently up to try use PyPy's sandboxing features for one of his projects, cool!

Conference itself was well organized for the 230 attending people - although the venue might be a bit small for next year's EuroPython. Ah, and we gave three well attended talks, find the slides here:

Holger, Maciej, Anto (associated through merlinux, btw)
Lucian wrote on 2008-09-17 19:36:

Is the work done on this ( useful conceptually?

René Leonhardt wrote on 2008-09-17 21:27:

Is the new multiprocessing module going to offer improved multi-core performance?

Colin Walters wrote on 2008-09-17 22:40:

Jython? About to hit 2.5, has a threading model for free from the JVM.

Luis wrote on 2008-09-18 00:39:

I wonder how the new javascript improvements compare to pypy technically. For example, the tracing techniques of Mozilla's Tracemonkey look impressive, but I don't know if these techniques are conceptually close or not to pypy's. Is there anything you can learn from them (tracemonkey, chrome's v8, etc).


Miguel Filipe wrote on 2008-09-19 15:19:

Yes, the new multiprocessing module will improve multi-core performance if you use it.
That module allows a easy way to use multiple processes cooperatively in python. It tries to mimic the threading API.
If you use multiple processes instead of threads you will avoid the Global Interpreter Lock.

About new chalenges for PyPy, multicore isn't the major problem.. the absense of a powerfull JIT and GC is. Please keep working on a super-fast "VM+JIT" for python! (super linux performance is a must)

holger krekel wrote on 2008-09-20 13:57:

colin: true, jython and also ironpython can make use of multiple threads. should have mentioned it. Doesn't mean that pypy-c shouldn't go for it, rather the opposite i'd think :)

renel/multiprocessing module: i can imagine it helps with multi-core cpus. are there practical experiences using it yet?

luis, miguel: there are considerable efforts on the PyPy/JIT front - particularly from Armin, Carl Friedrich and Antonio - would be worth a dedicated blog post to relate this to tracing JITs, V8, squirrelfish, etc. One thing i know is that we probably want to apply for funding to help completing the JIT.

Miguel: We do have advanced GCs and are working on on improving them, currently.

to all: thanks for your feedback!

Düsseldorf PyPy sprint 5-13th October, 2008

The PyPy team is happy to announce the next sprint, which will take place in the Computer Science Department of the University of Düsseldorf, Germany. Sprinting will start on the 6th of October and go on till the 12th. Please arrive on the day before if you want to come.

Topics of the sprint will be aiming at a 1.1 release and to work on integrating PyPy better with small devices. Other topics are also welcome!

We will try to find a hotel with group rates, so if you are interested, please sign up soon! See the announcement for more details.

pylib/py.test 0.9.2 released

PyPy and its 14638 automated tests use the py.test tool which is also used by many other projects. PyPy developers have actually driven and contributed a lot to its development. I just released version 0.9.2 of the py lib mainly fixing Windows issues and providing better packaging and integration with setuptools. It's usable completely independently from PyPy - "easy_install py" gives you the py.test command line. Of course you can run py.test on top of a translated PyPy version as well. Here is a quick summary of what the py lib provides besides py.test:

  • py.execnet: ad-hoc code distribution to SSH, Socket and local sub processes
  • py.magic.greenlet: micro-threads on standard CPython ("stackless-light") and PyPy
  • py.path: path abstractions over local and subversion files
  • py.code: dynamic code compile and traceback printing support
  • tested against Linux, Win32, OSX, works on python 2.3-2.6
Good general entry points for installation and documentation: have fun, holger krekel
Anonymous wrote on 2008-09-02 18:30:

We use py.test in the development of the Translate Toolkit and Pootle - many thanks :)

I use Fedora so here are Fedora RPMs for pylib:

holger krekel wrote on 2008-09-23 15:44:

Hi Dwayne!

thanks a lot. I added a link to your RPM from the download page. Let me know if there was anything that hindered packaging.


New translation option: --opt

Hi all,

A few command-line options for have changed. Most interesting is that optimization levels are selected with the option --opt, or -O for short. This replaces --allopts, which was also called --faassen in reference to a person who is actually not involved in PyPy (so that was a bit of a strange joke). Also, --allworkingmodules is the default nowadays, and can be cancelled with --no-allworkingmodules. Threads are also included in --allworkingmodules now.


  • (reasonable default, corresponds to --opt=2)
  • --opt=3 (best, maybe 10-20% faster)
  • --opt=1 (translation is faster and less RAM-hungry)

For more information, see:

holger krekel wrote on 2008-08-19 16:48:

maybe for a bit of background: Martijn Faassen regularly asked at our talks "how fast is PyPy now?" - at times when PyPy was going from 2000 to 500 to 50 to ???? times slower than CPython (nowadays at 1-6 times, btw). so with "--faassen" we were trying to translate an "as-fast-as-possible" pypy. so now we are getting dead serious (also Martijn actually asked for removing his name from the commandline) and introduced a we-are-becoming-a-real-compiler-with-opt-levels "-O" option :)

Martijn, to be clear: i really appreciate having you and your questions in our talks and in general - it also pushed me to get out py.test releases ... :) holger

Anonymous wrote on 2008-08-22 02:56:

Congrats to PyPy on having a more sensible option! (though an option called 'opt' made me think it stood for 'option' first :).

Thanks Holger for the background. "actually not involved in PyPy" depends on your interpretation of what the word "involved" means. Besides performance related questions, as Holger indicates I've asked other questions of the PyPy project too. I wasn't altogether successful at it (nor altogether unsuccessful), and I'm on an extended break from asking any questions right now.

I didn't write any of the PyPy code and also had nothing to do with the design of PyPy. I indeed asked for the --faassen option to be removed earlier this year. It was amusing (and flattering), but it also lead to some confusion concerning credit that I certainly don't deserve - that goes to the PyPy developers and project managers.

Europython 2008 PyPy talks and sprint sum up

The EuroPython 2008 conference and sprints have finished - it certainly was a very eventful and successful conference for PyPy. And many very interesting non-PyPy talks as well. PyPy presentations are available online: PyPy status talk PyPy for the rest of us, PyPy behind the scenes. Armin and Maciej also did a well-attended talk about PyPy's garbage collection, but that was quite interactive, no slides.

The talks were all well visited and we got good questions. However, we still need to work on sorting out the "PyPy technology cloud" and how to present it to different audiences. Anyway, we are happy to hear feedback or questions about the talks!

After the conference there was a three-day PyPy sprint. Despite the fact that most PyPy core developers were zombies, we made good progress. Particularly our newcomers did very well. Here are some results:

  • itertools rewritten in RPython for performance by Jakub Gustak and Andrew Durdin
  • a new ctypes based dbm and hashlib module, both by Gasper Zejn with support from Henrik Vendelbo, they also got ctypes to nicely work on OSX. (sorry for lack of proper letters in names :)
  • implement builtin function call profiling by Stephan Diehl, Antonio and Armin.
  • running Pinax on top of pypy-c, by Henrik, Holger, Gasper.
  • Jim Baker started a for Jython using JNA aiming to provide support to run PyPy's ctypes on top of Jython. When Jython gets this to run, PyPy's JVM backend should be able to use it. Talk about Code Reuse :)
  • oldstyle classes are now the default, this makes PyPy mimick very closely cpython's 2.5 object model.
  • Andrew started a port of the Malbolge interpreter written in Python to RPython (obviously the only missing link for PyPy to take over the world).
  • various cleanups (a new option "--lonepycfiles" helps with saner imports, remove int-float comparison shortcuts, ...)
At the end of the sprint we also discussed initial plans for a 1.1 release which we'd like to make happen this year. So we are generally looking forward to a busy rest of 2008 and luckily this starts by many of us taking a good vacation first :)

fijal & holger
Armin Rigo wrote on 2008-07-16 00:20:

The option is not --lonepycfiles but --objspace-lonepycfiles, and using it makes imports *less* sane.

holger krekel wrote on 2008-07-16 08:44:

oh, right. I meant to say that with the introduction (not the enabling) of the option imports are saner - in that pypy now by default ignores pyc files if there is no ".py" file. thanks for the attention.

Finding Bugs in PyPy with a Fuzzer

Last week I played a bit with Fusil, which is a fuzzing framework. The idea is to feed the interpreter code that calls the functions of a module with random values of various types as arguments in the hope that one hits an unchecked case. This is done until a problem is hit , the most common problem being a segfault. Victor Stinner, the author of Fusil, is a regular in the PyPy IRC channel and thankfully helped me getting started with Fusil. I used his project description for CPython as a starting point and tweaked it a bit. Reason is that PyPy is harder to segfault and so I tweaked Fusil to also count uncaught RPython-level exceptions as such a problem. (RPython has full exception support, and if an RPython-exception escapes to the top level, the Python interpreter aborts. One should not be able to exploit this but but for a user it is bad enough, because such exceptions cannot be caught from Python code.)

Using Fusil I found a number of cases where such exceptions happened (in some pickle support-code, in the expat parser, in the os and in the termios module) and also one or two segfaults (in the parser module, of all places). I fixed all these problems so that by now the fuzzer just runs for a very long time and only finds things that take too long (so they count as a way to do a DoS attack) like pow(12355123123L, 12351512123121L) or round(1, 1000000000) (the latter should probably be fixed). This probably just means that the fuzzer is not good enough, because there are certainly segfaults left in PyPy. However, the fact that it is rather hard to find them validates our approach of using a high-level memory-managed language for our interpreter. Victor tells me that it is rather easy to find segfaults in CPython this way, he already found quite some problems.

Marius Gedminas wrote on 2008-07-13 20:42:

Nice post!

I especially like your certainty that PyPy has segfaults left in it. :-)

Maciej Fijalkowski wrote on 2008-07-13 22:02:

What? Segfaults in PyPy? Shouldn't have any left by now :-)

Armin Rigo wrote on 2008-07-13 22:04:

That previous comment was from me, accidentally logged in as Maciej, sorry. As usual, in PyPy confusion comes for free.

Maciej Fijalkowski wrote on 2008-07-14 07:17:

heh :) I was a bit surprised to see my comment which I did not write. Anyway, I agree with it :-)

Anonymous wrote on 2009-02-07 06:37:

You said you will love me wow gold the whole life, but WoW Gold you marry her. You said you will wow power leveling,come to marry me, but this will not be carried out forever.WoW Gold I am trying my best to forget you and do not love you anymore. wow leveling But I failed and I still love you. Maybe wow leveling she needs you more compared wow leveling with me. So I tell you that world of warcraft power leveling you should love world of warcraft power leveling her and take good world of warcraft leveling care of her. You said I was so of warcraft leveling Yes, because I love you,world of warcraft leveling I hope you will be power leveling happy forever.

PyPy's Python runs Pinax / Django

During the EP2008 sprint we got Pinax running on top of PyPy. At our play1 server we have it running on top of pypy-c. Not that you'll notice many differences to the original site but that's the point, isn't it? ... Well, in fact i am too lazy to customize our play1 version now - i rather spent a nice evening with the other sprint guys :) Pinax integrates numerous reusable Django apps to take care of the things that many sites have in common. Many thanks particularly to Henrik Vendelbo who sorted out various Pinax and PyPy issues, and wrote up a nice DjangoAndPyPy wiki page describing the installation process. greetings from Vilnius (Lithunia), Holger

EP2008: PyPy meets Jython

One of the great events at EuroPython 2008 were our chats and meetings with the Jython and Sun people. The Jython people recently are pushing into releasing Python version 2.5 and they currently pursue many interesting sub projects. Coincidentally, PyPy also has tons of interesting areas and results :) So we eventually got into brainstorming a number of possible technical collab ideas. Further below is a first list as i wrote it down from our 10 people PyPy / Jython 30 minute close up meeting yesterday. It felt great to be able to talk to the Jython people this way - kudos to Sun for their clear commitments and open ways to go about things! I sense a genuine interest on fair collaboration with non-java developer communities. Seems like they are serious about not focusing on "Java this", "Java that" anymore but rather focus on the JVM platform. Good! And about language independent interest in ambitious technology. Even Better! I am tensed to see how things go from here. So here the list of technical collab ideas:

  • ctypes - try to create _rawffi module in Java for Jython, which will enable Jython to reuse our existing ctypes implementation (and have PyPy use the Jython-rawffi for its own for PyPy.JVM)
  • generally see to share work / (continue) collaborate regarding extension modules
  • Jython/PyPy (and eventually IronPython): document known differences to CPython, maybe in a PEP
  • Python Interpreter for Jython (in order to run CPython's .pyc files): re-use pypy's bytecode evaluator, implement a "Jython object space".
  • re-use rpython-extension modules for jython (e.g. SRE), by compiling them to Java and reusing as a native library.
  • collaborate on testing framework / benchmarking, have a common site to show test results
  • make py.test compatible with jython
  • come up with a set of "pure Python language" tests, which would gather and refactor tests from CPython, PyPy and Jython.
  • look into using java types / jython approaches for implementing free threading.
  • share knowledge regarding JIT / psyco
If you have any more ideas, comments or would like to join efforts, let us know! Cheers and thanks to Ted Leung, Frank Wierzbiki, Jim Baker and Tobias Ivarsson from Sun and Jython fame respectively, Holger
cartman wrote on 2008-07-10 09:50:

This is great news, a common VM for all Python implementations would be real cool :)

Michael Foord wrote on 2008-07-10 11:44:

It would be great to get the IronPython folks involved as well.

For example, .NET has an FFI and with the right effort could take advantage of ctypes extensions as well.

holger krekel wrote on 2008-07-10 13:52:

fuzzyman: do you have anyone particular person/group in mind?

PyPy at the EuroPython 2008

Greetings from Vilnius, Lithuania. There were already two pypy talks, one performed by Jacob Hallen PyPy for the rest of us and second by Maciej Fijalkowski PyPy status talk. The thing that we forgotten to tell is that PyPy sandboxing feature can also easily limit CPU and RAM usage as well as any other possible resource (like network transfer). For anyone who would like to join, there is a PyPy sprint after the conference.

arigo & fijal

Jonathan Ellis wrote on 2008-07-07 18:13:

Can you post PDFs of those slides? The text is not rendering for me in NeoOffice.

Anonymous wrote on 2008-07-07 22:08:

Is Maciej using some secret cool reST presentation tool?

Anonymous wrote on 2008-07-08 03:52:

You can convert to PDF online.

This link should work for the next 24 hours. You can regenerate it on the same site after that.

Maciej Fijalkowski wrote on 2008-07-08 06:19:

I checked in pdf version. I use rst2beamer + hacks :) Ask Antonio Cuni for details.

Antonio Cuni wrote on 2008-07-08 08:20:

yes, we are using rst2beamer:

to have some hints how to use it, look at this script I used to generate my pycon-italy talk:

I also wrote some rst macros that allows you to put some paragraphs in those nice beamer's exampleblock and alertblock: