Restricted tinypy to C++ compiler
I’ve spent the last week working on a tinypy to C++ converter. It works! See the screenshot to the right – I’ve managed to port a pygame game over to C++.
Here’s how (and some of the catches):
- I require type annotation of all the functions and methods. “def test(x:str)->str: return x”
- I do two passes on each file, the first pass to catch all the function types and class members, and the second pass to generate the code.
- I generate C++ code that has automagic reference counting. So you have to code your script so it won’t have any cyclic references if you want garbage collected for you.
How is this different from shedskin (really cool project!)?
- Built-in reference counting, instead of using libgc.
- I require the user to type annotate everything.
- It only supports a subset of the tinypy subset of python. Shedskin supports a much larger subset of python.
So what’s the point?
- Well, I learned a lot about STL and C++.
- I know it will produce iPhone friendly code, I’m pretty sure libgc isn’t iPhone friendly? (At least, I haven’t found anything via a few searches…)
- Way less magic. Since everything is annotated, there are no surprises.
- Implementing C++ modules is pretty easy – the code can be inlined within the python code and it just works.
- This will make it easier for me to develop C++ games.
Anyway, if you’re super brave, you can check out svn://www.imitationpickles.org/tinypy/branches/tinypy2 .. I don’t have the elephants example in there, but the pygame.py that I include gives you a pretty good example of a complex module.
I’m going to chat with the tinypy folks to see if we’ll merge this into the tinypy trunk or have it as a separate project. I’m not quite sure what makes sense to everyone else 🙂 The nice bit about merging this in is that I could unify the test suites nicely. And tinypy would still function as normal, just better tested, and with function annotation parsing supported. All that said, I should make a tinypy module for tinypyC++ so that I can do some code evals!
December 1st, 2009 at 4:32 pm
How much does the generated code depend on C++ features? Any chance for translation to plain C?
December 1st, 2009 at 4:59 pm
I’ve been thinking of doing something like this for Javascript. Thanks for publishing your work! If you have any advice about what was difficult or what you could have improved, please post!
December 1st, 2009 at 5:17 pm
@Alex: it is very dependent on C++. However, take a look at encode2.py and builtins.h .. There’s no reason why you couldn’t comb through those two files and change them for outputting C code.
@Donny: No problem 🙂 The thing that “worked” was having a test suite for tinypy already, I really just had to comment out the tests that were “dynamic” and just grind through them one at a time until it all worked. And then I had to add more tests. Really, just a long string of tests that sort of forced me to coax the code into working.
The 2 pass type inference I do is pretty dodgy, and I think it could use some tightening up, but it seems to work in a pinch.
I’m not sure how practical all of what I did was, but I learned quite a bit of C++ which I’m sure will be useful.
December 2nd, 2009 at 4:11 am
nice work Mr H!
December 2nd, 2009 at 3:38 pm
@Donny: You might tae a look at pyjs.org. It does something like this for javascript.
December 3rd, 2009 at 5:16 am
Owesome, Phil.
December 3rd, 2009 at 10:36 am
This looks very fun and interesting. Do you have any sense of the performance between cpython, tinypy, and the generated C++?
December 3rd, 2009 at 11:08 am
@GWF: C++ is like 4-40x faster than Cpython which is about 2x faster than tinypy.
December 3rd, 2009 at 12:15 pm
looking at the shedskin test set, I’d say the performance difference is more like 1-100x, and depends very much on what you’re doing. the cpython builtins and memory subsytem are rather efficient. so if you’re using strings a lot, or sets, and/or many other small objects, chances are a (straightforward) translation to C++ is actually not that much faster and sometimes even slower. but of course usually there is a good speedup, especially if you avoid things that really slow C++ down (usually memory allocations).
December 3rd, 2009 at 1:47 pm
@mark – that’s quite true about memory allocations. With the julia.py example (included in svn right now) you can see that I create a single “pos” object for the pixel position to be passed to the set_at() method.
I did this because I found that creating a new list for each pixel was really slowing things down. By re-using the same list, I got a major speedup.
December 4th, 2009 at 11:42 am
Does it support creating C++ classes consumable within C++? That is, can I define a Python class and then refer to the C++ized class from outside of Python, or does it just do functions?
I’ve been thinking about prototyping an iPhone game I’d like to build in pygame, and it’d be really cool if I could reuse my game logic module in the actual game.
December 4th, 2009 at 11:51 am
@Ryan – yeah, it generates classes that can be used in C++ 🙂 Check out the svn and go into examples and run ./build .. Check out the pygame.hh and pygame.cc that are produced.
(You might need to do an ln ../builtins.h builtins.h to get it to work)
December 6th, 2009 at 12:02 am
[…] Restricted tinypy to C++ compiler […]
December 8th, 2009 at 10:19 am
OMG OMG I was thinking about doing exactly this !!