Hmn .. maybe someone did ..

Yep, that’s python+pygame alright ..

Yesterday and today I worked with Mike Fletcher and had a great time porting a couple of my games to the OLPC

Watermelons

Elephants (pic taken with an XO!)

I had a great time working on this - though a couple of odd bugs in the dev. environment proved to be a bit bothersome, we were able to get through them. The main product of this effort was the creation of the “port your pygame game to the OLPC” tutorial.

On a side note, I also gave an impromptu talk on networking games to help an OLPC group get some good ideas on how to write an easy to use networking API for their games. A fair portion of it explained how I implemented networking in Galcon. The other portion explained how “I did it wrong - and you should do it some other way such as _____.” Richard Jones and Mike Fletcher also attended and chipped to help explain the bits that I glazed over.

I never thought I’d see the day where someone would do a retro-remake of one of my games! Indeed, someone has - Joe Snow Test Demo Released!! Check it out and leave some feedback

(My original is here for comparison. The remake is much better!)

Updates - I’ve got it all fitting in 64k*. It’s amazing how many functions that don’t do anything you can come across if you look around long enough. Not to mention how many little things you can trim out that don’t actually do anything. I have no idea where all this cruft comes from, but having a nice suite of tests sure is helpful for re-working stuff. I also cut out a bunch of stupid features nobody would ever use**. I was able to reduce the number of native types from 9 to 7.

One of the challenges I faced was trying to fix up the incremental garbage collection. My initial implementation was rather inefficient and caused some odd problems with how I wanted to code things. I was using a dict to store all the “white” items, which caused loads of dict hash lookups.

So in my mind I crafted a grand vision on how to accomplish this goal. I would adjust all objects (sans numbers) to contain a pointer to some data which would have some header data for the GC to do some bookkeeping in. Great! However, when I implemented this, I found that a number of problems presented themselves: I had to perform a malloc for each and every string that I used, which killed performance, actually making things 2-3x slower. I also noticed that the weird struct I defined was maybe a bit less standards compliant. This attempt was a wash.

So I re-crafted my grand vision. This time I would do the same thing. Brilliant aren’t I? Anyway, the results were basically the same. Who’d've thunk? It was slower again, this time I was quite confused by it, since I had worked around some of the string issues. I also found that the API for creating new strings wasn’t quite as “clean” as my original simple one. This caused some issues in the exception handling mechanism. I had to toss this try as well.

At this point, having re-mangled the code twice and having poor results, I suspected something else might be wrong. My brain was turning into mush. Each time I had completely edited my “tp.h” with all my struct changes in one go. I decided to make a final attempt at reworking tinypy, this time *one* data type at a time. After each data type I added I was able to see if my changes caused any performance issues. I found that my function data type was the culprit. My hashing function (borrowed from lua) wasn’t getting enough entropy and was generating massive collision cases! A few tweaks later, this was resolved. I was able to also craft the string interface to be backwards compatible with the original string interface while also working with the new garbage collection. This “step by step” approach got me to my goal. All said and done, with a bit more tweaking, I was able to *double* the speed of tinypy

Lesson learned - even if it’s only 64k, it’s better to do changes step by step instead of in one big go. valgrind and callgrind are your friends. (Although I found that tinypy doesn’t entirely agree with callgrind … ideas anyone?)

To wrap up this excessively long post about me trying to get code to work — this weekend I’m hosting a Ludum Dare warmup compo. I’m going to give tinypy a run in the “real world”. Here’s to hoping! Next week I plan on releasing the 1.0 version of tinypy.

I’m also thinking about renaming some of my files. And although pylang, dumbparse, and dump2vm have a certain rustic charm, I wonder if I’d do better with names like goat, gorilla, and sausage. Or maybe more descriptive names like tokenator, parsalizer, and bytecodatron.

svn://www.imitationpickles.org/tinypy/trunk for the brave. If you want a zip or an exe, check back in a week. I’ll have all those and more (a game!) Note that I’ve split the SDL dependency out of the main tinypy code. tinypy-sdl.c lets you run my julia.py example. The bootstrapping process also has a final step of compiling with -O3, which I think might not work for everyone. It gives pretty good speed gains on my system, so if it works for you, great!

*python mk64k.py will do a bit of search-n-replace to cut it down to size. I’ve resisted doing anything really ghastly, the code is still indented and readable. See README.txt for more disgusting details on how I cheat to pretend this is 64k.

**Okay, I’ve used some of those features. But hey, this is a 64k implementation, I’ve got to trim the fat.

Yay - I got incremental garbage collection added into tinypy. It took a good deal of troubleshooting, but thanks to valgrind (and thanks to the people who pointed me to it) after some work I got all the memory leaks and crashes worked out. This is a nice step forward, since it enabled me to eliminate the libgc dependency.

tinypy is slightly larger than 64k, but I have vague hopes that will be remedied* at some point. In the meantime, I’m going to take a bit of a break from it. On Feb. 23rd, the Ludum Dare community is having a “warm-up” compo, and I shall give tinypy a real try at that point. After that I will probably put out the 1.0 release.

All that said, its been quite a learning experience for me. I learned how to tokenize, parse, generate byte code, build a virtual machine, and do garbage collection. I’m hoping this will make looking at the innards of other languages seem less intimidating in the future.

* if you’re a clever C / python coder and feel like taking a look at the tinypy code and give me ideas on how to compact the code a bit, I’d sure appreciate it. It’s 4k too big right now, and I’d really like to fix that. I’m cool with any suggestions though I won’t implement anything unless it leaves the code just as (if not more) readable. Please examine the README.txt to find out how I calculate the code size first.

svn://www.imitationpickles.org/tinypy/trunk or tinypy.zip for the brave. I’ve managed to compile it under mingw32 (Minimalist GCC for windows) as well as using GCC under MacOSX. Perhaps for the 1.0 release I’ll include binaries

I’ve managed to build a simple VM into tinypy - modeled after the lua VM. It’s register based and “stackless” in the “it doesn’t use the C-stack” sense of the word. (Not in the, it does anything fancy like “stackless python” does sense.)

‘ve just reached the 64k mark, so that means anything I add into the code will require me to clean up other code to save space. I’ve already done a bit of that with good results. The one rule I follow in shortening code is that the code must retain readability, if not improve it.

Garbage collection is rather complicated, so I think I’ve decided to continue to leave that to libgc. I read some literature on the matter, and it sounds like I could write one, but it probably wouldn’t be very good. I’ve put it at the bottom of my TODO.txt list in the section labeled “Probably not going to happen”.

At present tinypy supports basic python code with functions and loops and lists and dicts and classes. At least, in some basic form. I’m looking towards adding in list comprehensions and *args **nargs to tinypy as those are two of my favorite python features. After that I’ve got a handful of functions I want to write and then some packaging work.

For the curious svn://www.imitationpickles.org/tinypy/trunk - ./run_julia_o3 to see it all happen. It depends on python to compile the bytecode (we’re not bootstrapped yet), libgc, and SDL.

So .. I had a hard time resisting working on my 64k version of “python”. I’ve been able to get quite a few features into it and I’ve gotten my julia demo up to near-C speed as well as tamed the crazy memory problems I was having.
I suppose the question I ponder is “why bother”? Well:

• It’s fun
• I’m learning the basics of parsing, code generation.
• It might even be somewhat useful!

The first two don’t require much explanation, the third (usefulness?):

• By keeping the codebase < 64k, it will be readable by mortals
• By generating C code, it can build self-contained binaries easily
• It has a really simple “FFI” which auto-generates many of the “FFI” wrappers for you
• It’s sort of fast now (no promises for anything real)

It, of course, isn’t python, it just looks a bit like it. Notable differences are:

• Access to members is like lua / javascript. x["y"] and x.y mean the same thing.
• Most infix operators only work with numbers. “x”+”y” won’t work. (Rational: makes numerical math fast)
• It’s missing (and will never have) a bunch of really nice features. Syntax checking is notably weak. Maybe I should scrap my parser, etc and just use python’s.
• No exception handling. Incorrect use of anything will result in a seg fault.

Notable similarities:

• Language contains separate list and dict types. I thought about doing like lua / javascript / php and having a single type, but it just didn’t feel right.
• It’s indented (duh)
• Garbage collection via libgc

Well .. that’s about it. I expect before I’m done I’ll make a game with it, to see how it works in the real world. I’ve got a few more things on my TODO list to get done first. If you are brave, check out svn://www.imitationpickles.org/tinypy/trunk and run ./run_julia (linux) to see the julia demo.

Oh, and for all you “test first” folks, I’ve (more or less) been doing that. It’s made development considerably easier. See the bottom of “pylang.py” “dumbparse.py” and “dumpout.py” for all the testing fun.

I’ve been a bit quiet the last couple weeks .. after my spree of “python->c” converter posts I got pretty busy working on the Ludum Dare 10 website.  Ludum Dare is a 48 hour game development competition.

I used WordPress as the basis for this project and I think it was a pretty good choice.  Generally anything I needed to do, there was a hook in the API to let me do it.  I was able to keep all my compo code (theme voting, trophies, rating of entries,  tag clouds, screenshot grids, security tweaks) all within a module I wrote without having to modify any of the core WordPress files

There were around 150 signups on the site and 50 people completed entries.  The theme was “Chain Reaction” which won even in the first round of voting .. and still won after the 2nd and 3rd rounds.  I spent most of the competition sitting around on IRC doing nothing and occasionally working on my game.

The one notable glitch in the compo was the announcement of the theme ceremony.  I switched the voting to closed so we could see the results and it showed several themes with almost nobody voting for them.  Turns out those were two themes which I initially had in the final round but removed (because they didn’t make the cut).  The results of a few people voting was stored in the database so they still showed up.  Anyway, that was easily fixed.

At the end of the compo it took me an hour or so to get the entry rating system set up.  I also added the ability to leave a comment along with your rating to encourage more people to leave comments.  (They could use the WordPress blog commenting system, but that would take a few extra clicks and thinking.)  This way seems much nicer.

I think the funnest feature I added to the site was the Trophy feature.  This feature lets users award each-other 64×64 pixel trophies at any time.  It’s a nice community feature because it lets people recognize cool things that people did out-side of the pre-set rating categories in the contest.  (For example, one entrant recorded a tuba solo for their game.  Although the compo has a sound category, several people felt that the tuba playing merited special tuba trophies.)

I don’t know if I’ll be running the Ludum Dare competition again or not, but I’m sure I’ll be hosting some others now that I’ve got this swell compo system written

Hey, more fractal fun tonight as I attack PyPy! I must admit, this effort was more challenging - PyPy is pretty big and scary, but I found the folks in #pypy to be quite helpful. exarkun helped me quite a bit in working out the details. (Apparently PyPy doesn’t use ctypes anymore, it uses rffi .. which is sparsely documented.) Anyway with exarkun’s help I got a crude SDL wrapper put together!

Initially I just implemented my julia demo again, but I figured it would make for a boring blog entry to have the same pictures two days in a row. As it turns out, the julia was about 30% faster in PyPy than in shed-skin. Both shed-skin and PyPy appear to have similar limitations. shed-skin’s error messages are less cryptic than PyPy’s. Both FFI’s were somewhat challenging to work with, though I think I prefer shed-skin’s.

If you want to give it a whirl, download PyPy and my junk. The NOTES.txt file lists the command I used to build the executable. Sorry if the mouse interface is a bit lousy, but it was the best I could do in 2 minutes.

There’s this swell python to c++ compiler called shedskin. It’s really interesting. I took a few hours tonight to make it do something:

Yep those are real-time julia fractals If you dare - check it out (make sure python and SDL-devel are all setup):

wget http://www.imitationpickles.org/tmp/juliashed.zip ; unzip juliashed.zip ; cd juliashed ; python ss.py test.py ; make ; ./test

I’ve sent Mark an excessively long e-mail with my comments. I’ll spare you the trouble and just give the highlights:

• My C++ skills don’t exist
• python descriptors are probably the feature I would like to see most - so I could implement pygame.Rect style Rects!
• misc packaging issues (see his previous blog post)
• The README says it “does not scale very well beyond a few hundred lines” - so I guess I’ll wait a while before I try to make a game with it. (Even simple games I make are around 1500 lines.)

Anyway, it’s a cool project. I plan to keep an eye on it.

This just in - I made a python+pygame equivalent. The shed-skin version is ~100x faster! Also, the black areas in the middle of the shapes are caused by my access of out-of-bounds palette entries. I added the -b (bounds check) to the command line and fixed the bug. The bounds check gave me no noticeable speed penalty.