Comefrom0x10: day 1

Today, I am finally creating a language.

Inspired by Come Here and Come From, Comefrom0x10, pronounced “Come from sixteen”, will be a modern language based entirely around the much-maligned “come from” control structure.

I am writing the first interpreter in Python. I was hoping to have a grammar worked out by the end of day one, but failed to make significant progress. Originally, I thought I would use a parsing expression grammar, but soon reverted to Backus-Naur. The trouble is that I want cf0x10 to include syntactic indentation (as it’s a modern, Pythonic language). This means that I need indent and dedent constructions, which parsing expression grammars can’t easily express.

So, I decide to switch to lalr. As a bonus, using lalr, will, of course, let people easily build million-line cf0x10 programs without worrying about slow compile times.

The Python parser generator landscape is surprisingly bleak. Many of the parser generators available are abandoned or half-assed. Ply seems to be the only real contender. There are two main problems with Ply: first, its mechanism of embedding grammar specification within docstrings makes the parser obnoxiously verbose; second, it doesn’t provide an obvious way to generate a parser that does not require installing Ply.

Ply has a very strong point though: it provides outstanding debug output for diagnosing problems with your grammar. I’ll just live with the verbosity; I might try PlyPlus if it gets too annoying. As far as removing the Ply dependency goes, I’ll wait until the Enterprise customers for cf0x10 want to fork out money to fund writing a build script so I can use the generated parse table without depending on Ply.

Ply’s scanner generator has the usual Lex push and pop state capabilities, plus it makes saving arbitrary lexer state easy. I, therefore, spend a good deal of day one trying to avoid hand-writing a lexer. As usual, this is a waste of time.

The insurmountable problem is that Ply helpfully stops you from providing regexes that match an empty string, presumably assuming they would cause infinite loops. In addition to syntactic indentation, Comefrom0x10 uses syntactically significant blank lines and without either a facility for pushback or empty-matching regex, it is very hard to write a sensible grammar. But that’s another article.

So finally, I just decide to hand-write the lexer and begin to make progress near the end of day one.

Qwerty War: day 5

On day five, I plan to get the leaderboard persistence built and deployed: the only part of the project that requires a database back end. I figure that I’ll build this with Django and deploy on Heroku.

So I switch from JavaScript to Python and naturally this requires that I instate my traditional key customization that maps shift+space to underscore. Previously, I’ve used xmodmap, but the going wisdom around the Internet is that xkb is the newer, better way to do things.

hours pass

I can finally type underscores without over-extending my pinky, so I turn back to the problem of Django. Django normally uses a great many subdirectories to organize things and make them portable, which is great for medium to large applications, but annoying for something as small as Qwerty War. The article Django as a micro-framework is a good start to really stripping Django down, but it’s significantly out of date. To bring it up to date with Django latest, the main thing to realize is the django-admin command line should now read thus:

django-admin runserver --settings settings --pythonpath .

From there, you’ll find there are a series of errors that are all pretty self-explanatory and easily fixed. I skip most middleware, but keep the csrf and clickjacking parts.

Soon enough, Django is serving up my static files: I haven’t built any database stuff yet, but I figure I should try deplying the static bits on Heroku. First hurdle: Heroku wants me to install its command-line tool by piping stuff from wget into a shell. The script would install Heroku’s apt repository and go from there.

Isn’t the whole deployment done with git anyway? Why do I need the heroku command line interface? Much less another dpkg repo. It turns out this is easy enough, if undocumented

  1. Add an ssh public key to your account via the Web gui
  2. Use the Web gui to create a project
  3. Push to the project using the funky-looking location git@heroku.com:[appname].git

Mercurial, of course, does not speak Git natively, so I decide to try to the hg-git extension. Its homepage says to install via the old easy_install. Not an auspicious start, but ok, I just try pip instead, and in Pythonic fashion, it just works.

So far, so good. The first push via hg-git succeeds, but naturally, the app fails to start on Heroku, as I haven’t yet created a Procfile or wsgi.py. I add them, push again, and… nothing. Mercurial reports there are no changes to push. Did my first push really work at all? I figure I’ll try cloning the repo from Heroku to check, using actual Git…. time passes (but much less than spent on xkb) …I figure the problem is that either hg-git or Heroku or both dislike that I pushed some rather large files. In particular, I included a full history of my edits to the binary blobs that are audio files, plus the originals in .wav and .flac format.

Fair enough, pushing all that stuff is pretty wasteful anyway. I abandon the hg-git extension and opt for a simple shell script that deploys via Git proper. It works, so on day five, I at least have a deployable application, if still very much in development mode and without a database.