Why (not) pdf?

Text Collector lets you print text messages by converting them to pdf. What we call “text messages,” of course, includes messages with both text and pictures. Sometimes they include other types of attachments, like dirty .gif files, but that’s another article. For now, I’m just discussing images and text.

On the surface, pdf seems ideal: it’s universally viewable, supports pagination, and, unlike images, includes text in a searchable way. But I don’t really like pdf as an ediscovery interchange format.

Why? Pdf is too complicated. As file formats go, it’s far from the worst monster out there, but it’s also far from simple. As a consequence, the many different programs that generate pdf often get it slightly wrong; they’re not necessarily bad programs: they’re just dealing with a complicated problem and make mistakes.

Pdf viewers grapple with the resultant problems and show you something that looks correct, so everything seems fine at first.

When you want to edit pdf, however, things quickly go wrong. For a typical ediscovery operation like stamping Bates numbers on your pdf files, the small errors compound and you have a significant chance that the result will be illegibly damaged.

Assume some set of pdf files P , and an operation b that you want to do on them to produce an output set, O .

b:P \rightarrow O

You can visually check some number v for correctness.

When the size of O is larger than v there will be some subset E, larger than zero, that is terribly broken.

E \subset O \land |O| > v \Rightarrow |E| > 0

Ulfers’ Law of Batch Pdf Editing

Second, since it is complex, pdf allows all manner of invisible content. This makes redacting pdf hazardous and if you have a highly-developed sense of self-doubt, it’s hard to shake the feeling you’ve done something wrong that allows the redaction to be removed.

So, why does Text Collector convert messages to pdf and not something else?

There are no suitable alternatives. Html is universally viewable but has no notion of pagination and very limited image embedding. Microsoft Word format is too easy to edit, and comparable in complexity to pdf. Mhtml never got universal support and it lacks pagination anyway. Tiffs and text are too large and useless to average people. How about svg?

So pdf it is, for better or worse.

Keyboard Savior Xtreme released

Tired of websites trapping your shortcut keys for their own ends? Me too. Usually, the problem is that they take over the Firefox slash-to-search shortcut. But what can we do about it?

First, we could tolerate it and just use ctrl+f instead. On some slash-abusing sites, like Bitbucket, this isn’t a terrible option: they rarely include long pages that require quick jumps. In api documentation, however, it’s nightmarish.

Second, we could try to stamp out the evil by filing bugs and fighting for justice. There is some hope for this: Github, for example, used to abuse the slash key and no longer does. In general, however, it devolves into Whac-A-Mole. Django rightly rejected slash abuse in in 2008, only to have it sneak into the Django docs in 2015.

Finally, we could just fix it. This Greasemonkey script lets you list known abusers and prevent them from seeing slash keystrokes. After some time, however, I realized that my Greasemonkey approach did not go far enough: it only prevents abuse of one keystroke and only on selected sites.

In fact, there are only a handful of sensible reasons for any website to capture a keystroke, ever. Why not just stop them all?

So, I welcome Keyboard Savior Xtreme. Take back all your keystrokes.

Comefrom0x10 released

After a long hiatus while I built Text Collector, last week I finally returned to my paradigm shifting language, Comefrom0x10. It now has a home page on Read the Docs that features a tutorial, standard library documentation and more.

Except for a couple minor bugs, its implementation was actually functionally complete eight months ago. I hesitated to release it, however, because of rather embarrassing performance problems.

Now, it wouldn’t be fair to say that Cf0x10 is just slow. It’s catastrophically slow. The brainfuck.cf0x10 program takes 10 seconds to run helloworld.b on the laptop I’m using to write this, and gets dramatically worse as the program gets longer.

What went wrong?

It’s not a fundamental problem with the comefrom paradigm, but a consequence of the twisted way the language took on a life of its own during implementation. I started with the idea that I was building a rather ordinary stack-based interpreter, but Cf0x10 would have none of it. As it evolved, the original idea became a disfigured mutant: I can demonstrate with tests that it works, but it’s too convoluted to allow necessary optimizations.

Oh well, as they say, first make it right, then make it fast release it.

Print text messages: video edition

Today I published my first YouTube video, How to print text messages on Android:

I already knew the obvious choices of software to use for some elements: Audacity to record and edit narration, Pixly to draw the hand pointer animation. I’m a novice at making videos, however, so I spent a good deal of time figuring out what program I should use to edit the video.

First, I tried Kdenlive, and managed to put together the entire video how I wanted it, only to run into a fatal error: I couldn’t export successfully.

Eagle-eyed viewers may notice that the part where I demonstrate a purchase doesn’t use an actual currency. There are actually several layers of compositing in this shot:

Purchase screen with generic currency symbol

When Kdenlive attempted to render this, it just produced glitches: it flashed images like the sad face emoji, from completely unrelated parts of the video. No settings tweak I found solved the problem.

So, on to OpenShot, whose interface feels largely comparable to Kdenlive. I soon discovered, however, that it lacked some basic effects that I needed, particularly freeze frame. Apparently version 2 lost a number of effects that were present in version 1.

Finally, I moved to Blender. I guess, deep down, I always suspected it would come to this.

Blender is a ridiculously capable program, especially when you consider how lightweight it is. It manages to include 3D modeling, rigging, rendering, animation, compositing, video editing, a game engine and more in a download between 80 to 150 megabytes, depending on your operating system. Compare to, say Maya, which can take days to download.

How Blender accomplishes this is surely black magic, but it’s not for fear of the dark side that I avoided it till last: it’s the user interface. To the uninitiated, Blender feels like learning how to use a computer for the first time.

Want to select something in the timeline, or, in Blender-speak, the “video sequence editor?” It’s right-click, not left-click. Want to move it? You can click and drag, but it won’t release when you let up on the mouse button. You need to click again to release. Scroll wheel zooms. Ctrl+scroll wheel scrolls. And so on.

In other words, Blender’s interface is comparable to Dwarf Fortress.

Nonetheless, it only took me about day and a half to re-cut my video in Blender. On the bright side, if I ever need to add a 3D Text Collector mascot and some explosions, I’ll already be in the right program.

How Unicode can save math: part 2

It’s widely known that decimal – or “numbers” to most of us – is an inferior system. Decimal doesn’t work well for computers, which prefer base two and it doesn’t work well for humans either, at least not when compared to dozenal.

Dozenal is also called “duodecimal,” or “base 10” (when writing in dozenal) and it is a much more natural system for humans than decimal. The usual example of why is a clock. Look how neat it is with the number 10 right at the top:

Dozenal clock face
From the Dozenal Society of Great Britain

Dozenal has a big problem though, as we can see from the clock. What number does 10 represent, when you see it out of context? You just don’t know.

For decades, we’ve solved this problem in computer programming with funny prefixes. To a programmer, dozenal and decimal might be “base 0xA” and “base 0xC”. Likewise, in a dozenal world, we might write “hexadecimal” as “base 0z14” or something. If we need to start writing all our numbers with warts to indicate the base, however, dozenal seems doomed.

But wait, there’s hope. Unicode already contains the digits for “dek” and “el.” (That’s ten and eleven, if you’re not a cool dozenal kid.) If your browser doesn’t have a suitable font, refer to the clock above. If it does, they look like this:

↊ ↋

Now all we need is nine more Unicode symbols for the rest of the digits. Zero is special: for zero, there need be only one.

Text Collector version one released

Today, Legal Text Collector graduated from beta to production, version 1.0. It’s been a long time coming: I started working on it seven months ago. I made my first notes on the idea about five years ago.

Exactly thirty years ago today, Reagan said, “open this gate. Mr. Gorbachev, tear down this wall,” and in a small way, I think I can sympathize with Gorbachev. The main change from beta to production is that now people can start posting reviews: I’m opening the gates of public criticism.

Google allows public reviews only after a production release. In the safety of beta, Google provides a private feedback option, which nobody used. Many people did, however, send me questions and bug reports via email.

It’s hard to overstate the the value of that feedback. My friend Trish, my aunt Meg and my erstwhile colleague Jon of Sandline were especially helpful in testing the early alpha versions; strangers who stumbled upon later public betas kindly sent me information to resolve some of the final bugs. Through 14 alphas and 16 betas, Text Collector grew from 3.5 thousand lines to 5.5 thousand, a testament to how long the long tail can be.

So it is a dramatic moment. Much remains to do, but at some point I must tear down the wall to public criticism. Private feedback has been overwhelmingly positive, so this doesn’t worry me too much, but only time will tell.

It’s a good day to print text messages

Even though my elevator pitch for Legal Text Collector is, “It lets you print your text messages,” in all the time that I’ve been writing it, I never actually printed any messages.

Text Collector doesn’t (yet) print text messages directly. Instead, it converts them to pdf, and you’re free to copy and print pdf as you please. In development, I just examined output pdf. Many, many times. Printing is a formality.

Still, this gives me an itchy feeling. I find it unsatisfying to leave the final step untried.

So I celebrated the beta 15 release by actually printing messages. On paper. In color and grayscale.

Why celebrate beta 15? I expect it to be the last beta release before 1.0. Every 1.0 feature is in and every 1.0 bug is fixed. In other words, beta 15 is Version One. All that remains is to take a deep breath and bless it with a “production” designation.

 

How Unicode can save math: part 1

Every casual math enthusiast has by now heard of the raging war between tau and pi.

The what?

Ok, I mean, tau’s gaining a little ground, but really, pi has the weight of history behind it, so “raging” and “war” might be overstating things a little. The point is that 3.14, et cetera, is a bad circle constant and there’s a more intuitive option, “tau.”

Take a 90-degree angle. In radians, it’s half pi, but one quarter tau makes much more sense:

A quarter cut out of a pizza
Half pi!

If we call 2 times pi “tau,” this slice is one quarter, and things just make more sense.

Doesn’t seem to be catching on.

Not really, no. And tau has some problems too: for example τ=2π, but the tau glyph has only one leg and pi has two, so it looks like pi is twice tau. Shouldn’t tau have four legs? Clearly, the problem is this symbol: we need something more familiar. How about we just redefine pi to be twice itself?

Madness. Only confusion and chaos can result.

But not if we change the spelling. We’ll say that pie = 2pi.

Mathematicians will never go for it.

Perhaps, but food-based math has rich history. If you’re worried that it lacks a succinct one-character symbol, well, that’s where Unicode comes into the picture. In 2017, we finally have pie emoji, 🥧.

Where have my emoji gone?

Bob, what were your six biggest challenges in developing pdf?

Dave, that’s easy. Fonts, fonts, fonts, fonts, fonts and fonts. In that order.

The most noticeable limitation of Text Collector is that when it converts your text messages to pdf, it drops emoticons. Instead of smiley faces, it displays diamond-question mark things, “replacement characters.”

As ever, fonts are the problem.

When you need to display something laid out just so, such that it prints on a page and everything fits correctly, text alone isn’t enough.

Text, nowadays, usually means some encoding of Unicode. Technically, it’s a “Unicode transformation format,” which is where we get the “utf” in utf-8. Impress your friends and family with that.

Unicode represents abstract concepts, like the idea of the letter A or a smile. Emoji, just like letters A through Z are Unicode text, but precisely how letter A or winky face actually gets written to the page depends on your font.

A A

If you want to ensure that your text fits on the page, doesn’t run over or under the things around it, or, indeed displays at all, you must also include a font. In pdf terms, this is called “font embedding,” and it is what Android’s built-in pdf library does.

So far so good: just embed the emoji font and smiles or piles of poo come along for free, right? Wrong.

Most fonts specify letter outlines and leave color of the letters up to the user. Thus, I can say that I want letter A in blue or orange and both are the same font:

A A

Not so, Google’s emoji font. It includes cats, dragons and piles of poo in glorious, and heavyweight, color. Including this font makes for a massive pdf: almost 10 megabytes, on Android 6. It only takes two or three such documents to blow past Gmail’s attachment size limit. Typical collections would run into hundreds of megabytes, or gigabytes. They could easily consume all available space on the device.

This isn’t a new problem, and there’s a well-known solution: font subsetting. That is, only include glyphs for the small handful of characters that you actually use. Android’s built-in pdf library can’t do that.

So, to get emoji, I have to bring in an entirely new pdf library… and I’ve been working on this long enough. Version 1 needs to ship. Emoji will have to wait till version 2.

Painless Android releases

Android apps require not one, but two version numbers:

  • Version code: an integer that Android uses to check whether one version is more recent than another
  • Version name: a friendly version to display to the user, conventionally something like 1.2.3

This means that when you want to build a new release of your app, you have two things to manually update, and that is two things too many. You will make mistakes.

Luckily, it’s not too hard to automate this away in your Gradle build script.

Gradle inherited much of its design from Apache Maven. Maven defined a standard release feature that automatically handles typical pitfalls and mindless details of making a release: tagging in source control and incrementing your version number. For Gradle, there is a nice third-party implementation, the gradle-release plugin. So long as you don’t fight Maven-style version conventions, it can make cutting releases almost entirely automatic, modulo prompting you to confirm that it guessed correct version numbers.

If your project only has one version number, you just apply the release plugin and you’re done, but Android’s two-version-number system takes some customization.

I only discuss version numbers here, but the release plugin also does several other useful sanity checks.

First, move the versions out of your app/build.gradle into app/gradle.properties. They should look like so:

app/gradle.properties

version=1.0-SNAPSHOT
versionCode=1

app/build.gradle

android {
    // ...
    defaultConfig {
        versionCode project.versionCode.toInteger()
        versionName project.version
        // ...

“SNAPSHOT” is Maven’s convention for “between releases”. Version 1.0-SNAPSHOT means the code leading up to version 1.0. This convention is how the release plugin guesses what version number you are releasing: it just lops off the suffix.

When you run ./gradlew release, the release plugin updates the version thus:

  1. Edits gradle.properties, removing the “snapshot” part
    1.0-SNAPSHOT becomes 1.0
  2. Commits the change and tags this as version 1.0 in source control
  3. Builds the release
  4. Edits gradle.properties again, to next dev version
    1.0 becomes 1.1-SNAPSHOT
  5. Commits so you can immediately start working on version 1.1

Thus, out of this box, this handles the user-friendly version number, but not the “version code.”

Updating the version code

When Android installs an update to an app, it knows by version code whether the update is newer than what it currently has installed. 3 is newer than 2 and so on.

Thus, the obvious strategy for updating your version code is to add one on every release. If using the release plugin, you might do this as a manual step after it finishes a release. If you forget, you’ll accidentally build your next release with the same version code as you just used. If you have other branches, you need to remember to update them as well. Ouch.

There is a better way. Version codes need not be sequential, so instead of incrementing 1,2,3…, we can derive it from the date. A format like [2-digit year][month][day][0-9] works nicely. A release today gets version code 1704080, tomorrow, 1704090.

This format will cover you for 82 years at up to ten releases a day. If that’s not enough for you, use a four-digit year and a two-digit suffix, but watch out for integer overflow in 130 years or so.

The date-based strategy, however, means that you have to set your “version code” immediately before you release, instead of after. To do this, add a Gradle task right before updating version name.

app/build.gradle

task setVersionCode { doLast {
    // Add a task that updates version code
    def current = project.versionCode.toInteger()
    def releaseAs = new Date().format('YYMMdd0', TimeZone.getTimeZone('UTC'))
    if (releaseAs.toInteger() <= current) {
        // More than one release today
        releaseAs = current + 1
    }
    def releaser = project.plugins[net.researchgate.release.ReleasePlugin]
    def propsFile = releaser.findPropertiesFile()
    def props = new Properties()
    propsFile.withInputStream { props.load(it) }
    props.versionCode = releaseAs.toString()
    propsFile.withOutputStream { props.store(it, null) }
}}
// Execute our task before unSnapshotVersion, provided by the release plugin:
unSnapshotVersion.dependsOn setVersionCode

With this simple build script change (plus applying the release plugin), a single command updates both version numbers:

./gradlew release

The release plugin also runs the “build” task at the point of release, so this single command leaves you with both a release .apk and your working directory updated to the tip (snapshot) code ready to start work on the next release. There’s still a problem though: if you haven’t configured your build script to sign the build, you won’t be able to publish the release .apk.

Signing the build

To make Gradle sign a build, you need to add a “signingConfig”:

android {
    // ...
    signingConfigs {
        release {
            storeFile file('/home/myname/.javakeys/mykeys.jks')
            keyAlias 'myappsigningkey'
            // These two lines make gradle believe that the signingConfigs
            // section is complete. Without them, tasks like installRelease
            // will not be available! (see http://stackoverflow.com/a/19350401)
            storePassword "notYourRealPassword"
            keyPassword "notYourRealPassword"
        }
    }
    buildTypes {
        release {
            signingConfig signingConfigs.release
            // ...

This fails, so you put your real password in the “password” config place and get pwned. Your wife leaves you, and your dog dies. You didn’t that, right?

So where should you put your password? The top-voted answer on Stack Overflow says ~/.gradle/gradle.properties, presumably protected by 600 permissions. I don’t see the point. If you’re relying on file system permissions to keep the password secure, why have the password at all? You could just protect the keystore with file system permissions.

What you need is a prompt for the password.

Thanks to bug 1251, Gradle running in daemon mode (the default) doesn’t let you use System.console().readPassword("Password:"). You can disable daemon mode, but then you run afoul of (orphaned?) bug 2357 because Android Studio generates a default gradle.properties that includes jvmargs. Once you remove that configuration, you find that prompts don’t display when you build not in daemon mode (bug 869). That’s a pain because you can’t see the version number confirmation prompts.

As a result of this epic adventure, you’ll eventually find that the only reliable way to prompt for password is via Swing. No, I’m not joking. It’s not as gruesome as it sounds, thanks to Groovy’s Swing builder, so pop over to where Tim Roes documented how to do it.