Ideas & Analysis

Discussions of various issues including suggested solutions to problems and pure speculation

The Last Box

We moved here from Arizona in 2003, and (as usual) it took us literally years to unpack everything. Some stuff was not meant to be unpacked, really–I left my vinyl collection and 8″ reel-to-reel mix tapes in boxes on the big shelf in the mechanical room, knowing they’d be there if I needed them but not actually expecting to need them. (I admit, I’ve gone looking in the boxes for a vinyl album a couple of times.) But there’s one box on the high shelf here in my office, containing stuff that was in odd places in my Scottsdale office, stuff that I wasn’t really sure where to put or what to do with. Every so often I sift through the box for an hour or so, trashing some stuff and filing some stuff and putting the rest of it back in the box. It’s only about 1/4 full now, so I guess I’m making some progress. It should be empty by the time I’m 80.

One of the items was a favorite cartoon, from brilliant Maine cartoonist George Dole (George La Mendola) 1920-1997. Dole did a lot of work for the Saturday Evening Post, which is where the cartoon I show here came from. Year unknown; I’d guess the late 1960s. (The slogan “God is Dead” went viral in 1966, when it was the topic of a cover story in Time Magazine.) He did a lot of cartoons for both Playboy and the Wall Street Journal, which many of you probably didn’t realize even ran cartoons. (They do one each issue, in a well-hidden department called “Pepper…and Salt.”) Dole’s is one of only two cartoons that I would be willing to frame and hang on my office wall, and the other one is already there, signed by the artist. My copy is lousy, with one corner torn off, but I may frame it anyway, or perhaps photoshop it up a little and print it on new paper.

Oh, and the cartoon below, which goes back to 1973 and used to be stuck to my bedroom door when I was in college and writing unfinished novels with pompous titles like The Beast of Bronze. Does anybody here even remember the name of the strip? (I do–it’s a test for oldguyness these days.)

Other oddments include a piece of faded green paper on which I scribbled the information for the interview I had with Xerox in September 1974, which led to my first full-time job; business cards from Xerox, PC Tech Journal, and Turbo Technix; a deck of FORTRAN Hollerith cards containing a program I wrote in high school; and a small plastic stock of holy chrism that Bp. Elijah of the Old Catholic Church FedExed to me in 2003 when I was depressed over losing Coriolis, with the message: Anoint yourself and move on. Oh, and a broken Handspring Visor. Fan letters not from flounders. Several of those stupid lanyards that used to come with every single thumb drive you could buy. Uncle Louie’s discharge papers from the Coast Guard.

Things like that. Everything that would easily fit in one of my existing file folders is already there. (I now have one for “cartoon clippings.”) The rest of it, well, I just don’t know. I think everybody has a box of stuff like that, and there should be a good, terse word for the concept. I’m willing to hear suggestions.

At the Sign of the Green Cross

Aug 28th, 2010

by Jeff Duntemann.

2 comments

The closest retail cluster to our house (a mile and a half down the hill) has a fair number of vacant storefronts, but the last time Carol and I went down for lunch at China Wok, we noticed that the storefront right next to the restaurant was no longer vacant. Who had moved in was unclear: There was no big sign, and the small sign under the portico was painted over and blank. It looked like a doctor’s office: a couple of couches and chairs and some flower arrangements in a waiting room with a receptionist’s window. If I were hipper (and I am about as hipless as they come) the neon sign in the window would have given it away immediately: a green neon cross with orange letters in the middle, reading “OPEN.”

Fast forward a week or so. Carol was reading the free paper, and in the back were something like fifteen or twenty ads for medical marijuana dispensaries, including one in the Safeway Plaza. Yup. That’s it.

You’re probably expecting a tirade here, but alas, you lose. I’d paid little attention to the whole issue (let’s just say that I am not a potential customer) but a little research left me fascinated by the speed with which this all got going after a referendum here made Colorado one of the most grass-friendly states in the union. Suddenly it’s an industry, to the extent that somebody is manufacturing green cross neon “OPEN” signs that I doubt you’ll see in the window of a dry cleaners.

The green cross has become the informal symbol of the organized medical marijuana industry. I think this goes back to a primordial cannabis delivery service in San Francisco, which publishes an online menu and will bring the goods right out to you.

Is this a good thing? On the balance, probably. I am fiercely against depriving the ill of medications just because they’re psychoactive. Furthermore, I’ve read a lot of history, and prohibition just doesn’t work. All it does is enrich and embolden the bad guys. And although the one time I (successfully) smoked the stuff back in 1971 I felt depressed and creepy for days, I may not be typical, and feeling a little creepy is probably better than dying in agony or going blind. Certainly I was grateful for post-surgery painkillers when I had a hernia fixed, even though they made me feel stupid enough to enjoy “The Dukes of Hazzard” on TV.

The bottom line is this: The violence of our reaction to marijuana seems out of scale to its hazards. I know that it sounds like a cliche, but I’d rather see it regulated (and yes, taxed) and ideally merged in with the rest of our prescription drug distribution mechanism and treated no differently than chemicals like codeine.

Anyway. That’s just me; I’m easy. It’ll be interesting to see what happens to the place next to China Wok when some of my fussier neighbors figure out what they’re doing down there at the Sign of the Green Cross.

Rant: Higgsism and the Moral Dimension of Health

Aug 16th, 2010

by Jeff Duntemann.

12 comments

As most of you know (or can guess) I’m not content to accept received opinions about things like health insurance reform. I’ve been researching it and working it out for myself for a couple of years now. Most of the discussion online has been tribalist bullshit and not particularly useful, but I’ve managed to define a few things that make the whole issue of health insurance a lot harder to deal with than otherwise might be. What surprises me the most is that these issues almost never come up.

The first of these, in fact, I had to coin a name for: Higgsism, from the clueless protagonst of Samuel Butler’s wicked little gem, Erewhon. If you recall, while Higgs is jailed in Erewhon, he befriends his keeper’s daughter, Yram. Higgs watches, astonished, as people who fall into bad health are convicted of a criminal offense, yet people who have been caught embezzling are treated as though they were suffering from a headcold. When Higgs himself gets a headcold, Yram scolds him severely, and only at that point does he put two and two together. As poor Higgs puts it: “I never remember to have lost a cold so rapidly.”

It’s a great book; a sort of steampunk Gulliver’s Travels, and bears close reading. In a nutshell, what I call Higgsism is this: the belief that we have complete (or almost complete) control over our health, and that when we get sick, it’s because we have done something wrong, making illness our own damned fault. Just as the lucky prefer to ascribe their success to hard work, the healthy generally ascribe their health to pure clean livin’.

Alas, the more we look, the more evidence we see that health is more luck than skill. Matt Ridley’s 2004 book The Agile Gene describes the emergent nature of the human body, and how we’re at the mercy of not only our genes but also poorly understood environmental stressors of gene expression that come into play starting at the moment of conception. Beyond avoiding a handful of obvious hazards like smoking, recreational drugs, and promiscuity, there’s not a whole lot we can do. Eat moderately, walk a little, and get your sleep–but hell, if body weight is almost 80% heritable, health may be a steep climb indeed.

I’ve lived long enough to see a fair number of people die for no known reason. A healthy, trim, athletic nonsmoking man like Harry Helms with no family history of colon cancer dies of it. What did he do wrong? Carol’s late Aunt Berenice lived a modest life and never held a lit cigarette, yet she died of lung cancer. What did she do wrong? The truth is that they did nothing wrong at all. Yet these days, when somebody gets cancer, everybody starts thinking back to try and identify what the poor slob’s sin was.

The truth is grim: We control little of our own health, and what little we do control is often misunderstood (like carbohydrate metabolism) and not always universally applicable across the human species. (Milk is great for you–if you can digest it.)

The health insurance industry can only get away with medical underwriting because of the implied moral culpability of the unhealthy: If you’d just lived a cleaner, healthier life you wouldn’t have cancer or diabetes or ALS or whatever, so you’re a poor risk and deserve to go bankrupt and die. This widespread belief is why high-deductable catastrophic health insurance is unpopular: People see it as money going out of their pockets directly into the pockets of heedless reprobates who can’t or won’t adopt a healthy lifestyle, whateverthehell that is, while those who practice clean livin’ still have to pay for their own broken ankles and flu shots.

As long as we continue to believe that, we’ll be unwilling to face the truth: Health insurance is a sort of luck tax. The lucky pay the money while getting little back in terms of benefits. The unlucky get their lives saved through expensive treatments that they could never afford out-of-pocket. The moral dimension of health is almost entirely an illusion.

What’s the solution? I didn’t say I knew of one; in fact, I’ve often wondered if universal health insurance as we understand it is even possible. Yet even if it is possible, as long as we embrace Higgsism, I guarantee you we’re not going to get anywhere with it. You might as well give antibiotics to embezzlers.

There’s another part to this, which I’ll try to get to in coming days.

What Dogs Gave Us

Jul 30th, 2010

by Jeff Duntemann.

17 comments

We domesticated dogs. And dogs, in return, made human civilization possible.

Work with me here. A lot of my recent reading has been about human origins, stemming from my fascination with Homo Neanderthalis and what became of him. Two books of note: The Third Chimpanzee by Jared Diamond (1993) and Before the Dawn by Nicholas Wade (2007.) Jared Diamond is always a good read, and even though the book is showing its age I strongly recommend it. Wade covers much of the same turf, but does so with the tools of DNA analysis that simply didn’t exist twenty years ago, when Diamond was doing his research. By counting mutations and working backwards through Y (male) chromosomal DNA and mitochondrial (female) chromosomal DNA, we can infer a great deal about human populations, where they came from, how they changed, and when. Of some of it I’m dubious–the extrapolation about the sources of human language, for example, seems a stretch–but most of it is no longer controversial, nor even exotic.

Both authors draw on anthropological research of stone-age peoples who survived into the 20th century. (Diamond did a lot of that research himself, in New Guinea.) The picture they paint of early humanity is grim: We are not fallen angels. We are risen apes. The hallmark of early humanity was deliberate genocide: New Guinea tribesmen told Diamond straight-out that their overall tribal goal was the extinction of other tribes. The homicide rates among such tribes are many times that of the homicide rate in Detroit; men who cannot claim to have killed another man often cannot persuade women to marry them. This seems to have been the pattern for hunter-gatherer societies as far back as we can see via the fossil record. Many Neanderthal skeletons show the marks of multiple healed bone and skull fractures, and a couple of them evidence of spear impingement on bone. Constant warfare was the pattern, and the method (judging from modern stone-age peoples) was the dawn raid: Raiders would stealthily draw close to a rival tribe’s encampment, and wait for the rivals to turn in. Then, when there was just enough dawn light to move well, the attackers would fall upon the sleeping rivals and spear them where they lay.

This worked, and worked well. People have to sleep, so the attackers had the advantage. Then one day about 15,000 years ago, something unexpected happened: Animals around the rival encampment sensed the attackers creeping in for the kill, and set up a huge and unfamiliar racket. The rival group, awakened by the animals, grabbed their spears and gave chase. The attackers had been up all night waiting for just the right moment. The defenders had just had a good night’s sleep. They could outrun their sleepy-eyed assailants, who had a ways to go to return to their home turf. More than a few attackers probably took a spear through an eye socket, and once enough of your dawn raiders take a spear through an eye socket, dawn raiding becomes a lot less compelling.

All because of some previously unknown animals who looked like wolves but made noises that wolves did not make–and appeared to consider the rival camp to be friends rather than food.

As best we can tell, dogs were first domesticated about 15,000 years ago, which was just about the time that Homo Sapiens was moving from wandering hunter-gatherer societies to settled societies that eventually became agricultural and pastoral societies. Just how they were domesticated is still unknown, but the work of Belyaev and his silver fox suggests simple selection by temperament: Ancient wolves became camp followers, and ancient humans tossed them scraps. Wolves who could stand to be near humans ate better without working as hard and had more pups. The few stone-age tribes we’ve been able to study sometimes captured wild animal juveniles and kept them as entertainment until they became grouchy on maturity. Dogs need to be handled as puppies to be fully at peace with humanity as adults; perhaps those wolves-in-transition descended from adult wolves who were handled by humans as pups and remembered: Those two-legged whatchamacallits handled me without hurting me–and they toss me aurochs bones!

15,000 years ago, that was a helluva deal if you were a wolf.

Explaining the bark is tougher, but group selection suggests that if some quirk in the genes of certain wolves allowed those two-legged whatchamacallits to survive and thrive, there’d be more aurochs bones and more yappy wolf/dog pups. Evolution works fast: Belyaev turned wild fox into peculiar (if not completely domesticated) pets in only 40 years, simply by selecting fox who were most willing to be handled when young and least snarly and aggressive when mature. A fox who will lick your face instead of biting your nose off is most of the way to a dog anyway; in another hundred years, he’d be sleeping at the foot of your bed and fetching tennis balls.

The bottom line is this: Without dawn raids, settled living rather than wandering became possible, and settled living fostered the development of villages and agriculture and trade and writing and all the other precursors of the lives we live today.

The Neanderthals had bigger brains than we do. What they didn’t have were dogs. And, lacking dogs, the unfortunate louts dawn-raided one another to extinction, leaving homo sap and his faithful yappers to pick up the turf and eventually take over the world.

Raise a glass of Laughing Lab Ale to canis familiaris: Everything we are we owe to him. Good dog!

App Inventor for Android

Jul 13th, 2010

by Jeff Duntemann.

6 comments

Whoa. Yesterday morning Google took the wraps off App Inventor, a visual development environment for the Android mobile OS. I’m still trying to slurp from the firehose, even though I’m finding that all the hoses have basically the same information, and in truth not a great deal of that. But I’ll tell you right now: It stopped me in my tracks on the iPad decision. As of yesterday morning, I wanted something that runs Android. The new search is on.

You know me. I’m the Visual Developer guy, and the fact that my magazine’s been dead for ten years doesn’t change that. I still believe that visual metaphors for programming are not only useful but necessary, if certain kinds of software development are to happen at all. (More on this below.)

If you haven’t looked into App Inventor at all yet, a very good place to start would be Jason Kincaid on TechCrunch. He’s got a good overview and some screenshots (including the one I show above) that will give you a sense for what Google’s cooking up. I’ll summarize here. App Inventor has two major subsystems:

The Designer is basically a form designer, not conceptually different from that in Delphi, VB, and many other more recent environments. You drag UI components from a palette and arrange them on a form.
Far cooler (if less proven in its approach) is the Blocks Editor. Here’s where program logic happens, and it happens by snapping together logic blocks that look literally like jigsaw puzzle pieces. Clusters of blocks become event handlers. You connect a cluster to an event generated by a component on the form, and the blocks in that cluster execute.

(This may not be the correct jargon. Please understand that I don’t have an instance to play with yet, so all I can do is relay what I’ve read from the fortunate few who were given early copies.)

I knew what the major problem was going to be before Jason told me: In any system like this, you’re limited by the selectable elements on your palette. He didn’t mention where the blocks come from (I assume they’re written in Java using some relative of the MIT Open Blocks technology) nor whether user-created blocks will be importable into the product as shipped. I’m a lot less worried than he seems to be about this, because Google isn’t stupid, and they know damned well that the system lives or dies by the richness of the set of available logic blocks from which the apps are generated. If it’s anything like an open system, there will be an explosion in third-party blocks once a few Java guys get the system and figure out how to do it.

Jason provides some screenshots with his article, and I borrowed one above to get your attention. Here’s another page with a description of a more complex app, with a much more representative Blocks Editor display.

There’s not a lot more that I can say about App Inventor itself, at least until I can get a workable instance installed here. But it’s been interesting seeing all the dorks in the comments to the news stories, dumping on the system for its simplicity, and for the (frightening) possibility that the hoi-polloi will be able to use it to write their own software. They scream the obvious: You can’t write a word processor with a tool like this!

Fersure. And that’s not what it’s for. I’ll respond with something that should be equally obvious: The mobile phone environment is fundamentally different from the desktop environment. From the beginning, it’s been about smallish apps that do one or two things of interest, and no more. Mobile phone computing for the most part is about getting in, doing something with a few quick clicks, perhaps reading the screen, and getting out. The apps are very focused and often extremely specialized. Some are obviously going to be a lot more difficult to write than others, but a useful mobile app does not necessarily require man-years of development time.

And if it ever did, it won’t anymore once App Inventor hits its stride.

I think that I’ll use App Inventor for the same reasons that I use Delphi: To play with ideas, see how things work, and gen up one-time test apps that may lead in useful directions. I’m guessing that App Inventor will enable people to create apps for an audience of one–themselves–and not have to spend six months of free time to do it. Companies may experiment with different approaches to mobile computing without having to commit millions of dollars in dev costs to any one approach, just to see if it’s useful or even doable.

I have never had a smart phone, and I’ve been waiting for my current cell provider contract to expire early next year before getting one. I may have to accelerate the schedule a little. This thing’s making me itch in places I haven’t itched in for a long time.

Realtime Cloud Logging to Spot Band Openings

Jul 3rd, 2010

by Jeff Duntemann.

8 comments

(Note: This is a total ham radio geek-out entry, so if such things make your eyes glaze over, be advised that there’s an extreme glaze warning in effect until at least tomorrow morning.)

Anyway. I stumbled on a band opening yesterday by accident: I scanned the 6 meter band, expecting its usual near-silence, and instead heard something like a continuous pileup from 50.2 up to 50.6. Such openings happen semiregularly, especially in the summer and during sunspot maxima, but they’re not reliably present when you want them. Typically, people either monitor the bands for openings using a panadaptor (a way to visualize the whole band at once, often built into high-end radios) or they hear about it from their friends via Skype or some other chat system. (Hey Jeff! 6 is going batshit nuts!)

While copying my notesheet to my log last night, I thought of a better way. Suppose there were a sophisticated Web app allowing people to record their contacts in a central database off in the cloud somewhere. Serious contesters work their radios with both hands on a keyboard these days anyway, but they’re logging their contacts locally, on their own PCs. If enough people were logging enough contacts online in realtime, you could plot those contacts on a map as great-circle lines between one station and another. If you wanted, you could age the plots, so that a given line was displayed on the map for a selectable period of time, say the past fifteen minutes. Older plots would vanish and new ones would be continually added. What you’d have is a lookback time window onto what’s happening on the ham bands, plotted geographically. If you click on the “6 Meters” map and alluva sudden there’s a thick web of lines between Colorado and the east coast, you’d know that there’s a band opening underway.

This would be possible in part because the geographical coordinate locations of stations are implicit in logged contacts. Base (at home) stations are licensed by the FCC to particular addresses, and these addresses are matters of public record, easily queried by software. Mobile stations aren’t required to be at any particular location, but GPS logging for mobiles is possible, and I think has been done, if not commercially. Plus, there’s another way: More and more people (especially on higher bands like 6 meters) log the “grid squares” of the stations that they’ve worked. There’s a system for tagging 2 degree by 1 degree rectangles of the Earth’s surface, such that each rectangle has a 4-character callout. (There are an additional two characters of precision that almost no one uses.) My own is DM78. Here’s a map for the US and for the Earth as a whole. Plotting a line between DM78 and EM94 isn’t hugely precise, but it will tell you that radio signals are propagating usefully between central Colorado and northern South Carolina, and that’s all most of us need to know to make us scramble downstairs and turn the radio on.

I think this is one case where doing something out in the cloud that was previously done locally provides benefits that local storage alone does not. The whole point is to brag about how many locations you’ve worked worldwide, so privacy is not an issue. (If it is, just keep your logs local.) And the benefit of online collaboration is knowing just what propagation paths are open at any given moment of the day. I’d pay a quarter for that, or at least provide data by logging contacts.

I looked around just now to see how close we are, and whereas there are a couple of online logging systems in operation, they are nothing even close to realtime, and none that I can see makes any attempt to plot propagation paths for logged QSOs. That said, nothing I call out here is rocket science.

So. Did I miss something somewhere? And if not, what Ajax wizard is going to give this a try?

Query By Sketching

Jun 25th, 2010

by Jeff Duntemann.

9 comments

Earlier today, while Carol and I were out on an errands run, we were stopped for a light behind a beat-up pickup truck. On the back of the truck was an emblem on a sticker, and Carol asked me what it was. And in truth, I don’t know, though I’ve seen it a time or two before. It looks like a band logo, though not of any band that I’ve ever listened to.

It was a right profile of a cartoonish man running, with his hair streaming in the wind. In one hand he’s holding a sheet of paper in front of him. The whole thing is in red, inside a red circle. There’s no text of any kind. (The figure is filled in with red; my sketch above is in red Sharpie. Also note the painfully obvious: I’m a words guy, not a pictures guy.)

The challenge intrigued me when I got home. How would I look something like that up? I tried text descriptions in Google Images: “little red guy running”, “red logo running man,” and so on. Saw lots of interesting things, but not that logo. I didn’t spend a great deal of time on it and gave up after a couple of minutes.

We don’t really have a search system for pictograms, and we probably don’t need one all that badly, but it made me wonder how we would make the attempt. Text descriptions? “Right-side profile of cartoon man running, holding a sheet of paper in front of him. Enclosed in circle. Color solid red.” Or perhaps a sort of Visio interface where we could drag across cartoon fragments of describable things and drop them into a rough sketch that the computer could compare against images in its database. This would require machine abstraction, but would be useful for identifying more than just band logos.

As with “query by humming” for music that sticks in your head, it’s a difficult problem computationally, and not as useful. My guess is we that won’t do it, not because we can’t, but because there’s no payoff. And thinking about it for a few minutes reminds me how really really far we still are from genuine “strong” AI.

In the meantime, does anybody know what the little running red guy represents?

Coding vs. Compiling EPubs

Jun 21st, 2010

by Jeff Duntemann.

1 comment

It’s always unsettling to admit that the other side has a point, but it’s good practice and often absolutely necessary. I am the VDM guy, after all, and I’ve never been one for hand-coding what can be generated automatically. As I’ve mentioned here earlier, an awful lot of people take their text and hand-code an EPub framework around it to create an ebook, which I found borderline ridiculous…until this morning. Now I think I know why they do it.

It’s simple: Our EPub compilers have a very long way to go.

The process of creating EPub-formatted ebooks can be done two ways: Write your own XML/XHTML by hand, or let a utility of some sort generate it for you. I’ve done both in recent days, and I was bowled over by the conceptual similarities between that and the gulf between writing a program entirely in assembly and writing it in an HLL like C. I’ve done a fair bit of tracing through assembly code as compiled by GCC, and I’ve been very impressed by the cleanness and comprehensibility of the assembly files it produces. GCC is one helluva compiler, as is the Delphi compiler. (And that’s where my low-level code tracing experience begins and ends, mostly.)

Well, I’ve been spoiled. Compared to GCC (or even Delphi, which is now 15 years old, egad) the EPub format is a babe in diapers: poorly understood, still growing furiously, and, as often as not, smelly as hell. All of that will pass. (I remember my nephew Brian in his diapered era; he is now 27 and an investment banker.) But in the meantime, well, the immaturity of the EPub technology must be dealt with.

I did another, larger test case EPub yesterday. I took a 15,000-word article from an old theology journal, extracted the text via ABBYY PDF Transformer, cleaned up the text (which was in fact pretty damned clean to begin with; ABBYY does a superb job here) and loaded the text into the Atlantis word processor. Without a great deal of additional editing, I exported it to an EPub file. That file may be downloaded here. (40K EPub.) There are no images, and all the text exists in a single XHTML section. It’s about as simple structurally as an EPub can get, and what you see is just as it came out of Atlantis. I did not tweak it at all post-Atlantis, neither manually nor in Sigil. (Note well that Atlantis can export EPub, but it cannot import EPub files, nor display/edit EPub XML/XHTML.) I then took that file and loaded it into Sigil, added a cover image, and split the text into two sections. You can find that file here. (1 MB EPub.) Both of these files pass EPubCheck without errors.

The Atlantis EPub renders (reasonably) well in all the local readers I have here, as well as the online Ibis Reader. It’s small (only 40K) and if you can do without a cover it’s a perfectly reasonable ebook. The Sigil copy does not do nearly as well. The online Ibis Reader refuses to render any of the images at all, including the cover image, the copyright glyph, and the generated images of the two grapevine glyphs that I inserted into the title page as decorations just to see what would happen. The copyright glyph issue is disturbing for legal reasons, but worse, it’s a standard character with a standard HTML encoding, and should be renderable irrespective of font. Ditto Azardi, which renders the Atlantis EPub well but not the Sigil copy. Over and above Azardi’s leaving out all the images (including the copyright glyph) the Sigil copy of the EPub loses what little formatting it had in the Atlantis EPub. None of the centered text remains centered, for example.

There are some additional weirdnesses in the readers themselves: FBReader renders both files well, but (weirdly) the Go Forward button moves the reading window toward the beginning of the file, and the Go Back button moves the window toward the end of the file, perfectly bass-ackwards. Ibis displays the title three times, which is overkill. FBReader handles the images just fine, but renders the copyright notice for both versions in Greek letters, sheesh.

These rendering issues are probably reader failures, since the files themselves are EPub-compliant. However, the autogenerated XML/XHTML code is often obscure, and in one case, at least, dead wrong: The title tag includes only the first line of the title. I understand that the title text is split into two lines, but I was never asked to define the text within the title tag and can only assume that Atlantis picked the first Heading 1 style it found and plugged its text into title. (The metadata for the title was stored correctly, and all readers displayed the full title text. I don’t think that the title tag is used by the readers. An empty title tag is perfectly acceptable to EPubCheck.) The gnarliest part of the compiled EPub (in both versions) is the CSS. Atlantis took the page format settings and translated them into generically named CSS classes, which are accurate representations of the word processor settings, but not easily identifiable and in no wise good quality CSS.

This isn’t insurmountable, and most of the problems I’ve had so far can be blamed on incomplete and buggy reader apps, but it shows how young a business this is. The hand coders still have the edge, and I’d be better off on the readability side creating the ebook text in a WYSIWYG HTML editor like Kompozer or Dreamweaver and hand-coding the CSS myself. That is, however, precisely what I’m trying to avoid. Sooner or later, Atlantis or something like it will offer pre-written CSS style sheets designed specifically for text intended for EPub export. That will help a great deal. In the meantime, some manual futzing is unavoidable, and my opinion of Sigil has been greatly tarnished. I may have to try something else on the EPub editor side; suggestions always welcome.

And the readers, yeech. Don’t get me started. I may have to buy an iPad just to see what my own damned books look like!

Atlantis and the EPub Toolchain

Jun 18th, 2010

by Jeff Duntemann.

2 comments

You’ve heard me say this before, and I suspect you’ll hear it again and again: Creating ebook files is much harder than it needs to be, and creating ebooks in the EPub format is particularly–and inexplicably–hard. In my June 9, 2010 entry, I spoke about the EPub format itself, and how it’s not a great deal different from a word processor file format. In fact, Eric Bowersox pointed out that OpenOffice’s ODF files are also based on XML and organized in a similar way.

Bogglingly, most people appear to be hand-coding EPub XML. In recent days I’ve been looking for better ways to create EPub ebooks. Many places online cite Sigil as the only WYSIWYG EPub editor in existence right now, and I grabbed it immediately. It’s a very nice item, but appears to be an undergraduate’s Google Code project, and I certainly hope he will hand it off to others if he ever gets tired of hammering on it. Version 0.2.1 has just been released, and it fixes a number of bugs that I stumbled over in the last couple of weeks that I’ve been using it.

Then, yesterday, without any need for ancient maps or Edgar Cayce, I found Atlantis.

The Atlantis word processor is a $35 shareware item created by a very small company in France. It’s portable software, meaning it can live on a thumb drive and does not have to be installed in the usual fashion. It’s tiny; nay, microscopic (the executable is 1.1 MB!!) and lightning fast. It doesn’t have all the fancy eye candy of modern software, but it’s amazingly capable, and highly focused on the core mission of getting documents down and formatted. It has a spellchecker and other interesting features like an “over-used words” detector. It reads and writes .doc, .docx, and .odt (ODF) files, and here’s the wild part: It exports to EPub.

Furthermore, it does a mighty good job of it. I loaded a .doc of my story “Whale Meat” into Atlantis and then exported it to EPub. The generated EPub file passed the very fussy EPubCheck validator immediately with flying colors. Now, this was pure text, without any images or embedded fonts or other fanciness, but that’s ok. You have to start somewhere, and I would prefer to start with a genuine word processor.

I then loaded the EPub file that Atlantis had generated into Sigil, which I used to divide the story into chapters and add a cover image. Sigil isn’t really a word processor in the same sense that Atlantis or Word are, but it allows split-screen editing of WYSIWYG text on one side and XML/XHTML code on the other. Sigil 0.2.0 had a bug that generated an incomplete and thus illegal IMG tag (XHTML requires the ALT attribute) but I see that the new 0.2.1 release fixes that. Adding the ALT attribute manually in Sigil 0.2.0 allowed the EPub file to pass EPubcheck without further errors.

I have not yet generated a TOC in Sigil, nor have I attempted to create an EPub of any significant size. (“Whale Meat” is only 8,700 words long.) When I’m through playing around, I’m going to load the entire .doc image of Cold Hands and Other Stories into Atlantis, export it to EPub, semanticize it in Sigil, and see what I have. At some point along the way I may be forced to hand-code (or at least hand-correct) the XML or XHTML, and you’ll hear me bellyache about it when I do. But I will admit that I’m pleased with what I have so far. Yes, Atlantis and Sigil ought to be one product, or at least two closely-knit utilities in the same product family. Still, given the primitive state of the EPub reader business (I have yet to find a Windows or Linux-based EPub reader that I’m willing to use) I’m satisfied with the way that Atlantis and Sigil cooperate. Now that Apple has anointed the EPub format for iBooks, I’m guessing that EPub-related improvements will be arriving thick and fast in coming months.

EPub and Word Processors

Jun 9th, 2010

by Jeff Duntemann.

11 comments

Well. Got your heart medicine handy? Jeff is considering a Mac. Well, not exactly. (Put down that nitroglycerine.) I’m strongly considering getting an iPad. And I’ll bet you didn’t know that I already have an iPod, thanks to Jim Strickland, who may in fact persuade me to get a Mac someday. I worry about some of Apple’s cultural issues (like not providing clear guidelines on what you can sell in their stores and what you can’t, and changing your &!$#*% mind about it every other week) but their engineering is extremely good. I spent some quality time with an iPad at a recent Enclave Meetup, and basically, I’m sold. Those guys pretty much nailed the ebook experience, or at very least came up with the best possible compromise between fixed-page and reflowable presentation that anyone might strike. And I want my books out there in the iBooks marketplace.

This means that I need to be able to create EPub files, and good ones. What boggles me is the scarcity of visual tools for that purpose. Among the mainline desktop publishing apps, only InDesign CS4 and CS5 can export finished EPub files, and some people think the feature itself isn’t finished yet. (I don’t have either version so I can’t do my own testing–and at $700 for the app, I don’t expect to get it.) Some odd comments I’ve seen online suggest that the Scribus developers don’t think that reflowable document export is a suitable task for a fixed-layout desktop pubber, and that they’re not going to do it. There are lots of converter programs for taking various types of files and turning them into EPubs. As best I can tell, most people code their EPubs up manually, as though they were writing a C++ program. Gakkh. But also as best I can tell, affordable WYSIWYG EPub editors begin and end with Sigil.

The format itself is not a skullcracker. You’ve got one or more XHTML files expressing content (plus image files, if present), one or more CSS files defining styles, and one or more XML files describing document structure and metadata, all placed in a container file that’s not much more than a .zip with a different extension. There’s an optional DRM layer in the spec, but it’s technology-agnostic and not much used. The spec is simple enough so that people write the damned things by hand. I can’t imagine that parsing and generating the XML/XHTML/CSS would strain any sort of editor.

My point here is that you don’t need a fixed-layout desktop publishing program like InDesign or Quark to create and maintain EPub ebooks. In a sense, EPub is a modern XML-based word processor file spec, and even a middling WYSIWYG word processor could be twisted a little bit to read, render, edit, and write EPub files that could be loaded right into iBooks without further processing.

Sigil comes close. I’m using it and I’m reasonably impressed, considering that the team is basically writing a brand-new word processor from scratch. What boggles me is that it’s the only WYSIWYG EPub editor in the universe. And as a word processor, well, it’s pretty spare.

There’s no reason for this. Existing word processing apps like OpenOffice Writer and AbiWord could easily be extended to import and export EPub files, or forked to create a ramcharged ebook development system using EPub as its primary file format. Fork or not, I’m convinced of this: All word processors will eventually become ebook editors. The ebook market is closing in on reality. We now have the file format we need. The software will follow.

But sheesh guys, how about picking up the pace a little!