Jeff Duntemann's Contrapositive Diary Rotating Header Image

ebooks

Coding vs. Compiling EPubs

It’s always unsettling to admit that the other side has a point, but it’s good practice and often absolutely necessary. I am the VDM guy, after all, and I’ve never been one for hand-coding what can be generated automatically. As I’ve mentioned here earlier, an awful lot of people take their text and hand-code an EPub framework around it to create an ebook, which I found borderline ridiculous…until this morning. Now I think I know why they do it.

It’s simple: Our EPub compilers have a very long way to go.

The process of creating EPub-formatted ebooks can be done two ways: Write your own XML/XHTML by hand, or let a utility of some sort generate it for you. I’ve done both in recent days, and I was bowled over by the conceptual similarities between that and the gulf between writing a program entirely in assembly and writing it in an HLL like C. I’ve done a fair bit of tracing through assembly code as compiled by GCC, and I’ve been very impressed by the cleanness and comprehensibility of the assembly files it produces. GCC is one helluva compiler, as is the Delphi compiler. (And that’s where my low-level code tracing experience begins and ends, mostly.)

Well, I’ve been spoiled. Compared to GCC (or even Delphi, which is now 15 years old, egad) the EPub format is a babe in diapers: poorly understood, still growing furiously, and, as often as not, smelly as hell. All of that will pass. (I remember my nephew Brian in his diapered era; he is now 27 and an investment banker.) But in the meantime, well, the immaturity of the EPub technology must be dealt with.

I did another, larger test case EPub yesterday. I took a 15,000-word article from an old theology journal, extracted the text via ABBYY PDF Transformer, cleaned up the text (which was in fact pretty damned clean to begin with; ABBYY does a superb job here) and loaded the text into the Atlantis word processor. Without a great deal of additional editing, I exported it to an EPub file. That file may be downloaded here. (40K EPub.) There are no images, and all the text exists in a single XHTML section. It’s about as simple structurally as an EPub can get, and what you see is just as it came out of Atlantis. I did not tweak it at all post-Atlantis, neither manually nor in Sigil. (Note well that Atlantis can export EPub, but it cannot import EPub files, nor display/edit EPub XML/XHTML.) I then took that file and loaded it into Sigil, added a cover image, and split the text into two sections. You can find that file here. (1 MB EPub.) Both of these files pass EPubCheck without errors.

The Atlantis EPub renders (reasonably) well in all the local readers I have here, as well as the online Ibis Reader. It’s small (only 40K) and if you can do without a cover it’s a perfectly reasonable ebook. The Sigil copy does not do nearly as well. The online Ibis Reader refuses to render any of the images at all, including the cover image, the copyright glyph, and the generated images of the two grapevine glyphs that I inserted into the title page as decorations just to see what would happen. The copyright glyph issue is disturbing for legal reasons, but worse, it’s a standard character with a standard HTML encoding, and should be renderable irrespective of font. Ditto Azardi, which renders the Atlantis EPub well but not the Sigil copy. Over and above Azardi’s leaving out all the images (including the copyright glyph) the Sigil copy of the EPub loses what little formatting it had in the Atlantis EPub. None of the centered text remains centered, for example.

There are some additional weirdnesses in the readers themselves: FBReader renders both files well, but (weirdly) the Go Forward button moves the reading window toward the beginning of the file, and the Go Back button moves the window toward the end of the file, perfectly bass-ackwards. Ibis displays the title three times, which is overkill. FBReader handles the images just fine, but renders the copyright notice for both versions in Greek letters, sheesh.

These rendering issues are probably reader failures, since the files themselves are EPub-compliant. However, the autogenerated XML/XHTML code is often obscure, and in one case, at least, dead wrong: The title tag includes only the first line of the title. I understand that the title text is split into two lines, but I was never asked to define the text within the title tag and can only assume that Atlantis picked the first Heading 1 style it found and plugged its text into title. (The metadata for the title was stored correctly, and all readers displayed the full title text. I don’t think that the title tag is used by the readers. An empty title tag is perfectly acceptable to EPubCheck.) The gnarliest part of the compiled EPub (in both versions) is the CSS. Atlantis took the page format settings and translated them into generically named CSS classes, which are accurate representations of the word processor settings, but not easily identifiable and in no wise good quality CSS.

This isn’t insurmountable, and most of the problems I’ve had so far can be blamed on incomplete and buggy reader apps, but it shows how young a business this is. The hand coders still have the edge, and I’d be better off on the readability side creating the ebook text in a WYSIWYG HTML editor like Kompozer or Dreamweaver and hand-coding the CSS myself. That is, however, precisely what I’m trying to avoid. Sooner or later, Atlantis or something like it will offer pre-written CSS style sheets designed specifically for text intended for EPub export. That will help a great deal. In the meantime, some manual futzing is unavoidable, and my opinion of Sigil has been greatly tarnished. I may have to try something else on the EPub editor side; suggestions always welcome.

And the readers, yeech. Don’t get me started. I may have to buy an iPad just to see what my own damned books look like!

Odd Lots

  • The rate of toxoplasmosis infection in a given nation appears correlated to the level of neuroticism in that nation. I’ve mentioned toxo before, but it appears that we have better numbers now, and that the UK is nowhere near 50% infected, as the source I quoted in 2003 implied. France, well, now…
  • Here’s a nice piece that explains why the atmospheric CO2 measurements taken atop Mauna Loa are accurate. And here is NOAA’s explanation of how they do it.
  • Bruce Baker sent a link to an article describing how a few scraps of odd film allowed some dogged engineers to re-create the long-lost pallophotophone technology (later known as RCA Photophone), and by doing so give voice to 1920’s recordings of Thomas Edison speaking affectionately about his friend Henry Ford.
  • This weekend is ARRL Field Day, in which ham radio ops head out to the hinterlands to see how well they can get up to radio speed from a dead stop, using portable (not mains) power, from a place not previously set up for radio gear. I’ll be trekking out to my back deck and working the world on an inverted vee, draining 829Bs (Diet Mountain Dew over ice in peanut butter jars) Saturday afternoon, and showing all my geek friends how it works that evening during one of our semiregular geek parties.
  • We can look forward to the Roman Missal on iBooks soon. But will the priest process down the center aisle holding an iPad over his head?
  • I’ve been testing Windows-based EPub-capable reader apps for the last week or so, and guess what: They all suck. Bigtime. Why is it so hard to render reflowable documents that are basically HTML-in-a-sack?
  • If you’ve seen The Music Man as often as I have, you’ll remember how there’s trouble in River City, because the kids are memorizing jokes out of Captain Billy’s Whizbang. Well, I’d long thought that Meredith Willson had made it up, but not so: I found a scan of a 1921 issue of Captain Billy’s Whizbang (which is now in the public domain) and put it up in my pub directory so you can see it too. (Note: It’s a 15 MB .cbr.) My reaction? It’s not very funny, but in a world without Lileks, I guess people laughed at whatever they had on hand.
  • Ok, there’s a little profanity in it (like that’s unusual in Slashdot comments?) but damn, I like this one.

Atlantis and the EPub Toolchain

You’ve heard me say this before, and I suspect you’ll hear it again and again: Creating ebook files is much harder than it needs to be, and creating ebooks in the EPub format is particularly–and inexplicably–hard. In my June 9, 2010 entry, I spoke about the EPub format itself, and how it’s not a great deal different from a word processor file format. In fact, Eric Bowersox pointed out that OpenOffice’s ODF files are also based on XML and organized in a similar way.

Bogglingly, most people appear to be hand-coding EPub XML. In recent days I’ve been looking for better ways to create EPub ebooks. Many places online cite Sigil as the only WYSIWYG EPub editor in existence right now, and I grabbed it immediately. It’s a very nice item, but appears to be an undergraduate’s Google Code project, and I certainly hope he will hand it off to others if he ever gets tired of hammering on it. Version 0.2.1 has just been released, and it fixes a number of bugs that I stumbled over in the last couple of weeks that I’ve been using it.

Then, yesterday, without any need for ancient maps or Edgar Cayce, I found Atlantis.

The Atlantis word processor is a $35 shareware item created by a very small company in France. It’s portable software, meaning it can live on a thumb drive and does not have to be installed in the usual fashion. It’s tiny; nay, microscopic (the executable is 1.1 MB!!) and lightning fast. It doesn’t have all the fancy eye candy of modern software, but it’s amazingly capable, and highly focused on the core mission of getting documents down and formatted. It has a spellchecker and other interesting features like an “over-used words” detector. It reads and writes .doc, .docx, and .odt (ODF) files, and here’s the wild part: It exports to EPub.

Furthermore, it does a mighty good job of it. I loaded a .doc of my story “Whale Meat” into Atlantis and then exported it to EPub. The generated EPub file passed the very fussy EPubCheck validator immediately with flying colors. Now, this was pure text, without any images or embedded fonts or other fanciness, but that’s ok. You have to start somewhere, and I would prefer to start with a genuine word processor.

I then loaded the EPub file that Atlantis had generated into Sigil, which I used to divide the story into chapters and add a cover image. Sigil isn’t really a word processor in the same sense that Atlantis or Word are, but it allows split-screen editing of WYSIWYG text on one side and XML/XHTML code on the other. Sigil 0.2.0 had a bug that generated an incomplete and thus illegal IMG tag (XHTML requires the ALT attribute) but I see that the new 0.2.1 release fixes that. Adding the ALT attribute manually in Sigil 0.2.0 allowed the EPub file to pass EPubcheck without further errors.

I have not yet generated a TOC in Sigil, nor have I attempted to create an EPub of any significant size. (“Whale Meat” is only 8,700 words long.) When I’m through playing around, I’m going to load the entire .doc image of Cold Hands and Other Stories into Atlantis, export it to EPub, semanticize it in Sigil, and see what I have. At some point along the way I may be forced to hand-code (or at least hand-correct) the XML or XHTML, and you’ll hear me bellyache about it when I do. But I will admit that I’m pleased with what I have so far. Yes, Atlantis and Sigil ought to be one product, or at least two closely-knit utilities in the same product family. Still, given the primitive state of the EPub reader business (I have yet to find a Windows or Linux-based EPub reader that I’m willing to use) I’m satisfied with the way that Atlantis and Sigil cooperate. Now that Apple has anointed the EPub format for iBooks, I’m guessing that EPub-related improvements will be arriving thick and fast in coming months.

EPub and Word Processors

Well. Got your heart medicine handy? Jeff is considering a Mac. Well, not exactly. (Put down that nitroglycerine.) I’m strongly considering getting an iPad. And I’ll bet you didn’t know that I already have an iPod, thanks to Jim Strickland, who may in fact persuade me to get a Mac someday. I worry about some of Apple’s cultural issues (like not providing clear guidelines on what you can sell in their stores and what you can’t, and changing your &!$#*% mind about it every other week) but their engineering is extremely good. I spent some quality time with an iPad at a recent Enclave Meetup, and basically, I’m sold. Those guys pretty much nailed the ebook experience, or at very least came up with the best possible compromise between fixed-page and reflowable presentation that anyone might strike. And I want my books out there in the iBooks marketplace.

This means that I need to be able to create EPub files, and good ones. What boggles me is the scarcity of visual tools for that purpose. Among the mainline desktop publishing apps, only InDesign CS4 and CS5 can export finished EPub files, and some people think the feature itself isn’t finished yet. (I don’t have either version so I can’t do my own testing–and at $700 for the app, I don’t expect to get it.) Some odd comments I’ve seen online suggest that the Scribus developers don’t think that reflowable document export is a suitable task for a fixed-layout desktop pubber, and that they’re not going to do it. There are lots of converter programs for taking various types of files and turning them into EPubs. As best I can tell, most people code their EPubs up manually, as though they were writing a C++ program. Gakkh. But also as best I can tell, affordable WYSIWYG EPub editors begin and end with Sigil.

The format itself is not a skullcracker. You’ve got one or more XHTML files expressing content (plus image files, if present), one or more CSS files defining styles, and one or more XML files describing document structure and metadata, all placed in a container file that’s not much more than a .zip with a different extension. There’s an optional DRM layer in the spec, but it’s technology-agnostic and not much used. The spec is simple enough so that people write the damned things by hand. I can’t imagine that parsing and generating the XML/XHTML/CSS would strain any sort of editor.

My point here is that you don’t need a fixed-layout desktop publishing program like InDesign or Quark to create and maintain EPub ebooks. In a sense, EPub is a modern XML-based word processor file spec, and even a middling WYSIWYG word processor could be twisted a little bit to read, render, edit, and write EPub files that could be loaded right into iBooks without further processing.

Sigil comes close. I’m using it and I’m reasonably impressed, considering that the team is basically writing a brand-new word processor from scratch. What boggles me is that it’s the only WYSIWYG EPub editor in the universe. And as a word processor, well, it’s pretty spare.

There’s no reason for this. Existing word processing apps like OpenOffice Writer and AbiWord could easily be extended to import and export EPub files, or forked to create a ramcharged ebook development system using EPub as its primary file format. Fork or not, I’m convinced of this: All word processors will eventually become ebook editors. The ebook market is closing in on reality. We now have the file format we need. The software will follow.

But sheesh guys, how about picking up the pace a little!

Odd Lots

  • I’m not very good at one-liners. So, in my contrarian fashion, I will present an Odd Lots composed entirely of…two-liners.
  • Technical material (textbooks, manuals, computer books) rendered on an ebook reader? Now you’re talking.
  • As someone fond of both astronomy (especially telescopes) and Star Wars, I consider this a wonderful building hack.
  • Harrison Bergeron was evidently a Canadian kid soccer player. (Thanks to Bob Trembley for the link.)
  • What’s your favorite app for extracting text from PDFs? Any experience with ABBYY’s PDF Transformer?
  • And if you’re going the other way, slow but sure pays off: PDFCreator has finally reached version 1.0, after only seven years.
  • Sigil is the only WYSIWIYG editor for EPUB-format ebooks. Why? When will we start editing ebooks and stop coding them?
  • One of my cousins once had a sandbox in an enormous worn-out tractor tire. Now somebody’s recycled such a tire into a bike.

Odd Lots

CBZ Files as Image Archives

Last fall, I gathered a stack of Alma-Tadema‘s paintings from my pre-1923 images folder, wrapped them up into a ZIP file, and sent them to a friend who was looking for a copyright-free color cover for a novel. Some weeks ago, I learned that the CBZ (Comic Book Zip) file format is nothing more than a ZIP file with a different extension. I downloaded and installed a free CBZ reader called Comical. After changing the extension on the Alma-Tadema archive to .cbz, I double-clicked on it, and boom! There it was, beautifully presented and trivially easy to click through. And if you change the extension back to .zip, you can de-archive the images in the usual fashion using any ZIP-capable archiver. It’s all in the extension; no changes to the binary archive need to be made.

Not being a comics guy, I’d never heard of the CBZ format, though it’s been around since 2004. It’s basically an ebook reader protocol (since it is, after all, simply an ordinary ZIP archive) that opens a .zip file and displays the files in alpha order by filename. If the files are displayable as images, the reader displays them. If the files are not displayable as images, a well-behaved reader will ignore them. (Comical, one of the simplest free readers, sometimes crashes when it encounters a non-image binary.) If you need an indicia page, some readers will display text if it’s in an .nfo file. The .nfo will appear in a separate text window on opening the file, rather than in the page display area.

I’ve tested four free CBZ readers: ComicRack and Comical under Windows, and QComicBook and Comix under Linux. All but ComicRack are open-source. ComicRack is overkill in a lot of ways, though it works very well. (It requires the .NET framework, if that’s significant to you.) Comical is much simpler, and my only gripes are that it doesn’t display .nfo files, and it crashes when it finds certain kinds of non-displayable files in a .cbz archive. QComicBook is a Qt4/KDE app, and the one I find myself using under Linux. Comix (a Python app) works well but is not as capable as QComicBook. (Feature-wise, it’s on a par with Comical.) Others exist. Okular will open CBZ files without complaint, but it simply scrolls vertically through the images without attempting to show one per click.

Most of the comic book readers also read CBR and CBT files, which are RAR and TAR archives, respectively, and work almost exactly the same way. (I haven’t tested those formats.)

The CBZ system works best when all the images in the archive are the same dimensions and aspect ratios. I’m putting together some photo albums for showing the folks back home that are collections of digital photographs in one (big) .cbz file. The bigness is mostly unavoidable, since JPG files don’t compress very well. Still, it makes file management simpler

Here are some sample CBZ archives that I put together for testing: Alma-Tadema (14 MB). Hi-Flier Kite Catalog 1977 (6 MB). The “Elf” Space-Charge Receiver (1.7 MB).

Odd Lots

Odd Lots

  • Here’s a great article from NASA on the unexpected success it’s had with the WISE (Wide-field Infrared Survey Explorer) spacecraft in spotting previously unknown asteroids in the infrared spectrum. WISE is detecting hundreds of new asteroids every day, which is unnerving, since a rock no bigger than a Motel 6 could cause regional devastation greater than any nuclear weapon yet produced.
  • From Larry Nelson comes a pointer to the AirStash, an interesting $100 USB Wi-Fi gadget that can accept up to a 32 GB SD card and act as a content server over Wireless b/g. Anthough nominally a thumb drive, the USB plug also charges the internal battery, and (though it’s not screamed from the rooftops) the thingie works all by itself, no computer connection required. This suggests “wearable file sharing”: Drop one in your pocket and nearby people can download files from the device without having any idea where it actually is. Little by little, the jiminy (an AI wearable computer I thought up in 1983, and figured would be mature by 2027) creeps toward realization. The AI is actually the tough part; everything else already exists, if not in as small a package as I imagined 25 years ago.
  • And if you ever wanted to run Linux on one of your fillings (ok, one of your elephant’s fillings) this would be the solution. (Thanks to Bill Cherepy for the link.)
  • Here’s a gadget that builds you an external USB storage device by dropping in (literally) a naked SATA hard drive. I may not need it, but I admire the elegance of the concept.
  • I’ve been arguing in favor of dual-screen reader devices for years, and this one is a good start. Sounds like the user interface software needs work…but when has that not been an issue for a first-gen device? We’re closing in on it, though.
  • Nice status update on some of the current non-Tokamak fusion research approaches, link thanks to Frank Glover.
  • Also from Frank comes a reasonable article on how people would die in a vacuum and how they wouldn’t. I had heard of lung shredding; heart failure was new to me. But take, um, heart: Your blood wouldn’t boil.
  • If you ever wondered why you cry when you slice onions, well, it’s the sulfuric acid released by cells in the onion when they’re cut open. Supposedly living things evolved this mechanism (or at least key parts of it) half a billion years ago. Onions evolved their chemical weapons to avoid being laid on hamburgers in slices–but we evolved Vidalias to prove that we were smarter than onions, and that fast food will prevail against all threats.
  • Interestingly, the Canon G11 camera reduces the size of the image sensor to 10 megapixels, down from the 12.5 on the G10. The new sensor gives you fewer pixels but better ones, and faster, which is all for the best.
  • Burger King is testing a new retailing feature in Brazil. When you order a burger, they take your picture and print your face on the burger wrapper.

Odd Lots

  • Here’s the best discussion I’ve yet seen on why Flash may never work well–or perhaps at all–on touchscreen devices like the iPad.
  • Most recent laser printers have Ethernet ports, and some older printers (like my Laserjet 2100TN) can accept a JetDirect network adapter. Installing a printer on a network port means you don’t have to worry about whether the machine it’s attached to is turned on. If you’d like to do this but you’re not a network geek, here’s the best XP-based step-by-step on the topic I’ve ever run across. Same tutorial for Windows 2000.
  • Bruce Baker passed me a link to a nice item on the issue of broadening publisher book production to allow all formats to be generated from a single master file. Follow and read the link to The New Sleekness as well. Pablo should take it down a notch; XML is not a markup language; it’s a general mechanism for creating markup languages, and what may happen eventually (perhaps in ten years or so) is a standard book-production markup language derived from XML and built into a new generation of word processors. Still, what nobody in either article mentions is the problem of pages verses reflowable, which is the 9 trillion pound gorilla in the business. If you don’t solve that problem, absolutely nothing else matters. (And it is not as easy to solve as some may claim–I’ve been thinking about it for several years now and see no solution whatsoever on the horizon .)
  • Kompozer 0.8b2 has been released. I just got it installed in a VM and will be poking at it in coming days. According to Kaz, most of the changes are code cleanups, but any progress on the editor is a fine, fine thing.
  • I’ve done model rocketry here and there over the (many) years, and I’ve seen some very odd things lofted on D engines. Back in high school, my friend George built a Harecules Guided Muscle (which was from the Beany & Cecil cartoon show) in the form of a big whittled balsa wood fist on a short, thick body. I’m amazed it flew as well as it did. Well, here’s a fire-’em-together pack of 8 rockets shaped and colored like Crayola crayons. The guy took his time (six years) but he did a great job–and created a spectacular Web page documenting the project.
  • We rarely go to WalMart, but last time we did, I picked up a bottle of Diet Mountain Lightning. It has nothing on Kroger’s Diet Citrus Drop, easily the best of all the Diet Mountain Dew clones I’ve ever had the opportunity to try.