May 12thWriting Our Book

Yesterday, Dori and I approved the final proofs for our new book, Fundamentals of Italian Rapier, which should be available on Amazon and other distributors within the next few days. Today’s post is not fencing specific, but I wanted to talk a little bit about the technical process that we used to write our book. In my day job, I work as a Software Engineer for Dell, mostly focusing on GUI development. When we started working on the project, I couldn’t resist doing some exploration to see if I could come up with a better process suited to what I need. Now, our project has a git repository, a supporting docker image, build scripts, continuous integration, and a very tolerant co-author.

Early Stages

First, some background on how we ended up with this approach. In 2013, Dori and I posted our Introduction for Italian Rapier on this site. Initially, we did a lot of the work in Google Docs, then moved to Word once we were close to being finished. After we released the PDF, I reformatted everything for a 6×9 layout in InDesign as an exercise. This end up being the most updated copy of the file, so a few years later when we started work on expanding it, the InDesign file was our most current file.

A few years later, Dori and I started getting more serious about expanding our curriculum and turning it into a proper book. From a technical perspective, we had a couple of things that we wanted to be able to do:

• Lots of collaboration, often working on the same chapters at the same time
• Ability to quickly rearrange chapters and make major structural adjustments to the book
• Work with a large number of images
• Use consistent styling for fonts, images, tables, etc…
• Generate a print-quality PDF on demand

We ruled out some of the major options:

• Google Docs was excellent for collaboration, and handled lots of images reasonably well, but had fewer options for layout
• Word, might be ok for collaboration as long if we use DropBox and didn’t edit the file at the same time. We also ran into performance issues with all of our images.

So I looked around and found Scrivener, which seemed pretty interesting:

• Can move chapters and content into sections, which can be easily moved around to quickly adjust the structure of the document
• Images are easier because they are usually only a couple within a given section
• Scrivener can export to Word and PDF with predefined settings for layout
• If we keep the file in DropBox we should be able to both edit it.

We moved all of the content from our old InDesign file into Scrivener and started working.  Eventually, we ran into issues that made me start looking around some more

• DropBox isn’t made for simultaneous editing – we still couldn’t work on the project at the same time, and if we did we could easily create some major problems
• The project got pretty large and we started running into performance issues
• Layout options for export were still pretty minimal, and worse, the Word export didn’t preserve actual styles. I started writing macros in Word to try to create something more usable but eventually gave up.

Experiments in Self-Publishing

Around the same time, I got the idea to publish the posts from this blog as a book and started working on Lessons in Italian Rapier Fencing. I needed a way to transform all of the posts from this blog into a usable format. So, I exported the contents of the site to xml, split out all of the posts into separate files, converted the markup to asciidoc and manually downloaded the images that I needed for each article. Asciidoc can output to DocBook, which can then be converted to PDF using Apache FOP. There were a number of guides for this and creating some initial styling wasn’t too difficult. The big thing about this approach was using a markdown language for editing – as a programmer this is my preference anyway, and I had been looking around for something that might let me work this way reasonable well. I decided to go with Asciidoc for the following:

• More features than any other markdown language I could find:
  • Support for complex tables
  • Auto numbered captions on images and tables
  • Auto numbered footnotes
•  Specifically designed to output DocBook files, so all of the major elements of a book (front matter, glossaries, biblography, etc…) are built into the language
• Can use file includes – making it easy to quickly change the structure of the book
• Several output options and pretty good tooling

I initially did a quick run as two separate books, one of articles and one of lessons, and then went back to work on a combined version. Since the ultimate output was an XML file that could be run through Apache FOP to create a PDF, most of the styling changes were done through writing XML transformations. The overall process is: Asciidoc -> DocBook -> FO -> PDF. This was a bit of a learning curve, but I perservered and managed to come up with something reasonable after a few weeks of working on it and published the book through CreateSpace.

Final Approach

After finishing the other book, I was pretty happy with the overall approach – we could write in a text editor (with extensions for Asciidoc that lets us preview our work), run a script, and have a book. All we need to do is add it to git (I used gitlab because it allows you to have private projects for free) and we can get all of the benefits of source control. The styling options were still limited though – it turns out that Apache FOP has not implemented all of the features of the FO spec, and certain options, like floating an image were simple not possible without switching to a commerical (and extremely expensive) rendering engine. Any other complex image layouts were difficult and required writing XML code directly into the source files. At this point, I am sure someone reading this (if anybody is even reading it still) is ready to tell me to just use LaTeX. Asciidoc -> DocBook -> LaTeX -> PDF is possible, and I did experiment with it. I also hated it more than the previous option which I didn’t think would be possible. (I also went down a pretty long rabbit trail trying to get drop caps to work)

I mentioned earlier that I work as a GUI developer. It would be nice if I could use some of the same tools for formatting our book that I use in my job. I started experimenting with a new rendering process: Asciidoc -> HTML -> PDF. CSS supports the basics of page number, margins, etc… A commerical rendered, Prince XML, adds additional features that are not supported by CSS such as page numbers in a table contents, footnotes, settings for widows and orphans, line breaks and other features. It is also free for non-commercial use, so I could use it in development, and then use a subscription service, DocRaptor (which uses Prince as its backend) to create the final, production files.

The new toolchain let me use Chrome for a lot of basic layout debugging (paragraph formats, spacing, etc…), and Prince for page numbering, margins, footnotes, etc… This approach vastly simplified the layout work I was doing and let me work with familiar tools. Instead of writing XML transformations, I could just SASS. I could do most of it with the styles that Asciidoc generated, but it is also easy to add a style in Asciidoc that will be present in the generated html. Now, instead of having to write custom XML into our source files for any complex layouts, I could just add a style name and figure it out in css. The new approach made it possible to quickly style things that I would never have been able to do in the previous approaches.

As we continued working with this approach, there were a couple of new things to consider. While the HTML generated by Asciidoc was pretty good, it wasn’t quite what I needed. I found another set of templates that got it closer to what I wanted, but even then I had to customize it. I was also working on the project from multiple computers and I needed a consistent environment to build the book with. It was also getting difficult keep track of the latest copy of the PDF. I solved the environment issue by creating a docker image with all of the tools and templates I needed in order to build the book. I also used GitLab’s pipeline feature to set up an automatic build for the project – any time we push our changed, a new PDF (and a smaller, compressed one that was easier to share), would be created.

Results

I did most of this work as a hobby and wasn’t really expecting to use this for our final book. In the end though, we did decide to go with it and I’m fairly happy with the overall results (and appreciative that Dori was willing to put up with all of this). Like any project, there are always things that can be better, and I’m sure I’ll look at parts of the book and wish I’d done something differently. Hopefully I’ll be able to make some progress with whatever we work on next!

Share on Facebook