Editing of EPUB files - Mass correction - Removal of mid-word breaks

Is there a tool that will perform a task that is similar to spellcheck, but is only looking for mid-word breaks and prompt for correction (optional skip manual approval)?

For example, if a word in the text is found to be

excel lence

when that should be

excellence

it would flag or identify that word for the stripping of the mid-word space (or spaces or tabs or newlines).

I don’t know if tools that look at defined “breakpoints” in words could identify the words that have such “breaks” that should be removed or “healed”?

Does such a tool exist?

3 Likes

Hey @ericmarceau,

I’m not trying to speak as an expert in this response, but as (hopefully) a pointer in a helpful direction. TL;DR: I wasn’t able to find a tool that does exactly what you’re asking (hopefully someone else will).

My initial attempt at looking for a tool, I was able to find a tool called calibre that seems to have some editing capabilities. I installed it, opened up the default epub book that ships with the software, and plugged in your excel lence example (I just picked a random .html file and added the example in, nothing fancy). Then I tried the spellchecker. As expected, it only tried to correct lence to lance. So that doesn’t seem helpful. :frowning:

Calibre does have plugin support, but I wasn’t able to find anything helpful at first glance. Maybe I missed something.

At my current job, I’ve used Natural Language Processing (NLP) by using Python’s Natural Language Toolkit (NLTK) to process customer raised support cases to identify gaps in the knowledge base. In my experience with that project, I had to do quite a bit of spelling checking/word fixing.

Using that angle, I did some googling and ran across this Stack Overflow post with a helpful gist to a Python segmentation script. This functionality seems to both be able to split long strings of characters into words and also remove breaks/spaces between words. I also see that Python has an epub library.

It seems pretty reasonable that a Python script could be put together that handles your functionality for identifying the broken words, offering solutions based on a score, then healing them if you choose to do so.

Maybe some of this information can help you in your search for a tool?

4 Likes

Short answer, no single tool that does exactly that, at least not one I could find. But a few things get close, and honestly the DIY version is tiny.

The reason spellcheck whiffs on this is that hunspell and aspell look at one token at a time. Feed them “excel lence” and they see a real word followed by junk, so you get a squiggle under “lence” with suggestions like “lance” or “fence”. Nothing in there ever considers gluing it back onto the word in front of it. You need something that looks at pairs.

Stuff I found that actually does it:

LanguageTool is probably the best bet off the shelf. It has rules for wrongly split words and it works on token pairs instead of single tokens. Runs local as a server or through languagetool-commandline.jar. English coverage is decent but not complete. German is much better supported, which tracks, since split compounds are basically a national sport over there.

symspellpy has lookup_compound and word_segmentation. Dictionary plus frequency based, and it handles both bad splits and missing spaces. Closest thing to a library that just does your problem.

dehyphen, Python, comes out of the pd3f project. Built specifically for this on text pulled out of PDFs. It scores each join or no join call with a language model instead of a dictionary, so it does better on the ambiguous ones.

pd3f itself if your source is PDFs and you want the extraction pipeline to rebuild paragraphs properly on the way out.

Word and Google Docs will catch some of these in grammar check if it’s a short doc and you just want to eyeball it.

DIY version:

The logic is maybe six lines. Walk adjacent token pairs, and if the joined version is in the dictionary, flag it. Then the filter that kills basically all the false positives: only flag it when the second fragment is not a real word on its own. “excel lence” fires because “lence” is garbage. “may be”, “in to”, “a while” and “all ready” all stay put, and those are exactly the ones a dumb joiner would destroy.

For the approve or skip part, cheapest thing is to dump the candidates and run them through vim with :%s/old/new/gc. That gives you y/n/a/q per match for free. Or build the prompt into the script and add a --yes flag for when you don’t feel like babysitting it.

One thing on scope though. Tabs and newlines are the easy case. A word broken across a line ending, especially with a trailing hyphen, is a different problem than a stray space in the middle of a line, and you’re usually better off fixing that at extraction time than trying to fix it after the fact.

5 Likes

Thank you, Benjamin. What you offered seems to be leaning in the right direction.

I do have both Calibre and Sigil. But their “spelling” functions don’t deal with the scenario I’ve outlined.

Basically, I’ve “picked up” a poorly-reworked version of a book and it is riddled with such broken-word instances.


Those Stack Overflow and gist references were interesting as an awareness of “shotgun” approaches, but I was looking for something with more finesse like the case

  • When being offered an award of excel lence, he was left speachless.

and trying to identify instances of “adjacent” words that should be joined. For that, I don’t know if it requires an overlay of recognition of gradually-built sentence structure, to avoid making some context-based poor joins, but I was hoping that the logic would simply identify possibles for manual confirmation of the join, thereby offering the best of both worlds.

:slight_smile:

3 Likes

Thank you! The dehyphen and pd3f options look like they might be workable … but not for me. I’ve backed away from Python for reasons expressed elsewhere, namely, that the lack of characters delimiting logical code segments have made my every usage a frustrating experience to avoid repetition. Don’t want to go there again! :frowning:

Nonetheless, thank you for bringing those to my attention!

2 Likes