LaTeX-like handling of hyphens and apostrophes

[thinlines]

In some markdown implementations and LaTeX, --- yields an em dash in output, which is very convenient for editing in a text editor. In SILE, there are some different ways to get that output, e.g. use \unichar{u+2026}, define an equivalent command, or copy and paste an em dash. Where should I start looking if I want to modify input in this way? I see the examples section on the SILE site has something about input filters, but I'm not sure if that's the right direction.

Another situation is apostrophes in contractions like don't. I'd like it to be a U+2019 when it occurs in a word (but not in a verbatim environment). I implemented a hack by overriding the SILE.process function from sile.lua and doing regex replacement on the input, but clearly this is not a good solution to the problem. For serious documents, I currently replace them manually.

What would be an idiomatic or more robust way to do such replacements without resorting to wrapping things in environments and so on? Thanks for any pointers you can give!

[alerque]

First, there are a couple of options you missed.

• You can use an IME (input method editor) that allows you to enter the unicode symbols with easy typing. For example on Linux I use the Compose key to input these: tab the key, then a two or three letter shortcut for it. For example Compose--- will output a literal Unicode Em-dash (—), and Compose<" outputs a Unicode left typographic quote (“). This works in any program for me, including this browser text area. Similar systems exist for all operating systems.

• You can use an editor that transforms these. Many editors have plugins that catch various input sequences and transform them. For example I use vim-textobj-quote for typing in NeoVIM and it does a pretty good job at guessing. I can always override it with a Compose sequence if I see it guesses wrong. For em-dash things are even simpler, for prose file I sometimes have an insert-mode abreviation set.

• You don't have to write content for SILE in SIL or XML by hand, you can write in in any language you prefer and either convert it it SIL or XML, or you can teach SILE to understand the format. For example most of my production work I write in Markdown, then use pandoc with the smart extension to transform, then use my Pandoc fork (assorted Pandoc versions here) or binaries built into CaSILE) to convert them to SIL. Another path would be to use Pandoc to transform the input from Markdown to Markdown and then use a Markdown inputter for SILE such as @Omikhleia's markdown.sile.

• You could write an input filter (or package with an input filter) that does this. You will need to consider what input you want to actually transform this way and be cautious about being too aggressive. You might think you want to do it to your whole document, but is all the input in your document actually prose? In any case you could setup a class that always used the filter on everything or a package that wrapped many/most other commands to filter them. I mention this last however because it's going to be a lot harder than you think. Getting smart quote directions right is very hard. It is easy to start down the path, but vary hard to cover all the edge cases.

c.f. The smartquotes.sile package: not directly relevant for this use case, but has some nice quoting features if you wanted to transform your input using it's commands.

[Omikhleia]

Interesting approach @thinlines !
I am not fully sure settings are the right construct for keeping track of the level, as they work in a stack that can be wrapped / pushed / popped, possibly in temporary environments (in other terms, there might be scoping issues). It's a tough design issue though, and I am not that sure either how it should be implemented (typesetter states, maybe?).
The idea is cool nonetheless!

[alerque]

Good point on the scoping. I would probably have used a package-local variable if it is only used by this package, and something in the scratch space dedicated to this package if for some reason it would need to be used by anything but the package.

[Omikhleia]

A local package variable might be tricky too -- what happens for instance in footnotes (eventually from a different typesetter perspective)?

I mean, Say: \enquote{Lorem ipsum\footnote{Also \enquote{amet} sometimes} dolor.} = both cases ought to be quoted rhe same way here.
EDIT: in this very symmetrical case, maybe settings are adequate after all!

[alerque]

Indeed, this highlights our need to completely overhaul the scope of packages vs. typesetters and their respective use of settings and scratch vars. We don't have the correct interface for that use case.

[Omikhleia]

So to fix the scoping and have the footnote work as expected, it should likely be either:

self:registerCommand("enquote", function(options, content)
    local level = SILE.settings:get("csquotes.quotelevel")
    -- Temporary stack scoping
    SILE.settings.temporarily(function ()
      SILE.settings:set("csquotes.quotelevel", level + 1)
      if level % 2 == 0 then
        SILE.call("primquoted", {}, content)
      elseif level % 2 == 1 then
        SILE.call("secquoted", {}, content)
      end
    end)
  end, "Wrap something in quotes")

Or

  self:registerCommand("enquote", function(options, content)
    -- typesetter state hack
    local level = SILE.typesetter.state.enquote or 0
    SILE.typesetter.state.enquote = level + 1
    if level % 2 == 0 then
      SILE.call("primquoted", {}, content)
    elseif level % 2 == 1 then
      SILE.call("secquoted", {}, content)
    end
    SILE.typesetter.state.enquote = level
  end, "Wrap something in quotes")