@BrunoLeFloch I have never used pandoc, but I have looked at its documentation. I found it somewhat confusing on this issue, but concluded that it wasn't fully programmable. I may be wrong, of course.

I should perhaps explain the motivation for my "package". It is so that someone who is familiar with LaTeX (specifically) can write for a blog, forum, wiki, or whatever using their LaTeX skills. So it shouldn't require learning new stuff (such as Markdown, ConTeXt, or whatever-pandoc-can-do) but saying "You can't do X" is a bit more acceptable. In particular, some things that you can do with LaTeX just aren't possible on the web because of how it works.

(Paulo's last post in the editor room confused me as to which room was which ...). Silly question: I installed TL2010 via tlmgr (so not via a distribution package); what's the simplest way to upgrade to TL2011?

@AndrewStacey Two ways I know of. You can upgrade, but this is not guaranteed to work (and doesn't work on Windows). Or you can make a fresh install.

1. supporting LaTeX with no modification (except adding \RequirePackage{...} at the start), which requires that I support every primitive, and that I typeset the document as I collect characters

@AndrewStacey After I read your future blog article, it makes a lot of sense. I'm hesitating between

2. writing a new ?TeX engine (tough), and writing the corresponding ?LaTeX format. This allows me to add or remove primitives freely, but the more I change, the less it will be compatible with other macro packages.

@BrunoLeFloch Whew! Upgrading looks complicated. Presumably if I do a fresh install then it installs in to a separate directory (2011) so I keep my original one. I'm not short of space, so I think that's what I'll do.

@AndrewStacey I didn't try upgrading since I went from Ubuntu's ancient TeX Live 2009 to TeX Live 2011 directly. It can't be harder to upgrade than to write a new engine :-).

@BrunoLeFloch (bizarre: your comments ended up on my screen in the wrong order!). I'd go for 1.; moreover, you don't have to support every primitive, just those that you are likely to encounter in the actual document part. If the goal is just to produce text then some can actually be discarded (almost anything to do with lengths and skips, for example). Moreover, you can be honest at the start about which things are supported and which not, (ctd)

and wait for people to try it out and say "I wish it did X".

@BrunoLeFloch I meant "complicated relative to just installing afresh"! And it didn't make it clear whether there were any actual benefits to upgrading over installing, except that (presumably) the download is considerably lighter.

@AndrewStacey Well, it depends. I'd like \RequirePackage{almostaTeXengine}\documentclass{article}\begin{document}Hello, \textbf{World!}\end{document} to work out of the box. But I don't want to touch any LaTeX macro, only TeX primitives which reach TeX's stomach.

@AndrewStacey True, I think it gives a lighter download.

@AndrewStacey In particular, I need to cater for boxes. I don't have a good example of use of boxes in usual LaTeX macros, but imagine that "\textbf" is defined as "\def\textbf#1{\setbox0\hbox{\bf #1}\box0 }". Then there is no way out of performing the "\setbox" (with some hackery to get a text output), and using it afterwards. But then, what does "using it" mean? Putting it in the current paragraph, or in the recent contributions, or in a bigger box that is being built, etc. (ctd)

@BrunoLeFloch It depends on your goal. My goal for what I'm doing isn't that it take an already existing LaTeX document and be guaranteed to convert it to whatever. But that someone who knows that they are going to write it for whatever be able to do so using LaTeX as an input format.

So I don't intend to support every package available, just simulate those that someone would be likely to use. For many things, I'm thinking "When someone writes \href{url}{text} what do they actually mean? What does that translate to?" So I don't load the hyperref package, I just define \href#1#2{[#2](#1)} (for Markdown).

Very quickly I end up having to get all of TeX's typesetting capabilities. But this is a slippery slope since many macros use "\wd0" and such in tests. Hence I need to typeset boxes as I go. <- and I can't get ligatures to work, so boxes aren't quite of the right width, but well, I don't think I can do anything for that, apart from an engine solution.

@AndrewStacey For sure, some macros will need to be rewritten. For instance, in my no-macro-changing approach, "\begin{figure}...\end{figure}" would produce a float, which LaTeX's output routine (which I also run one step at a time) would then place in some box, somewhere. This placement would have been done using info which is completely irrelevant for a non-pdf output format. So instead "\begin{figure}...\end{figure}" should be indeed redefined. (ctd)

In my approach we'd have one "text-TeX" package/engine, and a "fix-LaTeX" package with rather few redefinitions. But I'd be happiest if running a generic LaTeX file with arbitrary packages could give a sensible result in any case. Then things could be improved for each macro (as you describe, "\def\href#1#2{[#2](#1)}" etc.).

I think that to contribute further to this discussion, I need to have a clearer idea of the purpose for what you are doing. I may just have missed it (in which case refer me back to it), but I'm a little hazy on why someone should use this package (or whatever). That would help me throw out more focussed ideas! (And to compare it to what I'm doing.)

@AndrewStacey I should clarify my goal to myself as well ;-). One problem you faced is that pdfTeX cannot output characters as plain text. Well, it can (\write), but then characters have to come as the argument of some command. So I think that we agree that it would be useful if "\input something Hello\par World!\end" could produce a text file with "Hello<newline><newline>World!" in it.

My original motivation for the same text-output thing is the following. I wanted to make TeX expand some macros selectively in a source file, and output another source file. I more or less did this, but it was clunky, and it would be easier to just let characters slip to the left and get "typeset" to a plain text file.

Another possible motivation is TeX4ht, which I believe does something close to what you do with pdftotext, but through the dvi format. It may be simplified if the output could be done as a text file directly.

@BrunoLeFloch That's certainly what I would use it for, and I would find that very useful as I could bypass some of the trickery I'm having to do to get pdftotext to work faithfully. Ah, looking at your "original motivation" then that certainly occurred to me too and that would be extremely useful. But again, only certain macros are going to make sense in this scenario.

(My concern would be that you spend so long trying to get it right that you don't actually get anything. Whilst, as a mathematician, I completely understand that attitude, I can see that it has its drawbacks in everyday life!)

@AndrewStacey I understand :-). Not sure I'm a mathematician anymore, though. But that's another question.

@AndrewStacey An extended motivation is that often my father recieves a poorly formatted source file, with a different set of common macros than he likes (e.g., \newcommand\cN{\mathcal{N}} instead of \newcommand\Ncal{\mathcal{N}}), or very long lists of useless macros. The task would then be three-fold: expand the other guy's macros ; contract macros that he uses ; reformat the source nicely (80 char per line, spacing, etc) in a way he specifies. This shouldn't be too hard once we have text output.

@AndrewStacey Actually, do you have a typical file that you send through pdftotext? It would help me decide what to support first. Also, why did you need to keep spaces at the beginning of lines? To get an indented output?

@BrunoLeFloch Indentation is significant in Markdown. For example, when in a list then indentation says that the next paragraph is still a part of the list. For the nLab, it turns out that the exact indentation is important so I need to be able to control it exactly.

@BrunoLeFloch What do you want, the original LaTeX source or the resulting PDF?

@AndrewStacey The LaTeX source. I won't be working with anything else than text files.

@AndrewStacey A priori it will be completely trivial to control spacing with what I'm doing. For instance using the LaTeX macro \@spaces to add 4 spaces anywhere you like. I can also make \hskip <N>em produce, say, <N> spaces (or whatever, I'll see).

@BrunoLeFloch But the problem I was having was that pdftotext did not interpret the resulting spaces correctly, no matter how they were specified.

@AndrewStacey If you add, say, a dot at the beginning of every line, would pdftotext ignore that space? You can then remove all those spurious dots.

@BrunoLeFloch That's effectively what I'm doing. Each line starts with a pattern X*: where the number of Xs is the indentation. Then I run it through a perl script to convert all the Xs to spaces (and remove the colon).

Here's an example of the sort of thing I mean. The issue isn't at the start of the line in this example, but it's the same problem. Here's the TeX code:

\documentclass{article}

\usepackage{pgffor}
\setlength{\parindent}{0pt}
\pagestyle{empty}
\begin{document}

xxx 0\par
\foreach \ind in {1,...,12} {%
xxx\foreach \sp in {1,...,\ind} {%
\
}%
xxx \ind\par
}
\end{document}

(I'm sure that you can imagine what that looks like)

here's the result of pdftotext

itex% pdftotext -enc ASCII7 -nopgbrk -layout indentation.pdf
itex% cat indentation.txt
xxx 0
xxx xxx 1
xxx xxx 2
xxx xxx 3
xxx xxx 4
xxx    xxx 5
xxx     xxx 6
xxx      xxx 7
xxx       xxx 8
xxx        xxx 9
xxx         xxx 10
xxx          xxx 11
xxx           xxx 12

(in case you can't see it, the bit inside the inner loop is \[space].)

@AndrewStacey I compiled ;-).

@AndrewStacey I almost want to star this message to see what it gives in the starred feed.

@AndrewStacey The behaviour of changing 1,2,3,4 spaces into 1 space might be considered a feature of pdftotext.

@BrunoLeFloch If it were as reliable as that, I might consider it so (though I'd like to know how to get 2 or 3 spaces as those are the ones I most often want!). However, my testing shows no discernible pattern on how much space it assigns to each indentation.

I'm still trying to reconstruct my thought (I didn't start thinking about writing a full TeX-engine-imitating package for no reason, even though it might not be the correct approach). One aspect that you don't consider (I may have missed it) is that a user of the tool you are writing probably doesn't want to redefine his macros which he's been using for years. And those macros happen to contain TeX primitives (users tend to do that a lot). (ctd)

That's why I'm keener on working at the level of primitives than LaTeX builtins.

What happens if a user does "\newcommand{\Abfii}{\ensuremath{\mathbf{A}^2}}" and then uses it, in your current system? Basically, how much expertise of the particular system is needed to use it, and what kind of TeX/LaTeX constructs are supported?

@AndrewStacey Hmpf. So I really need to finish up my textoutput to make your life easier. I'll think about performance issues later (but those will matter, because TeX is doing a lot of work behind the scenes).

@BrunoLeFloch First, I need to assume that the output that they are aiming for can handle mathematics. At the moment, that means itex, but others would be handled in a similar fashion. Then that command would work just find. I have a fake maths environment ($ is active) and \ensuremath is modified to check for that (I guess I could simply make \ifmmode redirect to the fake one). Then \mathbf is defined as a command which simply prints itself together with its argument. So \mathbf{A}

expands to \string\mathbf{A} where the {} have catcode 12. Similarly, ^ is active, takes one argument, and expands to ^{...} with catcode 12s.

@AndrewStacey In your case, you only have one place to output to, the pdf. In my case, I can both output to pdf and to a text file at the same time. Typesetting as I go allows to detect math mode in a more robust way: I won't be fooled by some crazy guy doing \catcode`%=\catcode`$. <- admittedly not a convincing example, but things can go wrong. I'm putting even more emphasis on "only TeX can understand TeX" than you: "only an unaltered TeX can really understand TeX".

Catcode changes are out of the question :).

\o/ I've finally reached \shipout.

I figure that anyone who knows about catcodes can look after themselves. I'm happy with a foolproof system, I'm not interested in making it idiotproof as well. (A fool doesn't know what they're doing, an idiot knows what they're doing and does it anyway.) In my scenario, the person writing the document can say "Ooops, I didn't get what I expected there; I'll change what I'm doing to get what I did expect.".

@AndrewStacey Makes sense :). I tend to try to make things bullet-proof.

I'll get some working system as soon as possible, with which you can hopefully very easily do what you need, and which hopefully will be extensible enough to do what other people might want to do.

@BrunoLeFloch Great1

(oops, forgot that I'd locked my screen and unlocked it again; that resets my keyboard. I typed G r e a t 1 which should have come out as G r e a t ! but it didn't.)

I'll have to do some work to get outer macros to work :-). But it should be feasible.

@AndrewStacey What do you do with tables?

I'm a little bit afraid of those. Last time I coded a fake \halign, it was very ugly code.

@BrunoLeFloch I don't, as yet, but when I do it'll be to convert a "table" to something in this format: michelf.com/projects/php-markdown/extra/#table

The reason I don't do tables as yet is that I don't really use them. One of my principles in writing software is that it should be useful to me right now. Otherwise, I won't keep up maintaining it (sad to say).

So I will eventually add tables, for completeness sake, but I'll work outwards from the stuff that I'm directly using and add more bits as people nag me for them.

(Got some errands to run, back in an hour or so.)

@AndrewStacey Makes sense. One of my "principles" in writing software is "write everything right right now" (I'll start my PhD really soon, and then my supervisor is likely to frown upon my rather high TeX involvement).

@AndrewStacey ok.

@AndrewStacey Sorry, my bad. ;-)

You guys could write a new blog entry with this great material!

@PauloCereda Andrew wrote something already (I think he points to it in one of his messages above). On what we just discussed, I think the best is to code some more before posting anything. So far, most of what I have is ideas (although I do know that they are feasible because I've coded some bits and pieces that I have to put together).

@BrunoLeFloch I see. Andrew's text is really a great addition to the blog. Anyway, I like your points of view, even if they are only ideas (we need to start somewhere), they are very consistent. =) I think this kind of discussion is great for a blog entry, specially when come from TeXperts like you and Andrew. A WIP (work in progress) is as great as the whole thing ready for production. =)

tikz/pgf anyone? =P

@PauloCereda My word! Someone has too much time on their hands.

@BrunoLeFloch @PauloCereda: Here's the code: math.ntnu.no/~stacey/HowDidIDoThat/LaTeX/internet.html

(For anyone else stopping by, that's a LaTeX class for writing stuff like blog posts using LaTeX)

@AndrewStacey And for anyone stopping by, my approach would be to write an engine rather than a class ;-).

@BrunoLeFloch Ah, but then I'm not you! Seriously, get that engine written and I'll happily use it. In the meantime, I'm going to use what I've written because it'll be directly useful to me. I've already written one nLab page, converted a LaTeX article to an nLab page, written one blog entry, one sort-of-blog entry, and am in the middle of writing another blog and another nLab page using it. He he!

@AndrewStacey Thanks for putting it online.

@AndrewStacey At the end of the day, I don't think that I will write an engine. Rather I'll be as close as I can to an "engine within TeX", because then I don't need to worry about macro expansion nor assignments (just detecting them, which is already done). The sad thing is that boxes remain, and they are a wee bit complicated in TeX (I'm skimming through tex.web, trying to get a feel for it).

I've got to go, sorry, and I'll be back only in a few hours.

@BrunoLeFloch I was thinking about this a bit. Whether the stuff gets in to the text file via a new engine, or via PDF and pdftotext, the underlying problem is still the same: how to convert what the author writes in to what the author means. However you get it in to the text file, you've still got to figure out what \begin{center} ... \end{center} should translate to.

Who else is using Adobe Reader under (Ubuntu) Linux? I'm using v9.4.2 and it keeps crashing when I want to see the font tab in 'Preferences' of LaTeX generated PDFs. Happens to me with my own documents but also with package manuals, e.g. xkeyval today.