TeX, LaTeX and Friends

12:01 AM

@DavidCarlisle Just found two things that can only be done at font loading time, that shouldn’t be done their. Glyph metrics, but this is a more optimization so can live with slightly slower font load, the other is ToUnicode mappings that need to take into account ligatures and these are only known at text layout time, so one is forced to use the less supported /ActualText spans.

@DavidCarlisle Both of these do not to be done at font loading time at all, but they are done this way I guess because that is how TeX has always done things.

@DavidCarlisle LuaTeX seems to be centered around the idea that fonts should be loaded once in memory and be done with it, this might have been true for TeX82 when fonts had 127 glyphs and simple TFM files.

David Carlisle

@KhaledHosny Is it worth trying to push any changes back to Hans?

Khaled Hosny

@DavidCarlisle May be, but I’m not looking forward to the discussion that would result from such suggestions ;)

I guess I better finish what I’m doing, then collect LuaTeX limitations and discuss them at once.

David Carlisle

@KhaledHosny sounds like a plan:-)

Joseph Wright

12:24 AM

@DavidCarlisle Not in e-(u)pTeX so not requested for XeTeX ...

@KhaledHosny Not surprising: XeTeX explicitly bakes in new font handling, so the entire set up is different, whilst LuaTeX aim to avoid adding any major features beyond Lua itself

@DavidCarlisle I worked from tug.org/svn/texlive/trunk/Build/source/texk/web2c/eptexdir/…

@UlrikeFischer Ah, sorry: the file dump one is (it's in e-pTeX), the hex one isn't

Werner

1:49 AM

3

Khaled Hosny

6:49 AM

@JosephWright LuaTeX changed how hyphenation works so radically it isn’t compatible with any other TeX engine anymore, it could have done that with fonts (to some degree at least).

Marcel Krüger

7:18 AM

@KhaledHosny If you do not want to load the characters at the beginning, you can use font.addcharacters. Then you only need to set the glyph parameters on first use of each glyph. Except for the glyph metrics and the index this should also allow overwriting the values of existing parameters, but I only used it to add new ones yet.

Khaled Hosny

@MarcelKrüger Thanks for the suggestion, will give it another (I think I tried it before, not sure why I didn’t use it at the end).

Joseph Wright

8:09 AM

@KhaledHosny Sure, but I think it has to do with core goals: Hans needed to change hyphenation to work on nodes, etc., but he is happy with monolithic fonts

Khaled Hosny

@JosephWright Exactly!

Joseph Wright

@DavidCarlisle, @UlrikeFischer Do we need \pdfunescapehex? I can sort it for XeTeX/(u)pTeX ...

Ulrike Fischer

8:43 AM

@JosephWright I don't know. I stumbled on it while converting the hyperref tests. But the command is only used inside a test command:

\def\GetFileContents##1{%
      \pdf@unescapehex{%
        \pdf@filedump{0}{\pdf@filesize{##1}}{##1}%
      }%
    }%
    \Expect*{\GetFileContents{\jobname.aux}}%
           *{\GetFileContents{\jobname-aux.aux}}%

Ulrike Fischer

9:08 AM

@KhaledHosny I don't think that the reason is "how TeX has always things done". But TeX and luatex has been created by people living in a culture who has book printing with lead letters since hundreds of years and this shapes fonts and letters a lot and the view on them too. Our letters are quite immutable. They don't change at text layout time. Adjusting to the idea that this isn't true for all scripts in the world is not so easy.

Joseph Wright

9:32 AM

@UlrikeFischer Isn't that a way to do something like \expanded but without the primitive?

Khaled Hosny

@UlrikeFischer My issues are with Latin too (that is actually what I’m testing right now). Take ligatures, LuaTeX wants to know what ligatures the font has at font loading time, but that is a waste of resources, since you only care about them when you encounter them at layout time. The only reason it would want to do what is does now is because that is how TeX always worked, nothing inherit about a particular writing system.

@UlrikeFischer It is impractical to parse all the font substitution rules while loading to fill out tounicode values, when you will need that info just at the time of writing the final PDF output, and you should only care about ligatures actually used in the document and you get that for free as a side effect of the layout process.

Ulrike Fischer

@KhaledHosny No it is the same with ligatures. If you are printing a book with lead letters you don't start with a small subset of letters and run back to the store getting more everytime you encounter a new one. And we have only a small number of ligatures just for this reason: not to waste resources due to the need to have lots of variants availables.

Joseph Wright

@UlrikeFischer I looked over TeX Live. Other than Heiko's code, there's only one place that \pdfunescapehex is used: media9. Heiko has, in pdfescape, emulation at the macro level anyway. I'm minded to say we can go that way if/when we need this: do the job ourselves, not have to ask for another engine addition. Or at least I'm not in a hurry ...

@UlrikeFischer @UlrikeFischer I guess if we really need the concept, a primitive is handy, but I'll need to check with Alexander Grahn about his use case. All of Heiko's are either for using it as and \expanded substitute or are for emulation, etc. of the primitive itself!

samcarter

@marmot This conference could benefit from your experience: eurekalert.org/pub_releases/2018-10/aps-hhc102418.php

Ulrike Fischer

@JosephWright no idea. I didn't really understood \expanded yet ... ;-)

Joseph Wright

9:45 AM

@UlrikeFischer You soon will do

Ulrike Fischer

@JosephWright Perhaps. If some real use case comes along ...

Joseph Wright

@UlrikeFischer It comes down to being able to do something like

\long\def\foo#1{\expanded{\noexpand\fooaux{#1}}}
\long\def\fooaux#1{...

@UlrikeFischer which makes \foo work like a function in other languages

Khaled Hosny

@UlrikeFischer My font loader loads the same Latin font at ~10% of the time luaotfload takes, that is the waste of resources I’m talking about. It really about how TeX works. Even TrueType that wasn’t designed with any complex text functionality in mind, is designed to allow constant time access to any glyph, without g=having to fully decode the font in memory.

Joseph Wright

@KhaledHosny Sounds good!

David Carlisle

@UlrikeFischer for oberdiek->l3build tests I was planning at some point to take them all out of the dtx files so they are fixed source files not generated, does that fit with your plans?

@JosephWright yes but if it means extensive forks in luatex the program rather than mostly in the build setup to link hb then long term it's a bigger maintenance commitment

Transcript for

TeX, LaTeX and Friends

$TeX - LaTeX$ TeX, LaTeX and Friends