Why is package and class options so tiring? I find it very hard to get some specification on what a package or class should do with its options (especially classes, what if I'm not the used document class but only got \LoadClass, which options should I process? And how can I tell that I'm not the main class?)

@Skillmonlikestopanswers.xyz Why not leave that up to \ProcessOptions? It has never done the wrong thing, in my experience. And why would your class need to know if it is the main class or not?

@HaraldHanche-Olsen take a look at packages like pgfopts, l3keys2e, etc. There is a reason they exist. Main class or not is important for setting up the \@unusedoptionlist correctly (if the main class actually used every option this list will be empty, if I'm now loaded with \LoadClass and there are options in the \@classoptionslist that my class doesn't know it would mistakenly populate the \@unusedoptionlist).

(unless I got a misconception for the \@unusedoptionlist and the usage of \@classoptionslist, \csname opt@\@currname.\@currext\endcsname, etc., which might well be the case, since everystime I start looking into this I lose interest)

quack

@Skillmonlikestopanswers.xyz Oh. Well, I have never used any of the keyword packages, other than indirectly (i.e., using packages that use a keyword package), so I didn't know that this makes troubles for handling the list of unused options.

@PauloCereda Lunch already?

@PauloCereda is currently not worked on. First I need to think of a proper interface to define rule sets.

@HaraldHanche-Olsen oh no

@Skillmonlikestopanswers.xyz oh

@PauloCereda I am not mean. ;-)

@HaraldHanche-Olsen you are m... oh wait

@HaraldHanche-Olsen it doesn't really make trouble. In fact they do more or less the same as the \ProcessOptions code. The problem is that I'm not sure what my code should do in the four possible scenarios (package before \documentclass, \documentclass, \LoadClass, package after \documentclass).

@PauloCereda and I need to decide which parsing code I should use. Currently my small test I played with one or two years ago is using pgfparser, but pgfparser has its limits (it can't distinguish \foo from \bar after you did \let\foo\bar, a limitation I'm not sure I should stick with, overcome, or simply remove from pgfparser, which would change its documented behaviour)

@Skillmonlikestopanswers.xyz You could need:

@Skillmonlikestopanswers.xyz /hug

@Skillmonlikestopanswers.xyz it would be not bad to have an overview over the current situation.

Dec,     Hex,      Strokes,   Symb,   Descr
12353,   U+3041,   3,         ぁ,      HIRAGANA LETTER SMALL A
12354,   U+3042,   3,         あ,      HIRAGANA LETTER A
12355,   U+3043,   2,         ぃ,      HIRAGANA LETTER SMALL I
12356,   U+3044,   2,         い,      HIRAGANA LETTER I   I
12357,   U+3045,   2,         ぅ,      HIRAGANA LETTER SMALL U
12358,   U+3046,   2,         う,      HIRAGANA LETTER U
12359,   U+3047,   2,         ぇ,      HIRAGANA LETTER SMALL E
12360,   U+3048,   2,         え,      HIRAGANA LETTER E

@cis I'm not sure whether I should parse a full csv-file every time it is used, better would be some \quack@output@specification{12353}{3041}{ぁ} (stroke count and letter description are not really important for an output mapping, the former would be if we provided a show strokes option). Also this provides an output mapping, but still no input mapping :)

@Skillmonlikestopanswers.xyz I mainly need this for the stroke information.

^^ ooh tikzlings chess

@UlrikeFischer, @yo' ^^

Ducks are very good at chess

And the output mapping would be engine dependent (Lua- and XeTeX can directly output the Unicode symbols and let the font do the rest, pdfTeX needs to call the correct font table entries from the correct font).

@Skillmonlikestopanswers.xyz well the idea was that you shouldn't need to touch that list but then handling the options yourself wasn't part of the idea anyway. I'm not sure it makes a lot of sense to use unusedoption mechanism at all in a keyval setting

@cis I have no idea.

@Skillmonlikestopanswers.xyz Mmmh, maybe better to work with data-tables at this point.

@cis I don't think you can (it's more of a source comment) but luatex has the whole font to hand so perhaps you can, @MarcelKrüger would know:-)

@DavidCarlisle Ok.

@DavidCarlisle well, we have to play nicely on the base structure we have, which is removing the entries from the unused options (which is what pgfopts and l3keys2e and probably the others currently do wrongly, they assume the value doesn't have braces around it in any case for the global option list, I know that this is a limitation the kernel has as well, but I don't see any point in not at least allowing it with more modern interfaces..)

@Skillmonlikestopanswers.xyz but it is clear what not using a4paper as an option means, but once you allow values, is a4paper=false used or not used? I am not sure there is any sensible answer.

@PauloCereda nice, but it looks as whoever put the pieces on the board confused queen and king.

@UlrikeFischer oopsie

@DavidCarlisle KOMA classes support key=val options for the class since ancient times (with a bug for braced values as well)... If any package or the class has a key=val option a4paper accepting a value (string, tl, or bool) I'd say that an option a4paper=false was used, since it got parsed by the respective package and hence should be removed. If no package or class has the key=val option a4paper and no package or class has the non-key=val option a4paper=false it wasn't used.

@DavidCarlisle I don't really see the problem here.

@PauloCereda and the board is wrongly rotated the right field (for a player) should be white.

@UlrikeFischer oopsie 2 :(

@Skillmonlikestopanswers.xyz yes so essentially you just check the key names not the values in terms of "used" I implemented it that way somewhere myself, but it doesn't seem very useful and implementing it as \let\@unusedoptionlist\@empty always seemed appealing

@Skillmonlikestopanswers.xyz I think I can lay the foundation for commands like:

But don't worry, I'll show you examples.

@DavidCarlisle no, I'd filter it out of the \@unusedoptionlist if a package grabbed the key and the value. If a package has an option a4paper which doesn't support a value, it doesn't get filtered out. More problematic is the case that a package provides a choice key which doesn't accept the specified value. It would throw an error for the unknown choice false and the pair would still be filtered out, even if it wasn't ever be meant for that package, but for another package...

@cis not sure whether all those options should be available in the same command. I wouldn't show the stroke order for a text, but only for single characters.

@cis See chat.stackexchange.com/transcript/message/54476965#54476965

@MarcelKrüger Sorry, I overseen that.

@Skillmonlikestopanswers.xyz I think you need some additional structure to store unused keys and perhaps even key-value combination. The simple list doesn't really fit to a key-value system.

@Skillmonlikestopanswers.xyz Yes, of course. Wait for my fantastic examples. Only then will it become clear what I mean.

With Beamer, I currently just generate the notes I have.

Can I set Beamer up such that the notes also have the same corresponding frame titles?

@PauloCereda Which moves can the bird make? Like a knight and jump over things?

@ComFreek Sure that's possible, however such questions would be easier if there would be some code to start with

\documentclass{beamer}
\usepackage{pgfpages}
\setbeameroption{show notes}
\setbeameroption{show notes on second screen=right}

\begin{document}
	\begin{frame}{Title}
		abc
		\note[item]{def}
	\end{frame}
\end{document}

*corrected

oh apparently the default shows titles

so must be quirks of my template

@ComFreek That's why I was asking for code :)

ah I use

\setbeamertemplate{note page}{\pagecolor{yellow!5}\insertnote}\usepackage{palatino}

copied from the WWW!

Unrelated: showing both contents and notes breaks many overlays for me

to the point where the slides are just broken content-wise, are missing things, ...

@samcarter_is_at_topanswers.xyz spoilers, he's the king. :)

\documentclass{beamer}
\setbeameroption{show notes}
\setbeameroption{show notes on second screen=right}

\setbeamertemplate{note page}{%
  \pagecolor{yellow!5}
  \vfill
  \begin{minipage}[c][\textheight][t]{\textwidth}
    {\usebeamerfont{frametitle}\usebeamercolor[fg]{frametitle}\insertframetitle\par}
    \insertnote
  \end{minipage}
}

\begin{document}
	\begin{frame}{Title}
		abc
		\note[item]{def}
	\end{frame}
\end{document}

@ComFreek ^^^

@ComFreek hard to diagnose without code :)

@PauloCereda Ohhh

I have a complicated project setup

but it broke for me in multiple unrelated presentations already

I use columns, align*, tikz extensively on slides

@samcarter_is_at_topanswers.xyz ooh

@UlrikeFischer well, the current option handling of LaTeX2e doesn't really fit key-value options at all and the \zap@space approach is questionable as well.

It only breaks with \setbeameroption{show notes on second screen=right}

if I leave that out, i.e. slides and notes intermingled one-slide-after-another, it's fine

@ComFreek Do you use absolute positioning of tikz?

i think so

but also align* breaks

and tikzcd

@ComFreek There are probably workarounds, but without mwe to test ....

would cost me too much time

with \setbeameroption{show notes on second screen=bottom} it also breaks

I think it breaks whenever I invoke \setbeameroption{show notes on second screen=???} at all

The TeXLive font looks very empty.

@Skillmonlikestopanswers.xyz we know. But changing this is not easy. David can tell you. E.g if we would remove the space zapping someone would complain that an option like 12 pt no longer works, or that it breaks the komascript option handling.

@ComFreek Then I don't understand what's the point of complaining if we don't get the information necessary to fix this.

Well, I hoped for this bug to be known or common causes to be known.

Also it's quite weird that slides and notes work each on their own, but not both at the same time.

So it smells like a bug.

@ComFreek There are so many possible causes, it would cost me too much time to list them all

@UlrikeFischer I know... But just because I know doesn't mean I can't blame you :)

@samcarter_is_at_topanswers.xyz it has a special power to make any square white

@Skillmonlikestopanswers.xyz you should definitely blame @UlrikeFischer for \zap@space, since I wrote it and you can't blame me.

@DavidCarlisle I could blame you and you could forward the blame to @UlrikeFischer (the "you" was meant as the plural for the whole team)

@Skillmonlikestopanswers.xyz more efficient to skip the forwarding and just blame Ulrike

@Skillmonlikestopanswers.xyz seriously though we do have some sort of a plan to fix this, probably involving not changing the current mechanism but exposing an l3keys based option system, somehow

@DavidCarlisle I'm still in favour of providing a raw option list in addition to the zapped one.

@DavidCarlisle :D

@JosephWright latexcgi.xyz (since I have timeout working:-)

@DavidCarlisle you are going public?

@UlrikeFischer well don't plan to advertise it other than chatting here, but I flipped the public switch on the github repository.

@UlrikeFischer If I get bibtex/biber running this evening I'll switch over my ll site to give it a better test.

@DavidCarlisle I just tried it and wanted to comment that this doesn't work yet ;-)

@UlrikeFischer it's not broken, it doesn't even try:-)

@DavidCarlisle ;-). Do all the engines work already?

@UlrikeFischer they should do, although I'm not convinced I have actually tested them all

@DavidCarlisle xelatex works. And the test document gives a nice log when there is an error.

@UlrikeFischer the thing that had me stumped for a couple of days was \def\z{\z}\z which hopefully does something sensible. timeout seemed harder than I expected.

@DavidCarlisle looks ok now. I just tried filecontents and it works too. What happens if people try to write to many files. Does it timeout too?

@UlrikeFischer the timeout is on the call to the engine so yes if the time is writing lots of filecontents or lots of tikz or an infinite loop it's all the same. I don't limit filespace (although I could) also currently i don't clean up at all (useful for debugging)

@DavidCarlisle you can find my log-files by looking for \blub ;-)

@UlrikeFischer bibtex latexcgi.xyz/testc2.html

@DavidCarlisle didn't bibtex worked already? Or is learnlatex cheating?

@UlrikeFischer bibtex worked at learnlatex but latexcgi just ran latex once and never ran anything else until ten minutes ago

@UlrikeFischer I just swiched ll over to use latexcgi and now it works er a bit less well than before but minor details davidcarlisle.github.io/ll/en/lesson-12

@DavidCarlisle curious I tried in the afternoon one of the bibtex examples in your learnlatex version and thought I got references. But perhaps I clicked on the wrong button.

@DavidCarlisle is there a precompiled bbl somewhere?

Or perhaps you should add a bit of cleaning up.

@UlrikeFischer something's not quite right each run is supposed to be in a clean directory 9but all main files are called docuemnt) but the bibtex examples aren't working quite right and in the japanese lesson the luatex one works but the ptex ones don't bit odd the perl decides they work so starts the pdf viewer which then says there is no pdf, but getting there

@DavidCarlisle well the bbl certainly interfere. I get in the natbib example errors from biblatex and the other way round (it is very nice that one can see the log ;-)).

yes and platex is giving this which is possibly the same thing

\heisei=\count174
)
(/usr/local/texlive/2020/texmf-dist/tex/latex/l3backend/l3backend-dvips.def
File: l3backend-dvips.def 2020-06-03 L3 backend support: dvips
\l__pdf_internal_box=\box62
\g__pdf_backend_object_int=\count175
\l__pdf_backend_content_box=\box63
\l__pdf_backend_model_box=\box64
\g__pdf_backend_annotation_int=\count176
\g__pdf_backend_link_int=\count177
\g__pdf_backend_link_sf_int=\count178
)
(./document.aux
! Undefined control sequence.
l.2 \abx@aux@refcontext
                       {nyt/global//global/global}

Ahhhhhh

@DavidCarlisle I guess that means you know what's going on ...

@UlrikeFischer well partly I made a change and now they don't intefere but every example times out

@Rmano chat.stackexchange.com/transcript/41?m=51522720#51522720

@UlrikeFischer ...and you are prefectly right (that's remind me of Dr. House).

@UlrikeFischer did you know, platex doesn't make pdf files:-)

@DavidCarlisle ;-).

@DavidCarlisle the biblatex example works now with backend=bibtex,.

@UlrikeFischer shoudl work with biber as well?

@DavidCarlisle A minute ago it didn't work, and now I get an internal server error (after I tried to edit the bib to look what happens).

@UlrikeFischer sorry missing } while I was adding dvipdfmx, but platex now works

@DavidCarlisle yes works again, but the biblatex example in lesson 12 errors with Timeout/Error status (/tmp/latexcgi-77737)(bibtex): 2.

@DavidCarlisle I got an error from This is e-pTeX, ;-)

@UlrikeFischer yes I got same with different number, its auto detection of biber in the log file must have failed

@UlrikeFischer which one? the platex one worked for me?

@DavidCarlisle I forced an error to check in the log the engine. It works fine.

@UlrikeFischer biber now works

@DavidCarlisle yay, and one can even edit the bib-file ;-)

@UlrikeFischer so I think all the learnlatex examples now work even on mobile (@JosephWright)

@DavidCarlisle even the size of the window is fine on my mobile.

@Skillmonlikestopanswers.xyz can you try davidcarlisle.github.io/ll with your setup, do you still get multiple output dialogues?

@DavidCarlisle hey, yes! working on my android tablet finally (it used to go into infinity rebuilds...)

@Rmano yeh! thanks for letting me know..

👏👏👏👏

@Rmano it's a completely new back end latex service, but the reason it works on android is that it's using PDF.js rather than serving the PDF directly.

@DavidCarlisle Ah, yes. Where is LaTeX running? It's quite fast.

@Rmano amazon aws ec2 instance (their smallest t2 layer available on the free tier, which means I have a year to find someone to pay for it:-)

@Rmano but it just requires a linux box running apache, perl and an install of texlive 2020 so anyone with an internet available linux box could run it (I run a test server on wsl on localhost on this windows laptop as well) amazon aws was just tempting as it's free for a year and I thought I'd try it.

@DavidCarlisle Hmmm... I do not think I can convince webhostingpad to install TeXLive, not with my contract... it would be nice to have a kind of LaTeX friendly hosting provider and share the load...

But anyway, nice job, congrats!

@Rmano thanks: latexcgi.xyz

@DavidCarlisle Yay!

@DavidCarlisle So we switch?

@JosephWright well do we need to have a long term plan? Also I'd like a few people to compare speed etc before I actually suggest switching. (I have some plans to add makeindex and other things as well) should probably open a gh issue?

@DavidCarlisle I looked a bit at the pdf output: pdf.js shows linkborder but no form fields. The latexonline system doesn't show linkborders, but the formfields works. (not that it matters for the target audience).

@UlrikeFischer I plan to allow a % !TeX PDF.browser option (defaulting to % !TeX PDF.js that woudl allow you to use the built in browser renderer, but given the mobile behaviour I think defaulting to PDF.js is safest,

@DavidCarlisle in my opinion it is fine. It works much better on my mobile. It is much more important to have a sensible handling on mobile for this than being able to create form fields ;-)

@DavidCarlisle and I managed a timeout by using the well known FiraSans package in lualatex ;-)

@UlrikeFischer any idea how long it would need? I already bumped it up to 10s (although if it triggers a font cache thing it takes a long time?

@DavidCarlisle sorry my wlan broke. fira needs quite some time. But even with a single font I have a failure. Curiously the log says that the pdf was created, but it is not served back:

\documentclass{article}
\begin{document}
abc abc \font\test={file:FiraSans-Regular.otf:mode=node;}\test abc
\end{document}

Output written on document.pdf (1 page, 4515 bytes).

PDF statistics: 22 PDF objects out of 1000 (max. 8388607)
 12 compressed objects within 1 object stream
 0 named destinations out of 1000 (max. 131072)
 1 words of extra memory for PDF output out of 10000 (max. 100000000)


Timeout/Error status (/tmp/latexcgi-78891)(lualatex): 124

@UlrikeFischer I ran it twice before I noticed you had neglected to suggest running it with luatex:-)

@DavidCarlisle sorry ;-) But I have the % !TEX lualatex in my example.

@UlrikeFischer ^^^^^^^^^^^^

@DavidCarlisle now it worked for me too. Would it make sense to store the database? I think currently it is regenerating at every run.

@UlrikeFischer I noticed some caches appearing in odd places, I just did a normal texlive install, why wouldn't it store the font cache system wide? (never really thought about it much before:-)

@UlrikeFischer vv

$ git commit -m "waiting for Ulrike"
[master 5a26ec5] waiting for Ulrike
 1 file changed, 2 insertions(+), 2 deletions(-)

@DavidCarlisle ah, more time for me. But \setmainfont{firasans} fails again.

@DavidCarlisle well I would expect it in a user root. But I need to check. Can you run luaotfload-tool on the server to setup a system cache?

@UlrikeFischer probably I installed texlive as root but the server is running with a www-user user without permission. I should probably change tha, I can certainly run luaotfload-tool anyway by hand, just type that?

@UlrikeFischer so many choices?

luaotfload-tool
Usage: luaotfload-tool [--help] [--version] [--verbose=<lvl>]
                       [--update] [--force] [--prefer-texmf] [--local]
                       [--dry-run] [--formats=<extension list>]
                       [--find=<font name>] [--fuzzy] [--info] [--inspect]
                       [--list=<criterion>] [--fields=<field list>]
                       [--cache=<directive>] [--flush-lookups]
                       [--show-blacklist] [--diagnose=<procedure>]
                       [--no-compress] [--no-strip] [--local]

@DavidCarlisle yes luaotfload-tool -u should create the database. Probably it would also make sense to precompile the various lua for the fonts, for some fonts and font packages this can take quite some time. Probably @MarcelKrüger has an idea how to script that.

@UlrikeFischer it ran, is your example any quicker?

@DavidCarlisle it no longer says that the database is not found, but \setmainfont{firasans} still times out.

@UlrikeFischer try now

@DavidCarlisle now it worked. What did you do? More time or did you precompile the cache?

@UlrikeFischer changed it from 15 to 150 second timeout, just to check it basically worked. not sure what a reasonable timeout is really. It may be best to have a small timeout like 3s so most things caught quickly but allow a TeX comment to set it bigger up to some maximum. Not sure...

@DavidCarlisle well I think that one really should have the lua/luc files precompiled in a system wide cache. You don't want every user to wait until e.g. all the fira fonts have been created. cjk fonts are slow too.

@UlrikeFischer so can I do that with a single command?

@UlrikeFischer davidcarlisle.github.io/latexcgi

@UlrikeFischer presumably if I fix up the write permissions so the server user can update the cache it wouldn't be too bad if the cache was updated on demand, as long as the timeout was long enough for the first user of each font. But pre-caching everything is probably better, than I can reduce the timeout a bit

@DavidCarlisle I don't think that there is premade command, one probably need some script that goes through all fonts. @MarcelKrüger any idea?

@DavidCarlisle The problem is that a writable cache might allow a DoS attack by writing malformed cache entries of basic fonts. If e.g. lmroman10-regular is cached in a form which misses all glyphs, then LuaLaTeX becomes effectively useless by default.

@UlrikeFischer The creation of the font caches is pretty integrated into the fontloading system, so the best approach might be to do an actual TeX run over a document which iterates over the cache and "loads" all fonts.

@MarcelKrüger yes I thought so. I think I even saw at some time a script which used the database to create such a document.

It could even be that Frank has something.

@MarcelKrüger hmm currently I don't filter file uploads so you could in theory upload an opentype file into the current directory. I am not sure I could stop that, I could filter known extensions but does luatex actually require fonts to have sensible extensions?

@UlrikeFischer @DavidCarlisle ^ That should cache everything.

@DavidCarlisle LuaTeX doesn't care, but the file couldn't shadow a default font then so it shouldn't be much of a problem.

@MarcelKrüger thanks running that by hand now on the server ooh something killed it

echo $? gives 137

Adding print(i, index[i].fullname) at the beginning of the loop might show at which font the problem occured.

oh that's out of memory:-)

its taking a long time on 154 ibmplexsanssemibold

152     ibmplexmonobold
153     ibmplexsanscondensedbolditalic
154     ibmplexsanssemibold
155     ibmplexseriflightitalic
Killed

@MarcelKrüger just make it skip if the fullname starts ibmplex? and see how far it gets?

@DavidCarlisle Might be worth a try, but you will probably get the same issue at least for the CJK fonts

\directlua{
  local index = luaotfload.aux.read_font_index().mappings
  for i=1,\csstring\#index do
    print(i, index[i].fullname)
    if index[i].fullname:match('^ibmplex') then
      print'SKIPPED'
    else
      local path, subfont = index[i].fullpath, index[i].index
      assert(fonts.definers.read(string.format("[\csstring\%s](\csstring\%i):mode=node", path, subfont), -1000))
    end
  end
}
\stop

Could you temporarily add lots of RAM or swap for cache creation?

@MarcelKrüger no, or at least not RAM I may be able to fiddle with swap but my linux sysadmin skills are a bit rusty (this is a virtual instance on amazon aws so It could have as much ram and swap space as it needed if I paid, but this is the free tier...)

@DavidCarlisle You can use e.g. dd if=/dev/zero of=swapfile bs=1048576 count=4096 && mkswap swapfile && swapon swapfile for 4GB of swap, but i'm not even sure if that's enough for all fonts. Probably the only real way is to manually create a list of excluded fonts.

@MarcelKrüger keeping the server busy my shell is unresponsive:-) its thinking about heuristica fonts

@DavidCarlisle Please start again with the following version:

\directlua{
  local index = luaotfload.aux.read_font_index().mappings
  for i=1,\csstring\#index do
    print(i, index[i].fullname)
    if index[i].fullname:match('^ibmplex') then
      print'SKIPPED'
    else
      local path, subfont = index[i].fullpath, index[i].subfont or 1
      assert(fonts.definers.read(string.format("[\csstring\%s](\csstring\%i):mode=node", path, subfont), -1000))
    end
  end
}
\stop

I had the wrong name for the subfont index field...

@MarcelKrüger if I can get back in...

@MarcelKrüger phew had to reboot from the ec2 management console and wait a bit but it's all back:-) that's it for tonight, good night, thanks for the help.

@DavidCarlisle Good night.