quack

@PauloCereda In joy or despair?

@mickep FEAR

@DavidCarlisle Oh, it is getting worse!

@mickep well it takes a while to prepare lunch, so we'd better get started....

@mickep ooh

@DavidCarlisle You can start by playing Duck hunt on NES.

@DavidCarlisle oh no

@mickep ooh or playing SMB

@PauloCereda I play some Secret Maryo Chronicles with my daughter. It is also great fun! And free!

@mickep ooh that sounds cool!

@mickep have you heard of this game? goose.game I have it on my Switch.

@PauloCereda Playing on an android, and it does not work exactly as it seems to do on computer (looking at youtube). It is really difficult sometimes. And with some bugs.

@PauloCereda No, I have not. Is it fun? Looks violent!

@mickep oh my phone heats a lot, so I do not play games on it... :(

@mickep it's a lot of fun actually. And there's a dedicated honk button! :)

@PauloCereda Oh. Mine died two weeks ago. Still mailing with support to make them want to look at it for repair/new phone.

@PauloCereda Oh, LaTeX should have a honk possibility.

@mickep sorry to hear. Mine is about to die, so I bought me new one. It's still in the box, I will move things later on this week, hopefully. :)

@mickep ooh

@DavidCarlisle honk

@PauloCereda I think @DavidCarlisle will implement it in picture mode.

@mickep it's quite possible. :)

@PauloCereda important question for you on the main site tex.stackexchange.com/q/517069/1090

@UlrikeFischer ahahahahahahah :)))))

Why isn't there any \pdfescapestring equivalent in XeTeX? (or did I just miss it) And what can be done with \pdfescapestring? Some mysterious unknown user used it here for example: tex.stackexchange.com/a/273084/117050

@Skillmon Not at the engine level (I've not copied it), but you could emulate it: Bruni's done that in expl3 (non-expandable code)

@JosephWright yes, but if it's non-expandable it can't be used for tricks as in that fabulous answer by the mysterious unknown user...

@Skillmon It can be sorted expandably if really required, and if it's super-important, I can presumably add it to XeTeX ...

@Skillmon You really don't know who wrote the answer?

@JosephWright the mergesort provided in that answer is just pretty fast (the point of my insertionsort was that it is faster for small lists compared to l3sort, but that one is faster than insertionsort for every number of elements)

@JosephWright no, I know who it was.

@Skillmon Hmm, OK, so perhaps there is a case to answr ..

@JosephWright oh, and it is expandable... So in pretty much every aspect it is superior to my insertionsort. Don't know whether it performs good on large lists compared to l3sort, though.

@Skillmon I think emulating \pdfescapestring is likely not so bad at the macro level: it only special-cases a few chars

\cs_new:Npn \str_pdfescape:n #1
  {
    \exp_after:wN \__str_pdfescape:n \exp_after:wN
      { \tl_to_str:n {#1} }
  }
\cs_new:Npx \__str_pdfescape:n #1
  {
    \exp_not:N \__str_pdfescape_main_loop:w #1
      \c_space_tl \exp_not:N \q_recursion_tail \exp_not:N \q_recursion_stop
  }
\use:x
  {
    \cs_new:Npn \exp_not:N \__str_pdfescape_main_loop:w ##1 \c_space_tl  ##2
      {
        \exp_not:N \tl_map_function:nN {##1} \exp_not:N \__str_pdfescape_inner:N
        \exp_not:N \quark_if_recursion_tail_stop:N ##2

@Skillmon ^^^ A first pass attempt: not optimised nor fully checked just yet

@Skillmon I'n not included e-type expansion: it could soon be added

@Skillmon Do you want this adding to expl3? We have a team meeting coming up, I can raise it

@Skillmon Probably I need a bit more on the 'chars to escape': I think a mapping ...

\cs_new:Npx \__str_pdfescape_inner:N #1
  {
    \exp_not:N \if_int_compare:w 0
      \exp_not:N \if_charcode:w #1 \c_backslash_str 1 \exp_not:N \fi:
      \exp_not:N \if_charcode:w #1 ( 1 \exp_not:N \fi:
      \exp_not:N \if_charcode:w #1 ) 1 \exp_not:N \fi:
      > 0 \exp_stop_f:
      \c_backslash_str
    \exp_not:N \fi:
    #1
  }

@Skillmon ^^ Slightly better auxiliary

@Skillmon Looking at the C for \pdfstringescape, the other area to cover is control chars, which can be done quite fast but you have to decide how to handle the UTF-8 business for XeTeX

\cs_new:Npx \__str_pdfescape_inner:N #1
  {
    \exp_not:N \if_int_compare:w 0
      \exp_not:N \if_charcode:w #1 \c_backslash_str 1 \exp_not:N \fi:
      \exp_not:N \if_charcode:w #1 ( 1 \exp_not:N \fi:
      \exp_not:N \if_charcode:w #1 ) 1 \exp_not:N \fi:
      \exp_not:N \if_int_compare:w `#1 < `! \exp_stop_f: 1 \exp_not:N \fi:
      \exp_not:N \if_int_compare:w `#1 > `\cs_to_str:N \~ \exp_stop_f: 1 \exp_not:N \fi:
      > 0 \exp_stop_f:
      \c_backslash_str
    \exp_not:N \fi:
    #1
  }

@Skillmon ^^^ That's teh same approach as the C-code in pdfTeX, but really it's not right for UTF-8 .... I'd have to check the LuaTeX emulation I guess ... (@DavidCarlisle: oberdiek!)

@JosephWright I was just looking at pdfescape.dtx, but isn't the issue also that (at least for some pdf strings) you need utf16be anyway so simply \ escaping the utf-8 doesn't help?

@JosephWright isn't it OK for utf8 (as ascii chars always denote themselves) ?

@DavidCarlisle Er, I'm not sure: I was working from the C for \pdfescapestring, rather than what the PDF standard wants. The latter should be covered by str-convert

@DavidCarlisle I mean that the pdfTeX code escapes everything over ASCII 127, as they are control chars in an 8-bit source: presumably just skip that for XeTeX?

@DavidCarlisle I'm not sure if @Skillmon will get the performance required using macro-based emuation anyway: might make the different in which approach to use

@JosephWright yes but I think it's mostly targetting pdfdocencoding (@UlrikeFischer would know more) so yes the pdfescape emulation needs to just escape the () etc but (I think) it's not so useful (even in pdftex) if you are aiming for Unicode pdf strings

@JosephWright Heiko's Lua does the same, looking at pdf 1.7 spec I don't see why, it seems to say literal strings can have the full byte range and you only need to quote \ ( )

@JosephWright yes it goes on to say "This notation provides a way to specify characters outside the 7-bit ASCII character set by using ASCII characters only. However, any 8-bit value may appear in a string." So you do only need to quote \ and unbalanced ( or )

@DavidCarlisle imho this is a bit messy when you are outside ascii. I asked that earlier chat.stackexchange.com/transcript/41?m=35608723#35608723 and chat.stackexchange.com/transcript/41?m=35609294#35609294. One probably should at first get a better idea what should be the result here.

@JosephWright the macro emulation won't do anything I guess (not sure), it was just used to get the expansion of the following tokens done in the mergesort answer (I don't really care for the functionality of \pdfstringescape, just for the expanding abilities, though I'm not sure whether I quite got it yet).

@JosephWright I guess I can just replace it by \expanded in that code. Still have to give it a bit more thought, and probably I should try to contact him whether he's fine with my grave digging his code.

@Skillmon If you are only after expansion, then yes, just use \expanded (which is emulated in expl3 for older engines, if you need it)

@DavidCarlisle Seems we are in the clear :) @Skillmon is really just wanting the \expanded-like behaviour, not the actual escape

@JosephWright but @UlrikeFischer someone needs to fix this bug in pdftexcmds:-)

\documentclass{article}

\usepackage{pdftexcmds}

\begin{document}
\makeatletter

% utf-8  C3 A1 C3 A9 C3 A4 C3 BC D0 BC D0 B5 D1 82 D0 BE D0 B4
% utf-16 00E1 00E9 00E4 00FC 043C 0435 0442 043E 0434
% utf-32 00E1 00E9 00E4 00FC 043C 0435 0442 043E 0434

% pdetex \303\241\303\251\303\244\303\274\320\274\320\265\321\202\320\276\320\264
%         C3  A1  C3  A9  C3  A4  C3  BC  D0  BC  D0  B5  D1  82  D0  BE  D0  B4

% luatex \341 \351 \344 \37
%        00E1 00E9 00E4  1F?
% xetex boomf

\typeout{[[\pdf@escapestring{áéäüметод}]]}

@UlrikeFischer I may try to have a look before the release aimed for next weekend

@DavidCarlisle I'm not sure if it is a bug, at that time I had more the impression it is a missing feature. But yes, we need a set of reliable tools to produce "safe" pdf strings.

@DavidCarlisle \str_set_convert:Nnnn \l_tmpa_str {áéäüметод} { } { utf8/string} (LuaTeX/XeTeX) or \str_set_convert:Nnnn \l_tmpa_str {áéäüметод} { utf8 } { utf8/string} (pdfTeX) (we still need to address that variation ...)

@UlrikeFischer expl3

@UlrikeFischer looks like a bug to me:-)

@JosephWright tomorrow then:-)

@JosephWright I know that this is better! (I already switched to it in tagpdf). Btw did you see my nice qrcodes?

@JosephWright @UlrikeFischer sad thing is that (áéäüметод) is a safe PDF string already isn't it?

@DavidCarlisle if you view it as bytes (probably) yes, but is it meaningfull?

@DavidCarlisle We could fix pdftexcmds using a variation on what I've suggested above in XeTeX: wouldn't take long to convert to 'classical' code

@UlrikeFischer Sure, I mean that really we do have the tools here, things like pdftexcmds really need to be 'retired'

@JosephWright I didn't get the latex-dev update yet ;-(

@UlrikeFischer Tomorrow I think

@UlrikeFischer The hold-up with deployment meant I didn't get it to CTAN until late

@DavidCarlisle, @UlrikeFischer As a temporary fix for hyperref, we could use simple macro emulation: the expl3 version would need more changes to hyperref that are likely best left for l3...

@JosephWright yes

@JosephWright I wouldn't have any problems to load expl3 in hyperref to get access to the expl3 version. Imho most documents load expl3 anyway (and before hyperref) so probably not many would even realize that it is doing this (and it will have to happen at some time anyway). But hyperref should probably not depend on an only one-day-old expl3 version, so we would need a local version for some time.

@UlrikeFischer we could just use the version I've suggested above with \expanded added ...

@JosephWright you are speaking about \pdfescapestring? I was in the implicit token stuff. But for both imho using a bit expl3 here and there is fine (and \expanded too).

@UlrikeFischer Well, I thought we needed to talk about implicit tokens on Thursday: I guess you do want Bruno's code added?

@UlrikeFischer I was thinking of \pdfescapestring: it's easier to do it using the expl3 rather than re-code the loops and whatever

@JosephWright then we should do it with expl3. Where should it go? Is this l3pdf or l3str stuff?

@UlrikeFischer Hmm, that's a bit tricky: I was thinking it was in hyperref as for 'new' code we'd use \str_set_convert:Nnnn

@UlrikeFischer The implicit-token stuff can go into expl3 if we all agree

@UlrikeFischer Adding to agenda

@JosephWright I haven't tried yet, if I could use it in the hyperref code but it sounds sensible to have such a scanner. I think in future with alternate description and so we will have more often the need to handle such input.

@UlrikeFischer We could consider the \pdfstingescape function in l3legacy ...

@UlrikeFischer I was just looking at the hyperref code: we could just hijack \Hy@pstringdef to use the str convert function we already have ...

@JosephWright if we are going to do that. should probably start at the bottom and lose the driver option files.

@DavidCarlisle ? which option files do you want to loose?

@DavidCarlisle Yeah, I guess it's a question of how far we go

@UlrikeFischer all of them?

@DavidCarlisle you mean hpdftex.def etc?

@UlrikeFischer yes the expl3 code would already have committed to a back end, so hyperref had better make the same choice, so it may as well use the same definition files.

@DavidCarlisle I do have a first version of a generic, expl3 based hyperref driver in the pdfresources repo. But it isn't completly trivial to do this, the backends are in some places quite different (I still don't know how to handle xform with dvips), and testing is quite time consuming.

@UlrikeFischer yes but I'm not sure it make sense to use expl3 in bits without that (perhaps it does, we'll see...:-)

@DavidCarlisle I think expl3 would be useful for the pdfstring stuff in any case. But yes, getting rid of drivers should be done too.

@JosephWright speaking of pdftexcmds, are there expl3 replacements for \pdf@filedump and \pdf@unescapehex available yet? Both are used in some of my pkgs, still requiring the use of pdftexcmds.

@UlrikeFischer You could use the xform code in pdfbase.

@AlexG I will ask you when I tackle this again (not this week ;-))

@AlexG Those are not done: I guess in both cases, would be useful to have an idea of what they are needed for. I'll take a look at your code

@AlexG \pdf@unescapehex can be replaced using \str_set_convert:Nnnn, assuming you are setting some tl - \str_set_convert:Nnnn \l_some_tl { str } { utf8/string } { utf8 } or similar

@JosephWright \pdffiledump for reading-in binary files in chunks, while leaving the chunk as hex-encoded (that means twice the size) text in the input stream. In dvips mode, using this for creating/preparing PDF file stream objects in self-contained PS output.

Otherwise, when to-be-embedded files are read-in by pdf2ps, -dNOSAFER gs-option would be required.

@JosephWright Thanks! Going to test this (Need to digest this first. Looks a bit beyond me at the moment)

@AlexG I can come up with an interface for file dumping: should be pretty trivial really. I guess \file_hex_dump:nnn?

@JosephWright Exactly! Would be great!

@AlexG I can probably just add that directly

@JosephWright Nice, for all engines/backends?

@AlexG Where it's available: really just a matter of picking up the Lua code from Hekio's package, then adding some wrappers

@JosephWright Thanks!!!

@AlexG I'll look at this evening if I get time

@UlrikeFischer, @DavidCarlisle, @Skillmon Heiko wrote pdfescape which implements the equivalent of \pdfstringescape in TeX; I wonder why he didn't use that for hyperref/XeTeX ...

@AlexG I've got the Lua sorted I think, now working on the macro layer

@egreg Ah, the very person: @AlexG wants \pdffiledump in expl3: sorting it now

@JosephWright Did I propose to add it? ;-)

@egreg Yes ...

@egreg, @AlexG github.com/latex3/latex3/commit/…

@JosephWright I stumbled upon it today, too, and wondered pretty much about the same :)

@Skillmon Ah, of course: they are not done by expansion, as he needs \edef (Well, actully he doesn't, but Bruno's implementation of \expanded in macros is of course pretty hairy)

Does latexmk generate log files, or just write to standard output? I guess redirecting to a file is an option, but it seems surprising it doesn't generate its own log files.

@FaheemMitha never used it, but most the programs it runs, latex, bibtex etc already log to a file

@DavidCarlisle I'm aware, but it does add some chatter of its own.

@DavidCarlisle yes, but latexmk makes a lot of decision processes, so if it does not allow logging, debugging can be tough.

@yo' run it in an emacs shell buffer :-)

github.com/sponsors/FrankMittelbach

@DavidCarlisle -- But if a step is run multiple times, only the last log file for each different process would be kept unless extraordinary instructions were given. (I realize I'm a reactionary, but I still prefer to run everything explicitly from the command line.)

I'm trying to redirect to a file, but it's not working. It's being called from Lua, so I may be doing something wrong there.

Trying to track down a bug. It's proving rather more difficult to isolate than usual.

On the command line I can redirect latexmk to a file, no problem.

@barbarabeeton arara keeps log of everything. :)

Which gives me an idea. Hmmm.

@PauloCereda Does Arara compile TeX files?

@FaheemMitha Sure, why shouldn't it? % arara: pdftex is all you need.

@JosephWright already got infinitely more sponsors than me:-)

@AlanMunn Do you use it? How does it compare to latexmk?

@FaheemMitha I don't use it much, really. I use TeXShop as my editor, and it has custom scripts called Engines which work on a per document basis too. So for most things I just use the appropriate latexmk script. But I do use arara for some more fancy things. For example I use it to generate multiple versions of a syllabus for a multi-section course I supervise. I also use it when I need to use shell escape (just because it's easy to add options.)

@AlanMunn So is it more customizable/programmable than latexmk?

@DavidCarlisle And me

@FaheemMitha In my opinion they do different things. latexmk is great at general purpose automation: it figures out how many runs to do and which other programs to run. I use it especially for any document that is going out in public since I know that I generally won't have missing references or ? floating around.

@FaheemMitha arara is really better as a build tool, especially for complex builds. It's not designed to figure out by itself what to do. But it can do many more things than latexmk.

@AlanMunn What are examples of use cases for arara?

@FaheemMitha Well I'm not the best person to ask about that. I've given you the two examples that I use it for.

@AlanMunn Actually, multiple versions of a syllabus could just be done with a loop. And substituting variables appropriately. But of course I don't know exactly what you are doing.

@FaheemMitha Of course. But doing it with arara allows me to have the build associated to the file and run from within my editor. Tools are all about convenience. There are many ways to implement automation.

@FaheemMitha arara is very useful if you want some own implementation of "check whether this is in the log and only run it if it is". It can also compile dependencies if told to (e.g. externalized figures). And the whole compile process of a document is transparent to whomever the file gets to because it is entirely encapsulated within a few comments in the TeX files. So it is an awesome tool to get an reproducible build process (not builds because they depend on the TeX distro) across machines.

I just did tlmgr update --list and a rather large list of updates came up. Curiously, most are TeX engines (including tex) and related programs (bibtex, for example). Does anyone know what is this about?

@PhelypeOleinik this, I guess 9d44e46f7 do not log \openout files for original (ini)tex

@AlanMunn @TeXnician My current use case is just to compile TeX files. Nothing wildly exotic, but not completely vanilla either. latexmk can cope. Does arara offer anything here that latexmk doesn't?

@FaheemMitha Probably not, IMO. (Or If it ain't broke, don't fix it.)

@DavidCarlisle Ah, found it. Thanks

@TeXnician As I mentioned earlier, latexmk doesn't seem to log its actions by default. Though maybe it does, and I'm just missing it. But I suppose that's a relatively minor concern.

@barbarabeeton was that your doing? (tug github:-)

@DavidCarlisle where?