@UlrikeFischer morning, new failure today:-)

$ find . -name \*diff
./build/test/hycolor-test1.luatex.diff
./build/test/hycolor-test1.xetex.diff
./build/test/hycolor-test2.luatex.diff
./build/test/hycolor-test2.xetex.diff
./build/test/hycolor-test3.luatex.diff
./build/test/hycolor-test3.xetex.diff
./build/test/pdflscape-test6.xetex.pdf.diff
./build/test-config-pdftex/askinclude-test24.pdftex.diff
./build/test-config-pdftex/selinput-test1.pdftex.diff
./build/test-config-pdftex/selinput-test2.pdftex.diff
./build/test-config-pdftex/selinput-test5.pdftex.diff

@UlrikeFischer @JosephWright pdflscape-test6 gets a full path in a /CMapName entry? /CMapName /c:-texlive-2018-texmf-dist-fonts-opentype-public-lm-lmroman12-bold.otf,000-UTF1‌​6 def

and pdfrender test5 gets different random prefixes in font names ! <</Type/Font/Subtype/Type1/Widths 12 0 R/FirstChar 49/LastChar 61/BaseFont/APWWDF+CMR10/FontDescriptor

@UlrikeFischer @JosephWright unless I'm missing something both look like engine failures in the "reproducible build" settings.

@DavidCarlisle But also quite a lot more tests ;-). But I'm getting the impression we are not only creating a test suite for oberdiek but also for l3build and for "reproducible builds".

@DavidCarlisle but it fails only for xetex right? I could remove the test, pdfrender doesn't work with it, I was only to lazy to create a config-plain-noxetex and so accepted yesterday a compilation with errors.

@UlrikeFischer always the way:-) yes they both just fail for xetex (the others are white space diffs in the log which I think we should be able to normalise away in l3build)

@DavidCarlisle imho the pdfrender can be ignored. I will check if the pdflscape indicates a real problem. What is the problem with askinclude-test24? (I'm down to five luatex-tests in the source folder ;-). When they are done I will check the failures and the "unclear" folder.)

@UlrikeFischer ask include is white space but a bit trickier as it covers a line break but if we add a typeout or forced break to get the )) file close away from the [1] then it will pass once the l3 white space at start of line issues are fixed, the diff is:

*** ./build/test-config-pdftex/askinclude-test24.tlg    2018-11-06 01:20:38.091229500 +0000
--- ./build/test-config-pdftex/askinclude-test24.pdftex.log     2018-11-06 01:20:40.581590100 +0000
***************
*** 53,61 ****
  (etexcmds)             That can mean that you are not using pdfTeX 1.50 or
  (etexcmds)             that some package has redefined \expanded.
  (etexcmds)             In the latter case, load this package earlier.
! )) [1
  ]
! (askinclude-a24.tex [2
  ]
  Chapter 1.
  ) [3]

@UlrikeFischer of course the output will change again once \expanded gets added:-)

@UlrikeFischer @JosephWright is always breaking something.

Quack!

@PauloCereda /bunny sound/

@Skillmon ooh hi mr. rabbit!

Das Bunnysound

German is so easy

<3

@PauloCereda Es heißt: Das Karnickelgeräusch.

@Skillmon uh-oh

Hallo, die Kartoffel ist böse

Und ein bier bitte

Ich arbeite heute und der erpel ist sehr cool

@PauloCereda A schnelles Helles?

Sorry, dieser Regenschirm gehört mir nicht.

@Skillmon ooh I don't know it

@PauloCereda A "Helles" is a type of beer.

@Skillmon ooh

Not a brand?

I googled Ente bier :)

@PauloCereda no, not a brand, a type. Like Pils, which is not a brand but a type.

@Skillmon ooh cool

@PauloCereda other delicious type is Zoigl.

@Skillmon ooh

How could one expandably test whether something is a single token, not ignoring spaces? I came up with the following (which ignores spaces:( ):

\makeatletter
\newcommand\ifonetoken[1]
  {%
    \ifonetoken@inner#1.\endifonetoken@inner
  }
\def\ifonetoken@inner#1#2#3\endifonetoken@inner
  {%
    \if\relax\detokenize{#3}\relax
      \expandafter\@firstoftwo
    \else
      \expandafter\@secondoftwo
    \fi
  }
\makeatother

With it \ifonetoken{a}{true}{false} expands to true, but unfortunately \ifonetoken{a }{true}{false} does so, too.

@Skillmon I might have misunderstood your question, but have you seen \tl_if_head_is_N_type:nTF?

@Skillmon Oh, forget what I said :/

@PhelypeOleinik I forgot to mention: This is for educational purpose, so without expl3 would be best.

@DavidCarlisle I asked on the texlive list about this.

@Skillmon You can always translate a whole bunch of expl3 to plain :D

@UlrikeFischer just saw, thanks

@Skillmon I tried once. Not cool.

@DavidCarlisle I changed the askinclude test to load the package earlier, so it should be fine now too (perhaps with the exception of a space), and I moved the plain-noxetex files to a dedicated folder.

@PhelypeOleinik Understanding it isn't very hard (depends on macro in question of course), just tedious.

@UlrikeFischer do you think that changing the selinput tests to ad OMIT TIMO as:

\input{regression-test}
\NeedsTeXFormat{LaTeX2e}
\documentclass{minimal}
\usepackage{textcomp}
\usepackage{qstest}
\makeatletter
\let\BeginDocumentText\@empty
\def\TestEncoding#1#2{%
\OMIT
\SelectInputMappings{#2}%
\TIMO

is safe? I don't think it misses any of the actual test log, but it hides the logging from different versions of the inputenc files, especially the utf8 one seems to get longer every release

@Skillmon Actually, I just looked at the source, and the code for \tl_if_head_is_N_type:nTF is pretty short.

@Skillmon But it's definitely not what I would consider "for educational purposes" :P

@PhelypeOleinik but it doesn't do what I want.

@Skillmon This seems to do it, if you're willing to do the translation:

\ExplSyntaxOn
\cs_new:Npn \test #1
  {
    \tl_if_head_is_N_type:nTF {#1}
      {
        \int_compare:nTF { \str_count:n {#1} = 1 }
          {true}
          {false}
      }
      {false}
  }
\ExplSyntaxOff

\test{a} % true
\test{a } % false
\test{a{}} % false
\test{{}} % false
\test{ b} % false

More or less what happened here. :)

@AlanMunn quack?

@DavidCarlisle Imho it should be ok. Couldn't one silence utf8 a bit?

@AlanMunn ooh

@PhelypeOleinik the \str_count:n is the interesting one.

@PauloCereda @DavidCarlisle ^^^Crispy duck

@PhelypeOleinik but it still isn't completely right.

@PhelypeOleinik I'll see what I'll find.

@UlrikeFischer well unfortunately it just uses \wlog directly so I was worried if i disabled \wlog then some of Heiko's logging may also go.

@DavidCarlisle I didn't meant for the tests here but in general. At least if the definitions are from a file. Then a message like "defining unicode chars from file xxx.dfu" would be quite enough.

@UlrikeFischer hmmm

@JosephWright the summary messages are wrong in l3build:

  Check failed with difference files
  - ./build/test-config-pdftex/selinput-test1.pdftex.fc

Failed tests for configuration build:

  Check failed with difference files
  - ./build/test/pdflscape-test6.xetex.pdf.fc

Failed tests for configuration config-pdftex:

  Check failed with difference files
  - ./build/test/pdflscape-test6.xetex.pdf.fc

@JosephWright ^^^^ the last line should point to the selinput-test1 failure.

@UlrikeFischer Oh bugger

@JosephWright should I add an issue?

@UlrikeFischer Please

@UlrikeFischer ask @JosephWright if there is complete agreement on the amount of logging definitions should do in general:-)

@DavidCarlisle ooh let's take a verbose approach

@DavidCarlisle ;-).

@DavidCarlisle you asked the question I just wanted to ask too ;-)

@UlrikeFischer I'm good at asking questions

@DavidCarlisle ooh

@DavidCarlisle /CMapName /HEXFDP+LMRoman10-Regular-UTF16 def <--- with newest xetex from win32tex.org

@UlrikeFischer Cool

@UlrikeFischer I could compile myself one, but I may just live with failure for a bit

@UlrikeFischer although I suppose I should check I also get HEXFDP not some other prefix...

@DavidCarlisle This could be a problem - I just called xdvipdfmx again and got /CMapName /CKLUPJ+LMRoman10-Regular-UTF16 def ;-(.

@UlrikeFischer ooh map woes

@DavidCarlisle but if the prefix is always 6 chars it should be possible (for @JosephWright) to normalize them and to replace them with /XXXXXX+.

@PauloCereda did you get the threema message?

@UlrikeFischer I did wonder, as one of the other diffs we got was different font prefixes

@UlrikeFischer uh-oh... Threema does not work with my phone anymore... :(

@UlrikeFischer for the tests that would be OK but it does mean the "reproducible build" settings don't make reproducible results.

@DavidCarlisle I naturally meant "replace with /ZZZZZZ+".

$ find . -name \*diff
./build/test/hycolor-test1.luatex.diff
./build/test/hycolor-test1.xetex.diff
./build/test/hycolor-test2.luatex.diff
./build/test/hycolor-test2.xetex.diff
./build/test/hycolor-test3.luatex.diff
./build/test/hycolor-test3.xetex.diff
./build/test/pdflscape-test6.xetex.pdf.diff
./build/test-config-pdftex/selinput-test2.pdftex.diff
./build/test-config-pdftex/selinput-test5.pdftex.diff
./build/test-config-pdftex/soulutf8-test2.pdftex.diff
./build/test-config-pdftex/soulutf8-test3.pdftex.diff

@UlrikeFischer much better

@UlrikeFischer ^^^ current list

@DavidCarlisle I just tested with SOURCE_DATE_EPOCH and FORCE_SOURCE_DATE and then the prefix seems to be stable.

@JosephWright ^^^

@DavidCarlisle, @UlrikeFischer Busy for a few days on the blog ... please log on GitHub, and I 'll sort it ...

@UlrikeFischer Phew

@JosephWright Ok. @DavidCarlisle can you add an issue for the single space business? I have to go now. I will do something about the remaining tests in the evening.

Is there a list with macros using \mathaccentV anywhere? Currently I know of grave, acute, hat, tilde, bar, breve, dot, ddot, check, vec. Anything I've forgotten?

@UlrikeFischer OK

@Skillmon possibly this?

$ grep -i declaremathaccent /usr/local/texlive/2018/texmf-dist/tex/latex/*/*.* | wc -l
422

@DavidCarlisle there are 422 matche? Now I am frightened.

@Skillmon welcome to my world

@DavidCarlisle why not using grep -r but the shell? At least I get problems with the shell if there are too many matches.

@Skillmon habit mostly

@Skillmon A little digging reveals that not that many diffferent math accents are involved: Many packages (re)define the same math accents. An approximate list is below (without the initial backslashes, due to the way I created the list using a hodge-podge of grep, tr, and sed commands in a pipeline too ugly to list here):

acute, bar, barbar, barhat, bartilde, breve, check,
ddddot, dddot, ddot, dot, fouriervec, grave, harpoonacc, hat, hatbar,
hathat, hattilde, lharpoonacc, lrharpoonacc, lrvec, lvec, mathring,
middlebar, middleslash, strokethrough, symbols, tilde, tildebar,
tildehat, tildetilde, vec, wideOarc, widearc, widearcarrow, widehat,
wideparen, widetilde, widetriangle

Ay up me duck!

Oh wait, it's another type of accent...

@PauloCereda Watch out for those arrows and harpoons, though. They may not be safe for ducks.

@HaraldHanche-Olsen oh no

@PauloCereda Try wearing a \widehat for protection.

@HaraldHanche-Olsen ooh

@HaraldHanche-Olsen Why is there no "overduck"?

@marmot hi mr. marmot!

@PauloCereda Hi Dr. Duck!

@marmot time to create the duckmaths package?!

@Skillmon ooh

we ducks are very good at maths

@Skillmon Like this?

@HaraldHanche-Olsen in amsmath only \hat, \check, \tilde, \acute, \grave, \dot, \ddot, \breve, \bar, \vec and \mathring are defined using \mathaccentV. Most of the other I think are defined without it.

@marmot yep.

@HaraldHanche-Olsen but thank you for your effort.

@Skillmon Ah, I never noticed the V in \mathaccentV in your original query. I was chasing down the hits from @DavidCarlisle's grep invocation instead.

@HaraldHanche-Olsen -- uh, what's "symbols"? that is totally void of useful information. all the others i either recognize or can visualize. (not implying anything about how they're defined.)

@UlrikeFischer github.com/latex3/l3build/issues/78

@barbarabeeton The ugly pipeline I mentioned (which I can't get at now, since it was on my work machine and I'm at home now) did produce some bogus hits (a.k.a. “fake news”). I weeded out the others by hand, but evidently missed that one.

@barbarabeeton Regular expressions have their limitations, after all.

@DavidCarlisle good ;-).

@UlrikeFischer I'm just trying the full set now, with that change, and some hand edits to your tlg to remove a space before [(

@DavidCarlisle I made a few changes in the afternoon, just uploading. I'm now looking at the "failures" folder and try to decide if one can do something about them.

@UlrikeFischer you have the benefit of being able to declare any behaviour a pass don't you? :-)

@DavidCarlisle ;-). Imho luatex-test4 fails due to the kernel allocation and should simply be dropped.

@UlrikeFischer as I say, it wouldn't take much to persuade me to drop the whole package

@DavidCarlisle Fine with me, I don't need any persuasion at all. Imho it is only confusing. Do you have list of all oberdiek packages?

@UlrikeFischer theer is the manifest at github.com/ho-tex/oberdiek/blob/master/manifest.txt but that's a list of sources, i'd have to search for .sty files in the build area to get a full list of generated packages

how bad would it be to have a one line luatex.sty that said \RequirePackage{luatexbase} or perhaps with a guard for plain tex and \input luatexbase in that case.

@DavidCarlisle ctan has a list:ctan.org/pkg/oberdiek.

@DavidCarlisle luatexbase or ltluatex?

@UlrikeFischer oh yes that's the list that means the ctan upload form rejects the submission ever time:-) (it auto-fills in the description by default, then if you forget to delete it it rejects teh submission as teh description is longer than the current limit:-)

@UlrikeFischer luatexbase I think (so it includes the catcode table stack from cctablestack), or could do ctablestack directly, or perhaps it could have some other custom compatibility code if I knew what it was doing in detail...

@DavidCarlisle or you could forget the package at the next update and look if anyone complains ;-).

@UlrikeFischer leave it there but comment out the generation from the .ins file, document how people should get it of they need it?

@UlrikeFischer speaking of luatex.sty, do you understand Heiko's Lua module loading system? I never quite worked out why each lua file is duplicated, luacolor.lua and oberdiek.luacolor.lua can we get rid of the oberdiek.zzz versions?

@DavidCarlisle Yes, that would be an option too. Does this work with the texsystems or will Karl generate nevertheless?

@UlrikeFischer if i comment it out, don't put it in the .tds.zip and mailed Karl and ask him to lose it, it would probably go:-)

@DavidCarlisle luacolor loads oberdiek.luacolor.lua, magicnums oberdiek.magicnum.lua, but pdftexcmds pdftexcmds.lua. One would need some older systems to be sure, but to me it looks as if the oberdiek-versions are newer, and the other are for compability.

@UlrikeFischer I'd still be tempted to change the .sty to load oberdiek.lua and magicnum.lua and lose the duplicates (and lose luacolor-pre065)

@UlrikeFischer it's not as if breaking changes between luatex releases is a rare event, i don't think we need absolute stability

@DavidCarlisle Getting rid of the duplicates is fine with me. But which one should be dropped? The oberdiek.XXX oder the XXX?

@DavidCarlisle No imho cleaning up and getting rid of old stuff is more important here.

@UlrikeFischer I'd get rid of the oberdiek.zzz ones they just look weird, no other lua packages for luatex have a name scheme like that as far as I can tell.

once we get the test suite fully up and running, I'll feel more confident making changes:-) (or I'll give you write access so you can be blamed if something breaks:-)

@UlrikeFischer well there is pgfmolbio.domains.lua and pgfmolbio.chromatogram.lua so not no other package but...

@DavidCarlisle the question is if they will be name clashes, or if he changed to oberdiek.yyy because of some name clash (lua module searching is a bit curious see e.g. tex.stackexchange.com/questions/414102/…).

@DavidCarlisle ;-) But you are aware that not every package has a (sensible) test? Looking e.g. at github.com/ho-tex/oberdiek/issues/31 I realized that grffile has no real tests.

@UlrikeFischer yes but since both files are there, if some other file was loading a different luacolor.lua it would clash already

@UlrikeFischer but if anything goes wrong:

/blame

@PauloCereda Where is that bot when I need it? ;-)

@DavidCarlisle you should wide the blame specification. "lualatex font issues" doesn't fit on grffile.

@UlrikeFischer 's/ for lualatex font issues//`

@UlrikeFischer Is there a way to query luaotfload font database without loading the rest of luaotfload (specifically without registering any of the luatex callbacks)?

@UlrikeFischer, @DavidCarlisle, @JosephWright I have the all basic functionality of HarfBuzz loading and shaping that I had in mind implemented (sans the bugs of course), so I think it is ready for wider testing before I start optimizing things or adding more functionality. Not sure how to move forward from here, any ideas how to make this lua code testable?

@KhaledHosny Well it is a lua table/file so in theory everone can query it. Did you look at luaotfload-tool? (There is an issue about this github.com/lualatex/luaotfload/issues/413, but I hadn't really time yet to think about it),

All remaining issues I have in mind are in github.com/khaledhosny/luahbtex-harf/issues

@UlrikeFischer Thanks, will check that.

@KhaledHosny the doc for luaotfload-tool claims it was written by someone called Khaled Hosny:-)

@DavidCarlisle That is the young and foolish me, all memories of that person are long gone, also I’m pretty sure I didn’t write most of it.

@KhaledHosny your test files says

\iftrue
  \directlua{dofile("harf-load.lua")}
  \directlua{dofile("harf-node.lua")}
\else
  \input luaotfload.sty
\fi

Does that mean that your code and luaotfload is basically incompatible?

@KhaledHosny I suppose the binary is the biggest obstacle, up to you really whether you want to distribute (say) linux and windows binaries, or to get it added to texlive so they do it, or to see if you can get it added to the core luatex distrib?

@UlrikeFischer Right now yes, we both compete on registering the same callbacks. To coexist, we will need changes to luaotfload (and my code of course), I think we discussed that earlier.

@KhaledHosny Does that mean that with your code every (opentype) font is shaped/handled by harfbuzz like in xetex?

@DavidCarlisle Things are still fluid on the binary side as I keep updating the the luaharfbuzz module as I need tp expose more of HarfBuzz’s API to lua (the API is big and only subset of it is exposed right now, though the plan is to eventually expose as much as possible)

@UlrikeFischer The code is two parts, a node list processor (for pre_linebreak_filter and hpack_filter) and a font loader (for define_font) If a font was not loaded by the loader, the processor will not touch all nodes using it. I use that right now to skip TFM fonts and lets the engine handle them, but should work with luaotfload in principle.

@UlrikeFischer My current idea is to make luaotfload require my two modules, pass fonts that use a certain flag to my loader, and some how allow my node list processor to run before luaotfload’s, this way it can decide which fonts to process or skip. The rest of luaotfload can keep working as it does now (in theory at least)

@UlrikeFischer I’m open to other suggestions. I haven’t following the Lua(La)TeX world for long and I’m propably missing other possibilities.

@DavidCarlisle I’d like to provide binaries myself, but I’m having hard-time cross compiling ICU. Not sure how native TL Windows binaries are being built, if this is documented somewhere I can try it.

@KhaledHosny there is some doc, but the actual accepted practice is "Ask Akira to add it to the win32tex distribution"

@KhaledHosny sounds good but we will need @MarcelKrüger for the details.

@DavidCarlisle I’m likely to require frequent builds at this stage, so I better give it a try.

@KhaledHosny there is also github.com/aminophen/w32tex-build not sure if that has all teh setup checked in or just the results

@DavidCarlisle Linux binaries (and possibly osx) are easy, would they help anyone right now?

@KhaledHosny yes:-) (even on windows people could (if only for testing) use the linux ones via the windows 10 linux subsystem) (I assume I could make the cygwin ones available as well (although i have not tried since I originally got your code working, I assume it should compile still if I update)

@DavidCarlisle Shouldn’t cygwin binaries run without cygwin as well?

@KhaledHosny no unlike mingw binaries which are designed to run in a native windows environment cygwin binaries need the cygwin dll and its posix stack running over windows, and assume a unix flavoured view of the filesystem (installs in /usr/local/texlive by default for example) so you really need at least the core of cygwin installed

@DavidCarlisle I see, let me work on the linux/osx binaries first, then try windows again.

@KhaledHosny didn't I see somewhere that harfbuzz uses some continuous integration setup that included windows machines in its builds?

@KhaledHosny tug.org/texlive/doc/tlbuild.html#Cross-configuring says In order to, e.g., build mingw32 binaries on x86_64-linux with a cross compiler found as i386-pc-mingw32-gcc one would specify --host=i386-pc-mingw32 --build=x86_64-linux-gnu which makes it sound easy:-)

@DavidCarlisle HarfBuzz builds fine, it is ICU that does not seem to be recognizing the cross-compiling toolchain.

@DavidCarlisle It should be easy if autoconf is being used properly