The Heap™ – Consultancy ©®

> The themes these groups took on are:

Better mechanisms for community feedback
Building a moderator advisory group
How we communicate and interact with Meta sites moving forward

But also:

> Themes that you said frustrated you over the past few months were:

an unwelcoming community
site design (cluttered pages, confusing navigation across the Stack Exchange network, etc.)
artifact quality (outdated answers, poorly framed questions, etc.)
barriers to participation

They must have been ignoring meta if those are the most frustrating topics they identified

George.Palacios

8:50 AM

Morning

Faheem Mitha

9:23 AM

Does just reading (but not writing) an SQLite database (which is just a file) change the file, in the sense that the md5sum would change?

More specifically, does the SQLite db retain any information about reads? Or even creation time?

Peter Vandivier

seems like that'd be easy enough to test ;p but somehow i'm guessing you're asking because you have a more specific problem you're trying to solve rather than the general case

Faheem Mitha

@PeterVandivier Well, I've got a reproducibility issue. I'm running a LuaTeX file, which reads from a db.

latexmf seems to be unhappy about the database changing. I'm not sure why.

But if it's unhappy, then it doesn't stop trying to compile.

Which isn't ideal. So I'm just trying to track down what might be going on.

George.Palacios

Morning

Peter Vandivier

i'm just guessing - but i'm betting it's down to the specifics of how your check is running vs what it's actually trying to assert

but lemme try real quick

Faheem Mitha

And it's not easy to figure out what has changed in a db. It can potentially contain a lot of meta-information.

@PeterVandivier I don't follow that sentence.

Peter Vandivier

9:33 AM

if you're trying to assert that nothing has changed in one table, but you're checking the hash of a whole directory, for example

Faheem Mitha

@PeterVandivier I'm not personally doing any checks in the db, wrt to changes.

But latexmk seems to be some kind of check.

I've noticed that the md5sum of the SQLite db seems to be changing with different runs. But the size is remaining the same, and the data being loaded is the same.

So what is changing?

To be more explicit, every time I compile my LuaTeX file, I load a db and then read it. But the data loaded each time is the same. So in theory the db file should also be the same.

But it isn't. I'm wondering what the difference is due to.

Peter Vandivier

there's a number of ways you could check

Faheem Mitha

@PeterVandivier Ok. Is this worth asking as a question? Or not?

Peter Vandivier

if you can dump the whole database to a flat text file befor / after, you could run a diff

or do a binary diff, but that'd be more work to actually pin down to the changes

Faheem Mitha

@PeterVandivier Ok. Would that completely determine the SQLite db?

Peter Vandivier

9:37 AM

@FaheemMitha if you can minify the reproduction, then yes absolutely

Faheem Mitha

I mean the text file dump?

@PeterVandivier minify?

Peter Vandivier

"make it a good question"

[repro]

Faheem Mitha

@PeterVandivier Ah, yes. You mean provide a MWE.

A Minimum Working Example.

Peter Vandivier

:p yes although i've never used that acronym

Faheem Mitha

@PeterVandivier TeX SE uses it. They love their MWEs there.

Peter Vandivier

9:40 AM

it's called mcve 'round these here parts

18

Q: Minimal, Complete, and Verifiable Example

How can I best ask my question so people will be able to understand it and reproduce the problem if necessary?

Faheem Mitha

Minimal working example

In computing, a minimal working example (abbreviated MWE) is a collection of source code and other data files which allow a bug or problem to be demonstrated and reproduced. The important feature of a minimal working example is that it is as small and as simple as possible, such that it is just sufficient to demonstrate the problem, but without any additional complexity or dependencies which will make resolution harder. A minimal working example may also be referred to as minimal reproducible example or short self-contained correct example. == External links == Short but complete programs (Jon...

Should have been minimal, not minimum, sorry.

Peter Vandivier

tbh, the work of coming up with a repro usually answers the question in my experience

3

Faheem Mitha

But to return to my question. Would a flat text file dump of a SQLite db contain everything in that db?

Peter Vandivier

but fwiw - i can tell you that adding a record and then removing it does appear to consistently change the file hash of my sqlite db to a new random hash

Faheem Mitha

@PeterVandivier Ok. Thank you.

Peter Vandivier

9:42 AM

meaning the file is not re-written deterministically

Faheem Mitha

@PeterVandivier Ok. That's unfortunate. Any idea why?

Does it record timestamps, perhaps?

Peter Vandivier

just how it is, i suppose. don't know enough about sqlite db inernals to comment

Faheem Mitha

Ok.

Peter Vandivier

@FaheemMitha possibly 🤷‍♂️

i would be surprised is reads always changed the hash, but i would also be surprised if reads were guaranteed to never change the hash

Faheem Mitha

Are there people here who might know about the internals?

Though possibly the best thing might be to instruct latexmk to simply ignore the db.

Peter Vandivier

9:46 AM

so you might wanna determine how much of your day you wanna spend on just figuring out the underlying reason to satisfy curiosity, and how much you wanna spend fixing the actual bug in your test

@FaheemMitha boom goes the dynamite

;p

but if you can get an mwe, i do encourage you to post it on main, that's the place to ask a question properly

Faheem Mitha

@PeterVandivier I'm not sure that there is a bug. Like I said, latexmk is concerned that the db is changing, but it's also normally not the case that it doesn't concern itself with dbs. The number of people who use dbs with TeX is vanishingly small.

@PeterVandivier boom?

Peter Vandivier

@FaheemMitha goes the dynamite

Faheem Mitha

It's normally a sensible thing to check the the sources aren't changing. Typically like the aux file. That's a reliable way, in general, to make sure the file has compiled properly.

@PeterVandivier I looked at the page, but don't see the relevance.

Peter Vandivier

relying on the binary structure of a database file to be deterministic seems pretty brittle imo

Faheem Mitha

@PeterVandivier I don't think latexmk knows it's a db file. It just thinks of it as a source file, I imagine. Though how it figured even that out, I don't know.

Peter Vandivier

9:50 AM

it sounds like you're checking the database file hash to see if something has compiled properly - if this is correct then that's not something i'd recommend

then yea, probably just change the check

Faheem Mitha

@PeterVandivier latexmk might be doing so. I'm not. It's an automated tool.

I'd have to look at the documentation. And if that fails, the source. Though whether it's worth spending the time is doubful.

Peter Vandivier

if you need to assert that everything in the db is logically the same - then come up with a query that will deterministically return all data, dump it to a flat file, and check the hash of that

which is actually a thing i have done

Faheem Mitha

@PeterVandivier Fair enough. Though even that is probably overkill in this case.

Since the db isn't actually changing. Though it might be technically changing, as already discussed.

dezso

@McNets good luck - but why a Food and Drug Administration audit?

@Johnakahot2use الانضمام الصحيح

McNets

@dezso it is related to Medical devices, avinent.com/en

dezso

9:58 AM

@McNets I see

I thought it was a joke from me

Peter Vandivier

@dezso YMUNO'N LLAWN

dezso

@PeterVandivier Welsh for 'get off my lawn'

Peter Vandivier

10:19 AM

George.Palacios

@FaheemMitha seems like your question is faulty to me. Are you looking for change in data, metadata or both?

Faheem Mitha

@George.Palacios I know the data isn't changing, because I'm loading the same data every time, and not making any changes to it.

So any change is presumably due to metadata.

Tom V - Team Monica

10:41 AM

@FaheemMitha Loading the same data will almost certainly change the file, even if the data is the same, I would guess

Unless I'm missing something you started with saying that the database is only read from, not written to. Loading data will certainly write to the database, even if the data is the same. Unless by loading you mean "testing if the record is already there and only insert it if it's new" but that would seem like a lot of trouble if you are loading the same data

George.Palacios

10:59 AM

So at that point an assumption is made that writing into a database is a deterministic process, which in all cases I can think of it is not.

Peter Vandivier

@George.Palacios csv files 🔥

George.Palacios

> csv

> Database

which I guess technically it is

but eeeehhhhhhhhh

Peter Vandivier

wat is databās?

Faheem Mitha

@TomV-TeamMonica I was being imprecise. What I meant to say is that after the database is loaded (from a CSV file), no further changes are made.

dezso

12:46 PM

@PeterVandivier that's Latvian for database

sort of

today I understand all languages

Peter Vandivier

Tom V - Team Monica

@dezso Are you trying to make us use the plural meme?

Peter Vandivier

the... the... fór͜bidde͡n m̀em̷e

John aka hot2use

1:05 PM

@PeterVandivier I hate it when you do that.

And only because you can.

Peter Vandivier

ᕕ( ᐛ )ᕗ

ConcernedOfTunbridgeWells

1:36 PM

QOTD: “There are only two kinds of languages: the ones people complain about and the ones nobody uses.”
- Bjarne Stoustrup

Peter Vandivier

i feel like there's probably non-zero overlap on those categories, too fwiw

mustaccio

@Johnakahot2use Every join an expert uses is right.

Tom V - Team Monica

@FaheemMitha That's why chat is a difficult format to answer a question. It almost inevitably leads to scattered information which is better consolidated in a well thought out question on main.

Andriy M

To be fair, some questions are unfit for main (while being relatively fit for chat), but the point about the information getting scattered stands nevertheless.

dezso

2:43 PM

@TomV-TeamMonica no, why

it totally looks like a Latvian word

Faheem Mitha

3:02 PM

@TomV-TeamMonica I'm unsure whether the question is suitable for main. The quastion, again, is why loading the same data into an SQLite db n times produces n different files, as measured by md5sum, for example.

This isn't exactly a mainstream sort of db question. It might even be off-topic.

Tom V - Team Monica

@dezso Google translate told me it was databases (i.e. plural)

@FaheemMitha It will be hard to get an exact answer unless somebody is familiar with sqlite internal storage structures

But I would expect most databases to behave the same

Faheem Mitha

@TomV-TeamMonica Are there people here who are?

@TomV-TeamMonica For something as small as SQLite, it might be better if it didn't.

There is something to be said for deterministic behavior.

Tom V - Team Monica

If you load the same data in SQL Server n times you will end up with different files too. Due to different fragmentation perhaps, different ordering of data pages inside the data file, different statistics which are calculated based on a somewhat random data sample etc

Faheem Mitha

See, for example, Debian's reproducible builds initiative.

@TomV-TeamMonica Sure, but SQL Server is a much bigger and more complex beast, isn't it?

Tom V - Team Monica

I agree but I would assume there is something inside sqlite that could cause the file to differ by one bit if you perform the actions you describe

which that is exactly I don't know, maybe somebody with intimate sqlite internals knowledge could tell you

All I can say is that I don't think anybody in this room has that kind of knowledge. I don't know if anybody on main does.

Faheem Mitha

3:09 PM

@TomV-TeamMonica Would the question be on topic?

Tom V - Team Monica

The more pertinent question may be why you are loading the same data over and over. If you are sure the data didn't change, why are you loading it? If it only changes sometimes, do you have a way of determining that before loading it?

Otherwise, do what Peter described, check if the data has changed or not.

Faheem Mitha

@TomV-TeamMonica Fair question. It's for compilation of a TeX file. Which requires multiple compilations.

Tom V - Team Monica

@FaheemMitha It might be, it could be too localized if the community thinks it won't help anybody. If well researched and asked it might be received well.

Paul White says GoFundMonica

@FaheemMitha I think so, yes. There are a couple of SQlite experts who could answer your question I believe.

Faheem Mitha

And I don't see any obvious or simple way of checking whether it's changed between compilations.

Tom V - Team Monica

3:11 PM

@FaheemMitha But do you want to know if the file changed or if the data changed?

Faheem Mitha

@TomV-TeamMonica @PaulWhitesaysGoFundMonica Ok, thank you. I'll consider whether to ask the question.

@TomV-TeamMonica The SQLite file. Which is the same as the db.

Tom V - Team Monica

Well I'm not surprised the file changes if you reload the same data again

I just can't tell you the exact reason why

Andriy M

@TomV-TeamMonica My guess is, the file change is used as a simplified way of determining whether the data changed.

Faheem Mitha

Ok.

Tom V - Team Monica

@AndriyM Which it shouldn't be. But maybe the flle change triggers something unwanted in the build, which could be avoided

Faheem Mitha

3:14 PM

It's not particularly conventional to load data from a db to build a LaTeX file. But that's how I'm doing it.

Tom V - Team Monica

such as copying the data file somewhere, comparing the data with that backup after load and moving the old data file if the data hasn't changed

depends on the use case I suppose

or checking if the data changed so it even needs to be loaded

Andriy M

@TomV-TeamMonica I wouldn't rely on that method either. At least when I know that the act of uploading really takes place, regardless whether it's the same data or not.

Tom V - Team Monica

@AndriyM As I said, depends on the use case

I'm not really sure what the actual problem is that we're trying to solve

That being said, I think a question "which internals cause the data file to be different when loading the same data" could be interesting for sqlite, even if it only teaches something about how data is stored in sqlite

idk

Paul White says GoFundMonica

That's the bit that seems topical to me

Tom V - Team Monica

"help me fix my latex build" probably won't be

Andriy M

3:19 PM

"How to quickly determine whether any part of the uploaded data was actually new" might be.

Paul White says GoFundMonica

@TomV-TeamMonica we could migrate it to

$TeX - LaTeX$

😀😀😀

Faheem Mitha

@TomV-TeamMonica I wasn't asking for help with TeX. There's a site for that.

Tom V - Team Monica

@PaulWhitesaysGoFundMonica I heard Bill is our resident latex expert

Paul White says GoFundMonica

heh heh heh

ypercubeᵀᴹ

morning

Faheem Mitha

3:26 PM

@TomV-TeamMonica Bill who?

Paul White says GoFundMonica

@ypercubeᵀᴹ morning

mustaccio

@FaheemMitha Your "problem" stems from you treating a database as source code, which it isn't, and you should stop treating it as such. Contents may be source code, but not the database.

Faheem Mitha

@mustaccio I wasn't. latexmk might have been doing, though I'm really not sure.

mustaccio

Stop blaming latex. You're in control.

Faheem Mitha

I've already said this several times, but the issue is being caused by a third party build tool.

Written in Perl, I think.

@mustaccio It's a build tool. It's not TeX.

mustaccio

3:34 PM

If the tool doesn't do what you want it to do, throw it away

Or sell it

Faheem Mitha

@mustaccio It usually does the right thing. It's as good as it gets.

mustaccio

SQLite is also as good as it gets. Now what?

Faheem Mitha

If that question is for me, I don't understand what you are asking.

Paul White says GoFundMonica

This is why the issue needs to be posted as a question with succinct information all in one place.
It gets tricky for people to follow in chat.

Faheem Mitha

Ok. Anyway, I've got nothing further to add at this time.

Paul White says GoFundMonica

3:57 PM

82

Q: Focusing on race, gender and sexuality makes minorities feel exposed and unwelcome

Before the Code of Conduct debacle, I never felt like a member of a minority. I was just a developer asking and answering questions. Now I feel pressured to come out of closet and announce every minority I belong to whenever I want to discuss anything and not be branded as a bigot. During the la...

discussion code-of-conduct the-loop

some good points being made there

but I might be out of "the loop" on this one

Peter Vandivier

4:14 PM

@FaheemMitha fwiw, SQLite is "small" only if you're referring to "usual size of data stored", it's hardly a niche product db-engines.com/en/ranking

although, to be fair, i suppose sqlite is on page 3 of our top tags, so maybe you have a point 🤷‍♂️

Faheem Mitha

@PeterVandivier I meant size of the codebase.

Peter Vandivier

4:29 PM

i've been staring at this slack invite for 15 minutes like Leonidas look up at the arrows

> our marketing spam will blot out the sun

TIL about temp-mail.org

god bless the internet

also... is it just my browser or is @TomV-TeamMonica 's starred agile comment truncated... suggestively?... 🤨😬

Paul White says GoFundMonica

not just you

dezso

4:48 PM

@TomV-TeamMonica datu bāzes it tells me

Peter Vandivier

this chat is full of data baes

Max Vernon

hmmm stop the merry go round seems to have gone dark

dezso

@MaxVernon it finally stopped?

Max Vernon

well, no messages for 4 days.

I hate to think it's because I was removed

well, perhaps hate is too strong a word

ego, eh.

Peter Vandivier

5:57 PM

@MaxVernon I think you were added back in?

Yea looks like

Josh Darnell

The merry-go-round turns without ceasing, winding through space while the world stands still. Does time cease it's inexorable march? So, also, goes the merry-go-round.

ypercubeᵀᴹ

6:58 PM

@MaxVernon it's Thanksgiving in America's. Festive weeks, before and after ;)

Peter Vandivier

7:10 PM

@ypercubeᵀᴹ ...among other reasons (•ﺑ•)...

Forrest

8:51 PM

My ADR article got published! :D

5

First time I've written an article

Max Vernon

sweet, @Forrest !

@PeterVandivier yah, no. But, it's ok actually. I'm over it :-)

Peter Vandivier

9:26 PM

@Forrest 4/5 stars, needs more gifs :p

real talk though: that's awesome! 😁and the animations are pretty baller

Forrest

@PeterVandivier Thanks :)

mustaccio

9:45 PM

support.hpe.com/hpsc/doc/public/…

Some people seem to still live in the 32-bit era

> The issue affects SSDs with an HPE firmware version prior to HPD8 that results in SSD failure at 32,768 hours of operation (i.e., 3 years, 270 days 8 hours). After the SSD failure occurs, neither the SSD nor the data can be recovered. In addition, SSDs which were put into service at the same time will likely fail nearly simultaneously.

That's some ingenious encoding right there: "32,768 hours of operation (i.e., 3 years, 270 days 8 hours)"

Max Vernon

jeeeeeezus. makes you wonder what they were thinking.

strange that the digits from 3 years, 270 days, 8 hours is so close to 32,768...

3 years * 365 days + 270 days * 8 hours + 8 hours = 32768 hours.

Max Vernon

10:07 PM

needs some brackets in there, but you get the point, I believe.

Peter Vandivier

illuminati confirmed

Paul White says GoFundMonica

11:39 PM