/dev/chat

I thought this thing about SSDs being too precious to actually be used for their intended purpose of reading and writing data was a thing of the past.

Andras Deak

Sounds like one can overwrite less than all blocks to hide the data well

Stephen Kitt

@terdon it won’t harm it, but it will use up one write cycle everywhere and give the block management system a vigorous workout; trimming or secure erasing would produce the same result without that.

terdon

9:41 AM

@StephenKitt I am pretty ignorant about this sort of thing. Why is giving the block management system a workout a bad thing? I mean, isn't that what disks are for?

Stephen Kitt

The endurance of SSDs in the marketplace goes up and down, it’s a bit difficult to give a general rule. QLC and PLC have lower TBW ratings for example.

@terdon well, it’s unusual for disks to be written end-to-end quickly ;-). Manufacturers do take some shortcuts, so it’s hard to say “yes, it’s definitely safe to cp /dev/zero /dev/sdX on SSDs”, especially when there are better approaches.

The best solution for both HDDs and SSDs (and tapes, and cloud storage, and whatever else) is encryption. Then you don’t need to worry about anyone recovering the raw bits from a disk, as long as they can’t get the key...

For home use I would say anything goes on SSDs, you’re never going to run out of cell writes.

But I suspect many of us know sysadmins who treat their enterprise systems the same way they treat their home systems (and vice versa), and sysadmins who apply random trivia gleaned from forums and chats.

My 9-year-old Crucial m4, which sees a decent amount of activity (over 57TiB written), is only at 65% of its rated life...

terdon

Huh. Thanks

Andras Deak

10:25 AM

@StephenKitt isn't encryption hella slow on an HDD?

Stephen Kitt

@AndrasDeak why would it be slow? You end up reading and writing the same amount of data, it’s just different data...

Andras Deak

because HDDs are slow, and en/decryption is more overhead

So yeah, I meant encryption plus HDDs, together

Stephen Kitt

Well, my eight-year-old desktop can process AES-256-CBC at the rate of over 500MiB/s on a single core; my one-year-old laptop handles over 1GiB/s on a single core. So the cost of encryption/decryption is small compared to the I/O time on an HDD; adding encryption doesn’t significantly affect performance.

“Encryption plus HDD” is slow, but only because “HDD” is slow.

Andras Deak

10:45 AM

Thanks, I'll try to remember that

Faheem Mitha

@terdon Thank you for the feedback. Yes, I agree that SO seems like the most suitable choice. Though I don't know if it is necessarily off topic here.

Andras Deak

in any case the HDD in question is not mine, I'm just doing a favour wiping it clean

Faheem Mitha

1) OK, I'll look again, though I'm not aware of any other options. But I've only been using Python 3 for a short time. Which docs should one look at?
2) (Frame challenge?) Yes, I looked at it. And it works. Thank you, @JeffSchaller for the useful suggestion. But I still would like to know what is going on with pexpect. I don't see any option on the POP library for manually taking control of the session, so cannot compare that part directly.

Expect offers very general machinery, so it's useful to know about.

Stephen Kitt

TIL that Asterix has a tag on SF&F — I never thought of Asterix as SF&F!

Andras Deak

@FaheemMitha Do you need very general machinery? Don't you just want to download your email?

This is a very important question

a frame challenge is the nice way to ask if the other person has an XY problem

Faheem Mitha

10:51 AM

@AndrasDeak Sure. There have been numerous instances in the past where knowing about such a thing would have been useful. But I only learned about Expect relatively recently, when I used it to answer another question.

Andras Deak

Oh, I thought pexpect was also stdlib. This makes it even worse.

Faheem Mitha

Having various sorts of machinery available is generally useful. And I don't go looking for things I don't understand. But if I don't understand them, I try to figure them out.

@AndrasDeak I'm not familiar with that terminology.

@AndrasDeak Worse?

Andras Deak

@FaheemMitha That's a bit like saying that if you want bread it's better to buy the bakery because it gives you more options. But OK.

Stephen Kitt

@FaheemMitha cue youtube.com/channel/UCbNvfx3rYYxEopnRGxfu53Q

@AndrasDeak and the mill!

Faheem Mitha

An example is relational databases, which I thought was complete rubbish for a long time. But when I was forced to learn about it through circumstance, I was able to use it, and it's been very useful.

@AndrasDeak Weird analogy, but that's your opinion.

@StephenKitt ?

Andras Deak

10:55 AM

@FaheemMitha the documentation I meant was pexpect.readthedocs.io/en/stable/api/pexpect.html#spawn-class but apparently passing in the encoding you did is what you're supposed to do, so my hunch is probably wrong about where the error is coming from.

Faheem Mitha

@AndrasDeak Not necessarily. I'll check on it now. But if not, I'll go ahead and post the question. It doesn't seem like complete rubbish. Of course that won't stop the SO folks from downvoting it.

Andras Deak

It's a reasonable question but it doesn't have an MCVE due to the POP3 server stuff, so it's still a gamble how it will be received.

Faheem Mitha

@AndrasDeak To be clear, the encoding is definitely the issue. It's worth checking on the sys.stdout thing.

@AndrasDeak MCVE?

Andras Deak

@FaheemMitha but the docs I linked shows exactly what you're doing if you scroll down a bit

@FaheemMitha that's SOese for MWE

Faheem Mitha

I can probably reproduce it without using POP. Let me check.

@AndrasDeak I've looked through the pexpect docs quite a bit.

@AndrasDeak What does MCVE mean?

Andras Deak

10:59 AM

Do you know what MWE means?

Faheem Mitha

@AndrasDeak Minimal Working Example, I think.

Andras Deak

Yeah, it comes up a lot on tex.SE so I figured you might be familiar that one.

stackoverflow.com/help/mcve

Faheem Mitha

@AndrasDeak There is some relevant discussion on docs.python.org/3/library/sys.html

There doesn't seem to be a direct link to sys.stdout.

It seems to be saying that it picks up the system encoding, but I'm not sure.

> The encoding and error handling are is initialized from PyConfig.stdio_encoding and PyConfig.stdio_errors.

Trying to figure out how to check that.

> python -c 'import sys; print(sys.getdefaultencoding())'

utf-8

Interestingly, one of pexpect's own examples is exiting with an error.

faheem@orwell:/usr/local/src/pexpect/pexpect-4.8.0/examples$ python3 script.py
Traceback (most recent call last):
File "/usr/local/src/pexpect/pexpect-4.8.0/examples/script.py", line 114, in <module>
main()
File "/usr/local/src/pexpect/pexpect-4.8.0/examples/script.py", line 85, in main
fout.write ('# %4d%02d%02d.%02d%02d%02d \n' % time.localtime()[:-3])
TypeError: a bytes-like object is required, not 'str'

Andras Deak

11:29 AM

@FaheemMitha that's great! Check if there's an issue with it already, and if not, open one

Faheem Mitha

@AndrasDeak Will do.

Andras Deak

0

Q: Pexpect interact() exception in python3

I am in the process of learning Python so apologize if my question is naive. I was using the pexpect interact() in my script and it was working with python2.7. But when I use python3 or python3.5 I get the below error: Traceback (most recent call last): File "ap_access.py", line 73, in <module>

Faheem Mitha

Maybe this version is horribly out of date.

@AndrasDeak Yes, I saw that one, and it appears to be similar, though I did not run his example. Perhaps I should.

Andras Deak

Eh. It's the exact same issue as yours.

Faheem Mitha

However, the answers appear to be nonsense, as far as I can tell. Though I'm not really familiar with these encoding issues.

Andras Deak

11:32 AM

Oh, then there's no solution I guess ¯\_(ツ)_/¯

my bad

Faheem Mitha

@AndrasDeak Thank you for reminding me of that question. I'd seen it earlier, but not paid much attention. Let me at least see if I can reproduce what he is getting. I'll also check for more recent versions. Unfortunately Debian isn't always good at keeping up.

Andras Deak

Why not try the actual solution from the answer?

Faheem Mitha

@AndrasDeak You mean std.stdout.buffer? I did. It doesn't work. Actually, it crashes immediately. Also, it makes no sense.

Andras Deak

So why not lead with that?

Stephen Kitt

@FaheemMitha Debian 11 has pexpect 4.8.0, which is the latest release

Faheem Mitha

11:36 AM

@StephenKitt That's what I'm using.

Andras Deak

"it appears to be similar, though I did not run his example. Perhaps I should."
"the answers appear to be nonsense, as far as I can tell."
"I'd seen it earlier, but not paid much attention. Let me at least see if I can reproduce what he is getting."

Don't you think "I've seen that question and the answer doesn't work, it even crashes immediately" would be something important to note?

Faheem Mitha

@AndrasDeak I haven't actually tried running that user's example. I tried the "solution" in my own code, of course. Yesterday, I think. Along with other things.

@AndrasDeak Sorry. My bad.

Andras Deak

@FaheemMitha yes, that's the expected procedure

not running code in questions is the right call usually

Faheem Mitha

@AndrasDeak Well, it's OK as long as you read it and understand it first.

I run example code from the net all the time. But you need to know what it does.

Actual malicious code on forums is very rare, in my experience.

Andras Deak

yeah, not all fora are as dangerous as the askubuntu chatroom :P

Faheem Mitha

11:43 AM

github.com/pexpect/pexpect/issues/506

Looks like this is an open issue, if I'm understanding correctly. Sigh.

Andras Deak

yup, seems so

you can try applying the unmerged github.com/pexpect/pexpect/pull/539 :P

Faheem Mitha

A list of things I didn't want to think about today includes Python encoding issues.

@AndrasDeak If I can understand what it's about.

Andras Deak

you can always try poplib

Faheem Mitha

Probably no point posting an issue on SO then.

And if it's so broken, perhaps I should stick with Expect. Though I don't really want to deal with Tcl.

Andras Deak

@FaheemMitha it would be a duplicate of what I linked earlier

Faheem Mitha

11:45 AM

@AndrasDeak I already did. Like I said, it works.

@AndrasDeak It would be, yes.

Andras Deak

there's a workaround in github.com/pexpect/pexpect/pull/539#issuecomment-1015927720 that uses an actual file for logging, for what it's worth

Faheem Mitha

> pexpect implemented workaround for writing to stdout in the method defined above, but not for logfile.

Not sure what that means.

Andras Deak

pexpect's last commit was in April :/

@FaheemMitha sounds like the same error used to happen in other situations, that was patched (special-casing stdout, probably), but the same was not done for logfile handling

for what it's worth the PR's fix seems like a no-brainer github.com/pexpect/pexpect/pull/539/files

Faheem Mitha

@AndrasDeak Oh, I see. He means it's still unfixed for log files?

Andras Deak

but you're using system-installed pexpect so you shouldn't really mess with that

@FaheemMitha that's how I read it.

Faheem Mitha

11:48 AM

@AndrasDeak I wonder why it hasn't been fixed, then.

Andras Deak

because of github.com/pexpect/pexpect/pull/539#issuecomment-1015896047

Faheem Mitha

@AndrasDeak I could patch and rebuild the Debian package, but it doesn't seem to be worth the trouble.

Andras Deak

ah, yes, the PR's "fix" hard-codes utf8 encoding, that's bad

but if you're going to hack your local copy you can do whatever you want, e.g. use sys.getdefaultencoding which is probably OK most of the time anyway

Faheem Mitha

Also, doing unofficial patches isn't a great idea.

Andras Deak

@FaheemMitha oh yeah, you like doing that

Faheem Mitha

11:50 AM

@AndrasDeak Not really. But I do it sometimes. But only for official patches.

Andras Deak

as an alternative to editing a single line in a single file :P

(OK, two lines)

Faheem Mitha

@AndrasDeak Well, that would get lost in the next upgrade.

Andras Deak

true, but it's not like pexpect is being maintained anymore

Faheem Mitha

I thought it was user error, because it seemed like pretty basic functionality to not be working.

@AndrasDeak It isn't?

Andras Deak

It could be, but as I said the last commit was made almost a year ago

That's usually not a great sign, especially with 180 open issues and 19 open PRs. It's not like there's nothing left to do.

Faheem Mitha

11:52 AM

@AndrasDeak Yes, I see that. Not encouraging, certainly.

@AndrasDeak Agreed. Emphatically.

Fine, I will now return to my regularly scheduled programming, because it doesn't look like there is anything to be done here.

@AndrasDeak Thank you for the assistance.

Making a mental note that trying to use pexpect for real work will probably not end well. Which is a shame.

Andras Deak

last release almost exactly 2 years ago

Faheem Mitha

The poster appears to be asking two questions in one. The first one, the issue with interact appears to be an open bug. See github.com/pexpect/pexpect/issues/506 It does not look like Pexpect is a very active project, so it's unclear when this will be fixed. The issue itself is from 2018. — Faheem Mitha 22 secs ago

I wonder if a comment is appropriate. But I can't think of anything else to do.

Andras Deak

yeah, that's all you can do

Faheem Mitha

The TeX people try to take care of abandoned projects. At least fix the bad bugs. But I guess they're in a minority. Most communities don't try.

Though the TeX community is special in a number of ways.

Andras Deak

If there's a low bus factor and a maintainer meets a bus it's hard to carry on. It's enormous responsibility to pass on maintainership of a python project, because the maintainer can add malware any time they want, affecting all downstream users. I've seen more than one project where the maintainer said "I have to stop carrying this project, and all 3 people I would trust with it said no"

Faheem Mitha

12:08 PM

@AndrasDeak Agreed.

ibuprofen

2:44 PM

Anyone know how to prevent LibreOffice from changing text (I.e. When I write SOkAL, it changes it to SokAL, but I want SOkAL)

Have figured it out before, but suppressed from memory :P

It's annoying as ... something

Stephen Kitt

@ibuprofen Tools / AutoCorrect / While Typing

ibuprofen

Thanks, perhaps something new. Do not have it 6.0.7.3 - but found another setting :),

Removed the Correct TWo INitial CApitals

ilkkachu

7:41 PM

hah, that's one of the less annoying autocorrections, IMO

Faheem Mitha

8:28 PM

The getmail documentation is using the terminology ~USER. Anyone know what that signifies?

Probably something stupidly obvious.

Context: getmail6.org/configuration.html#destination-mboxrd

OK, I came across this randomly on SO when doing a search.

> git-config will expand leading ~ to the value of $HOME, and ~user to the home directory for the specified user.

That's probably what it means.

I've not come across this previously, though. Is it standard notation?

Jeff Schaller

yes: " ... the characters in the tilde-prefix following the <tilde> are treated as a possible login name from the user database ..."

jesse_b

8:45 PM

Home inspectors are flimsy. They have to much CYA verbiage in their reports

"Electrical panel is currently on fire which in my opinion is not ideal but you should contact an electrician to be sure"

Jeff Schaller

in a hot housing market, they're probably just happy to have work!

Andras Deak

@FaheemMitha in bash ~user expands to /home/user, yes

Jeff Schaller

(or whatever the user has configured as their home directory, yes) :)

Andras Deak

yeah yeah yeah :P

I press the button and the computer lights up

Jeff Schaller

it's a good day

just don't try opening any cans

Faheem Mitha

8:49 PM

@JeffSchaller, @AndrasDeak Wow, I must have missed that in 25+ years of using Unix. Live and learn.

Andras Deak

@JeffSchaller fair point though, because I only use that syntax on computer clusters where the home is usually not in /home

Jeff Schaller

It seems a little gratuitous in that example of fetching mail, since I'd think $HOME or ~ would suffice; but apparently you can download mail to someone else's home directory.

Andras Deak

$ echo $HOME
/heart

Faheem Mitha

I also didn't realise that ~ is actually replaced by $HOME.

Jeff Schaller

a cluster of love, just in time for Valentine's Day! Your wife must be very understanding!

Andras Deak

8:51 PM

@FaheemMitha don't you have a ~ in your shell prompt to that effect?

also scp must be a huge pain if you keep typing out full paths to homes

Faheem Mitha

@AndrasDeak Probably. Haven't looked at it in a while.

Andras Deak

@JeffSchaller ah, the two messages were unrelated :P

Jeff Schaller

until

Faheem Mitha

@AndrasDeak I know ~ represents home, I didn't realise it was explicitly the value of the env variable HOME. Or I've forgotten.

Andras Deak

Ah, OK. What else could it be?

Jeff Schaller

8:52 PM

if Stephen's been on your machine, it could be a subdirectory of $HOME!

Andras Deak

Note to self: never let Stephen on my machine

Jeff Schaller

Aug 9 '21 at 13:30, by Stephen Kitt

When I was a student a common trick for people who left their workstation unlocked was to create a ~ directory inside their home directory, and change their shell startup script to change to it — that way, on log in, their prompt would suggest they were in their home directory but they would see no files...

Faheem Mitha

@AndrasDeak It could have been sourced a different way, that's all.

Andras Deak

At least I know where to ask for help if I need to wipe a ~

ilkkachu

9:04 PM

@JeffSchaller ahaha :) one more reason to have the full path in the prompt, not just the last part (or the full path starting from ~, like Bash's \w vs \W)

Transcript for