@user I heard that people mostly get Masters in CS because they wanted to switch career

@RadvylfPrograms more like, "belongs in the trash"

@Neil ooh, that explains a lot; in a previous job, I taught programming to university students, and was confused at why the first years were so much better (on average) than the masters students

(a small proportion of the masters students were very good, but many were terrible)

the first years will mostly have been people who have been interested in CS/programming/computers for years and now are old enough to learn it properly, so if the masters students are just tuning into programming for the first time because they want a new career path…

Given the prevalence of programming related jobs, that doesn't surprise me one bit

reminded by the starboard: on the subject of dictionary compression, I've been working on trying to design a golfing language compression dictionary based on the Google Books corpus – it keeps surprising me with what's in there

this recent challenge got me to check the dictionary, and, e.g., żadnych was in there

this sort of thing would be helpful to a golfing language because it means that you don't have to break the SBCS in order to include non-ASCII words

although Huffman-coding is tempting, I've decided that golf challenges use obscure words disproportionately to their use in English prose, so my current plan is to have two-byte representations for common words and three-byte representations for rarer ones, plus syntax for handling words that aren't in the dictionary

i always assumed thats what existing dict compression did

Jelly uses that sort of dictionary compression but the word list is much smaller

my main problem with this word list is that it's 100MB long so actually distributing it may be hard

ah, well thats the one i was thinking of so i probably just dont look at a lot of dictionaries

maybe something something "a lot of words have overlapping/shared substrings" something something "hybrid compression" something something

@ais523 Nowadays that's not much, right? Like, I've gotten multi-gigabyte files off GH IIRC, and I know for sure I've gotten tens of gigabyte files from Google Drive and similar services

@RadvylfPrograms it's not that ridiculous, at least

but it may cause problems for people with slow connections in, e.g., a web interpreter

oh i misunderstood the issue :P

or who have metered bandwidth

I think the correct solution is probably to have the interpreter download just the relevant parts of the dictionary, and then cache them

Oh, true. I think for Ash, which was going to have an even larger word list, I was going to load just the common words, then asynchronously load a chunk of the longer word list when needed

…which in turn probably means that I need to alphabetise it so that you can determine which part of the dictionary to download either from the compressed or the decompressed representation

(i.e. the same index works for both)

but the two-byte words and three-byte words need to be alphabetised separately

Might it be possible to generate the word-list on the fly? Kolgomorov complexity golf, large-scale? Probably more effort to it than is humanly possible, but it'd be an interesting challenge. (Perhaps a dumb idea, if so just ignore me.)

You can get 70%-80% compression ratios with normal text, but there'd probably be a bit less of a pattern in a word dictionary

So you'd still likely be looking at 30-50 MB of data

sorting by codepoint + zstd on default settings got it down to 30331639 bytes

(although that's without a 2-byte/3-byte split as I haven't yet decided how many words go in each category)

I'm also planning to add multiple variants of punctuation, with different whitespace around them, but haven't decided on the details

How does bellard.org/nncp perform? mattmahoney.net/dc/text.html indicates that it outstrips ztd version 0.6.0 by a considerable margin. (Again, please ignore me if this is a dumb question.)

there are definitely better compressors than zstd around, but zstd is the most easily available out of compressors that get good compression in reasonable time on large amounts of data

also I don't have an NVidia graphics card, so I'm not convinced that nncp would be runnable

I use LZO for high-speed compression.

Got it. I haven't much experience with compression. Sorry for the inane suggestions.

also nncp compresses at the rate of single-digit kilobytes per second, so I'd have to leave it running for an hour or so to find out (and more to the point, the end user would have to leave the decompressor running for an hour or so to get at the data)

For really good compression (and if you like waiting for days), try lrzip!

@23TuringMachine "inane" is a pretty strong word, it's an interesting suggestion, just not really practical

@Neil Oh that's interesting

Maybe I can get by with just a Bachelor's then

(Personally, I rarely need anything other than LZMA for strong compression and LZO for extremely fast compression)

@RadvylfPrograms Don't ever feel like it's a bad idea to bring up a potential idea/solution to an issue here, or a need to apologize if an idea doesn't end up being practical. There's people here with hugely varying amounts of experience, and across a wide variety of topics, so even an utterly impractical proposal can still teach people new things, which is after all why I (and probably a lot of others) do code golf in the first place.

zstd -19 gets it down to 20699320 bytes, not so bad

@user why do I need to go to uni just to get a piece of paper that states my relationship status?

And sometimes "dumb ideas" can give you a different perspective on something, or lead to another idea, or actually not be all that dumb in the first place

the only bad idea is no idea unless its a really bad idea

@user IMO the main reason to get a masters' degree is if you want to spend more time at university for some reason

I did a four-year degree (master's and bachelor's mixed together), and I think it was the right decision for me, but that was for reasons which wouldn't apply to most people – there's nothing wrong with a three-year degree if you just want to go right out of university into work

Ah okay

@ais523 Here in the US, a bachelor's is 4 years and there are combined masters+bachelors programs that are 5 years

Which makes a master's even more unappealing

huh, I wonder where the extra year came from? do people enter university a year earlier? (we enter at 18 here)

It's 18 here too

Perhaps because of general education requirements?

I'm not sure where you are but it seems like most colleges here want you to take extra classes in English, Math, Science to round out your education

most universities here offer a year 0 that's only required if your education so far is substandard, it gets you up to the level that's required to understand the rest of the course if your school didn't

Welp, I guess everyone's education here is that substandard :P

Not exactly wrong lol

I'm in the UK

there's a regional language difference, in that "college" here normally refers to the education you get from 16 to 18, rather than being a sort of university (although some universities also adopt the name)

This random person on Linkedin claims it's because of tradition and because of useless classes, and I think they're right

it isn't that different from school – in most respects it's identical, but you're treated more like adults than like children

I will say, though, that even at year 3 of a 4-year course they were starting to run out of useful things to teach

Do people have jobs while they go to university in the UK?

the problem being that you can fill 1½ years or so with information that'll be of interest to anyone in the topic, for another half a year or so (maybe a bit more) you can break the students up into groups based on their interests and find subjects that will be worthwhile for the whole group

after that, everyone's interests diverge so much

The course load here is pretty light (only 5 classes a semester), which may contribute to the extra year

having a job while you're at university is common but not required, and it'd normally only be part-time

standard at my university was 2 semesters a year (+ exams), 6 classes per semester, except that some of the classes were doubles which counted as two (i.e. twice as much work, twice as much teaching time, and counted as 2 towards the six)

@user only 5?Tf kinda classes you taking where 5 of them is seen as "light"?

one of the classes I taught was a triple

Whoa

@lyxal Is 5 high or low?

that was teaching the first years how to program in Java

@lyxal These are freshman classes. One of them is literally just getting to know people that you'll eventually do research with

@RadvylfPrograms over here, it's the maximum number of classes you can do a semester

And even then, you need to get special permission to do 5 a semester

5 per semester seems really low to me, although probably high school bias

3-4 is the usual load here

@ais523 I didn't think learning Java was that hard. Did you compress an insane amount of material into a single semester?

also, we had a rule that if you failed more than 2 classes you had to repeat the year (or to drop out if it happened twice), so if you failed to learn Java, you couldn't do the rest of the course

@user not really? but some of these people had never programmed before

@RadvylfPrograms Yeah, but you do need the time to work a job

@ais523 Oh ok

@user Same with high school

so we were taking people from "potentially zero" to "enough Java to not fail horribly at an entry-level job"

At the university I'm going to, the intro to OOP/Java/Programming classes are all 3 credits (the normal amount)

@RadvylfPrograms You don't need the money that bad in high school

plus programming concepts like loops, variables, and so on, which can be difficult if you haven't seen them before

Intro to OOP/Java was only a single course and only a single unit here

it's interesting to look at languages where these basic things work in ways different from the consensus that people have agreed upon nowadays

@user You need it about as much during high school as college, not like you could afford college just by working a part time job, so every bit before then helps

It's not as urgent then

I guess, yeah

I mean sure, maybe you should be allowed to leave school for a day every week to work a job senior year of high school, but before that most people live with their parents and don't have to pay for much themselves

What no there's plenty of time already, that's my point

@ais523 gee that's way slower and more focused than what happened for me - first year was intro to OOP + intro to professional engineering + intro to computing fundamentals + intro to data structures (and low level memory management) + intro to Web dev split over 2 semesters

Two weekend days and time after school on weekdays could easily add up to 30 hours I'd imagine

in Algol 60 (assuming modern I/O), you can write code like procedure f(a, b) integer a, b; begin a := a+1; print(b); end integer x; x := 1; f(x, x+2)

and you will get an output of 4

It's passed as a reference to the original x?

Mmm juicy pass by name (I think that's what it's called)

yes, but not just htat

the x+2 is also a reference

Wow, that is odd.

To what? A temporary variable made by the compiler?

Why have sugar like that? Wat is happening under the hood?

@lyxal yes, "pass by name" is the CS term for this

@23TuringMachine this isn't sugar – this was the natural interpretation of how programming languages should work, at the time Algol 60 was invented

@23TuringMachine iirc it's literal text substitution

Times have changed, a lot.

the idea of "a function call takes a snapshot of the arguments, as they are at the moment of the call" is a bit unnatural if you think about it

nowadays, call-by-name (in the few cases where it's desired/needed) normally gets compiled into closures

@ais523 Pass by value? I thought pass by reference was more common today

That's just pass-by-value with pointers

@user nowadays pass by value is almost universal – however, in many cases you are passing references by value, so the recipient function still has a reference

C/C++ are pass-by-value, no? Primarily, at least.

@RadvylfPrograms Right, but I'd still consider a pointer to an object to not be the same thing as a "snapshot" of it

e.g. in Java, if you pass an object as an argument to a method, the method can change the object itself (because you passed a reference to it), but it can't change the variable in the calling function that holds the object reference

@user Sure, from a human perspective, but to the computer it's exactly the same operation

You just copy a 64-bit integer that might be a float, or an integer, or a pointer to an object of any type

@ais523 Out of interest (feel free not to answer), but was this in the UK?

@cairdcoinheringaahing yes

@ais523 Ah nice, I'm also on an integrated masters program

@RadvylfPrograms i mean, we’re talking about the human perspective here, right?

Idk lol

Like how it’s unnatural to have a snapshot of the arguments

fwiw I have a suspicion that the current argument passing system is suboptimal and that there must be a better alternative out there

@ais523 Are you still at UoB, or have you moved research facilities since your PhD?

(Again, feel free not to answer if you don't want to)

Snapshot makes me think of deep copies and having that happen every time a function is called would scare me

@cairdcoinheringaahing right now I'm not anywhere – my job ran out of funding, I had huge burnout and was pretty much unable to work, then Covid hit

and I'm still really burned out, and reluctant to get a job because I fear I wouldn't be able to put in any real amount of effort and would just get fired

I've hardly been able to program for months

that 100-byte Incident program (plus the script for optimizing it) is the largest thing I've written in ages

@ais523 it seems so weird to me that that couldve actually been considered the natural way for arguments to work, but I guess thats from a modern perspective of being trained on langs with pass by value/pass by reference

Best wishes for recovery in your mental health!

^

@ais523 Ah, that's a shame. I know quite a few people who either burnt out or ran out of funding during covid, so haven't finished their PhDs. Hope you can recover from it

I hope so too; I'm trying (although I keep getting distracted whenever I try to program, and doing something else instead)

@ais523 That feeling sucks, yeah

One of my dad's PhD candidates (students/graduates?) got her PhD a few weeks ago after 6 years (instead of 4) due to covid related delays

@user fwiw, I think that the operation of "act as though we're making a deep copy" is a very useful one from the point of view of programmer ergonomics – the only reason it isn't used more is that we don't have a good way to make it efficient yet

@ais523 if everything is immutable i guess you can pretty much have that

IIRC, her viva was due in summer 2020, and y'all can guess what happened after that

But then there’s other problems

I think many/most languages make the mistake of trying to deal with mutable and immutable things in the same way

Rust ftw

@ais523 Or, you can just not have mutable variables. I always find them to be too much trouble when implementing a lang

e.g. garbage collection works really well for immutable things, but horribly for mutable things – when dealing with mutable objects you need to really keep track of the objects' identities for the program to have any meaning, which means that manual memory management should be trivial

Seriously, clarify this for me: what are the benefits for mutable types?

@cairdcoinheringaahing you can go down the path of Haskell, which is a perfectly reasonable language, but it hasn't really caught on

@cairdcoinheringaahing Performance

but, to take an example that I've been thinking about, suppose you have a computer game where the character moves around an environment with multiple rooms, picking up objects in some of them and dropping them in others

@RadvylfPrograms I said "seriously" :P

I do love how Haskell’s gc can just clear out the space the younger generation occupied

(like the Tower of Hanoi but more complicated)

@RadvylfPrograms And convenience in some cases

@ais523 id play the hell out of this, is it out yet

the most natural way to represent this to represent the game objects as programming objects

and store information on them about, e.g., where they are

in a language like Haskell, you can write this, but the normal way to do so would be to have a big data structure that holds all the information about the current game state, which you make slightly modified copies of

and then you discover that you've basically implemented a VM for a language with mutable values, and it's really low-level so you don't have any of the safety valves that a high-level language would have

But why not simply create a copy of a given object whenever modified, with updated stats, and delete the old object?

@cairdcoinheringaahing because either a) you have no references to the old object, in which case everything is one big data structure, or b) you do have references to the old object, in which case you can't update them because they're immutable

@cairdcoinheringaahing Takes more time to think through, more code to write, and thus two more ways you can introduce bugs and waste bytes/time

Ah, but consider this: it's a lot simpler to implement immutable types :P

I think that the ideal programming language would treat mutable and immutable types very differently from each other

But you only implement a language once

Whereas your users have to deal with immutability everyday

we still haven't figured out the best way to handle mutable types, but immutable types are mostly a solved problem at this point (the only difficult part is whether to copy or share references, and when to deallocate if you're sharing references, and we already have good solutions to making that transparent)

It’s worth it for the language designer to do the heavy lifting so users don’t have to

@user Yeah, but if anything goes wrong for them, it's user error, not my fault

fwiw, I think the "correct" memory management model for immutable things is to have the compiler do some static analysis to work out if it can use a predictable allocate/free schedule such as stack memory, and just throw everything into a GC if it can't

but that this is completely incorrect for mutable things, which should be allocated and deallocated explicitly, and possibly even searchable

@ais523 like manual memory management only for the mutable types and no references from immutable types to mutable ones?

@ais523 doesn’t this already happen?

@user I don't think so? at least not in mainstream languages, there's probably a research language or someone's PhD project that can do it

@ais523 noob here, whats the "GC" stand for there?

Garbage collection?

ah, sounds about right, ty

@user a reference to a mutable object is itself immutable, although in this case it wouldn't be allowed to hold the object alive, so it would by nature be a weak reference

It's a feature in memory-managed languages.

@thejonymyster garbage collector, a program (or part of a runtime, virtual machine, etc.) that automatically deallocates objects that nothing has references to

very nice :-) every day i learn some more, thanks you both

@ais523 I’d heard java and go do escape analysis and put stuff on the stack if it won’t escape

@user that's possible, I'm several years behind the times on this

Wikipedia is citing papers from 2014 about escape analysis in Java – if it was being academically studied then, then it might be starting to reach practical compilers within the last couple of years

@ais523 You might be able to shrink it further by unifying with /dict/words or something

on the subject of weak references: something else I realised is that in many cases, you want object deletion to be recursive, i.e. if a has a reference to b, and you explicitly destroy b, then a should be destroyed too

in almost every case where this is undesirable, a is a collection, and the correct behaviour is to remove b from the collection rather than destroying a

@emanresuA my /usr/share/dict/words is around a megabyte, so that wouldn't make much difference

@ais523 couldn’t this possibly result in dangling pointers ?

@user if you fully recurse, no (although you may end up deallocating everything if you have more references than you should)

I'm not sure what happens when you mix it with destructors

although based on experience programming NetHack (which does try to do all this manually), my sense is that it's very hard to get it right when doing it by hand, so automating this would likely lead to fewer bugs

@ais523 Not sure if I'm understanding what you're saying right, but is the GC even necessary at all? From what I understand Rust gets away with having no GC, but the programmer doesn't need to bother with allocating/deallocating memory themselves either

@ais523 why not make a hold the value of b rather than a reference them? You won’t need to recursively delete stuff then

@RadvylfPrograms when you need lots of references to something immutable in Rust, the normal method is to use Rc, which is not technically a garbage collector but has very similar properties in practice

Ah okay

@user because recursively deleting things is normally the behaviour you actually want

Oh ok

the main differences between Rc and a garbage collector are that Rc will predictably deallocate objects at the exact moment they're no longer referenced (thus a destructor, if any, will run predictably); Rc doesn't need a separate runtime because whichever thread does the last unreferencing will be interrupted to run the deallocaiton/destruction; and Rc is generally a little slower

@ais523 if that stands for reference counting, i think many consider it garbage collection, right?

Rc works via reference counting, and is probably named after it (probably "reference-counted"); I guess it's a form of garbage collection, but an inefficient one

And no cycles :(

@user e.g. if you destroy an object in a computer game, you want everything that referenced it to know that it's now gone, rather than to have a copy of it

garbage collection on mutable things is basically a bug

because it keeps things around as long as there are references to them, so when destroying them, you have to iterate over everything that could have a reference and null it out

I made the same observation in three separate programs, which were in unrelated genres (a game, a build system, and a language runtime), so this is quite possibly a universal rule of mutable objects

Since emanresu's been talking about their (mis)adventures while writing a Jelly decompressor, considering writing one in Rust as practice

Seems like it'd be a good opportunity to get used to its sensible UTF-8 handling, something I as a filthy JavaScripter don't have much experience with

Rust is very low-level in terms of its UTF-8 handling

to the extent that string slices have their indexes into the string measured in bytes

"éé"[1..3] will give you a runtime crash because you tried to slice a character in half

this is a sensible design decision given Rust's goals, but it still takes a lot of getting used to

the standard library is good at guiding you to use strings correctly, though, e.g. you can't just iterate over a string, you have to explicitly say whether you're iterating over bytes or over "chars" (actually codepoints)

it isn't as good as Swift because it doesn't really focus on normalization or combining characters, but it's better than JS

I should probably try to learn rust at some point, it seems like a good lang to learn

it is definitely worth learning – despite having theoretically known C++ for years, I didn't really understand it until after learning Rust

my main issue with Rust is that it gives you too much of an incentive to try to be clever

my understanding of rust is basically that its C, if you gave it a compiler as picky as haskell's compiler

there's code that I'd feel is too risky to write in C, because I wouldn't be confident it was secure – Rust gives you the chance (and the obligation) to prove it's secure, which can be too much of a temptation sometimes

so you end up spending too much time writing the clever code and getting the compiler to agree with you that you got it right, when in fact all this is premature optimization

Rust is a really dangerous language to expose perfectionists to ;-)

@des54321 It's like a mix of C and C++, but with the borrow checker breathing down your neck.

tbf ive never even learned enough of any lang in the C family to be able to tell the difference between them

Rust's standard library is much better than C's or C++'s, that does make a big difference when using the language

@ais523 I don't think it's better. It's too fragmented and suffers serious supply-chain issues.

Just like npm.

@forest I think you're kind of underselling it

I think I've used C# some for some reason (is that what Unity uses? I took a whole high school class using Unity), but I wouldnt say I ever learned enough to know to program well in it

@forest I don't mean crates.io – that isn't a standard library, it's a package manager; I mean the standard library itself

Oh.

Well, I guess I'm just too used to C's that I think everything is inferior.

C doesn't have a package manager does it?

Nope.

How can you get any more fragmented than that

@RadvylfPrograms not an official one, which is not surprising given when it was created, and none of the unofficial ones really caught on

Because there are a few popular runtimes.

And everything is kept within those.

You don't use a string function from Bob and a syscall wrapper from Joe.

C is probably the language I've used most, and I haven't really missed the lack of a package manager

I feel package managers are the purview of an operating system, not a programming language.

there are a few very common dependencies, like zlib, but apart from that you can mostly get away with no dependencies at all except when doing GUI programming

^

Whereas with Rust, you have to trust a lot of people to maintain a lot of vital libraries.

Really? Like what?

I think Rust is making a mistake by delegating too much to repository crates long-term (although it makes sense as a short-term solution to test out additions to the languages)

Exactly.

There was a recent LWN article outlining the problem.

however, there's still an impressive amount of content in the standard library itself

I'm used to JS where the standard library is like, base64 and an HTTP client, so anything seems good in comparison lol

for programming esolangs in Rust, the only crate dependencies I've needed are a crate for bignums, and a crate for parser generation (and the latter is a dependency I don't really want)

So, in the language I'm working on, the Hello, World! program is literally `Hello, World!`. `is the compressed string delimiter and telss the compiler to store it in a dictionary-compressed format in the resulting binary.

And what's counted in bytes is the binary.

this makes sense for a compiled language

Yeah, and it leads to quite simple syntax

I came to the same conclusion about the best syntax for a golfing language (i.e. just let people write what's most convenient, and have a compiler that translates it to the source that gets measured)

also giving bulitin names lots of synonyms to save you from having to remember the correct one

Yeah :P

and if the builtin name is ambiguous, throw an error with the possibilities

e.g. index is a bad builtin name, index_into and index_with are better

Agreed :P

because otherwise it's easy to get the argument order wrong, indexing doesn't have a natural argument otrder

Some langs allow both

As langs become more complex to save on redundancy, compilers like that are going to become the only option.

Because it simply won't be possible for humans to read/write the compressed syntax.

That will take away some of the fun I think

I'd say it's a natural extension of current golflangs

There's sort of two sides to the advancement of golfing languages: more of a DLoscian language golf, where you're trying to get the golfiest language at the cost of how fun it is to use (e.g., source compression), and languages like Jelly which are close enough to optimal to not do horrendously bad, but are fun to use

@RadvylfPrograms I would not say that's DLosc's style

I think it's possible for a language to be golfy enough that source compression would hardly help

DLoscian as in DLosc's idea

Consider Pip, an ASCII-only lang that is constantly being updated in the SBCS age.

especially because programs in golfing languages are normally small, and small data is harder to compress

There's a pretty close connection between how many operations you do and how many bytes you use in third gen golfing languages, so finding a way to turn a dyad and two monads into two dyads is always a byte saved. Whereas with compiled/transpiled golfing languages, it's sorta going to be more of throwing things at the wall and seeing what's shorter I think

@ais523 Not necessarily talking about compression in the normal sense of the word

I've been planning a compiled golfing language, but in my design, the byte count is always very clear except when it comes to literals

E.g., catstruct's "compression" where the structure of the program and the actual operations at each node in the AST are encoded separately to save bytes in certain cases

like, if you're writing something that needs an extra byte to represent, you actually have to write a character into the source to demonstrate that you know you're using extra bytes

But anyway, it's late here so I'll see y'all tomorrow o/

(e.g. two-byte builtins all have names starting iwth @)

fwiw, I think structure + operation coding is the correct way to go, and I'm doing some of that myself, but you know the ratio of space spent on structure versus operation and can split up your bytes in that way

in my case, nilads and dyads need 1 bit of structure, monads need 0 bits of structure, and generators (the equivalent of Jelly's quicks) can be encoded with either 0 or 4 bits of structure depending on whether you need the extra bits for something or not

so, I reserve space for two encodings of each nilad/dyad, but only one encoding for each monad

and store the structure by varying the encoding

(the nilads/dyads share with each other – there are few contexts in which both are legal, and you can use the shape disambiguation bit to tell them apart in the few positions where either is possible)