@Zgarb Done.

@Dennis well if you and I collaborate I can handle the DB backend, while you handle frontend and such.

DB backend is easy, at its core is SQL, most of the admin part is just flat-files and creation of users.

I'll even one-up and provide a pgsqladmin - a nice gui to use :P

... if you don't mind HTTPS and a client cert to protect the pgadmin frontend lol

gotta keep things secure SOMEHOW right? :P

@Dennis can you please pull RProgN2?

Uh, Funky*

(I'm too used to RProgN2, for some reason)

@ThomasWard SQL somehow doesn't imply easy in my mind. :P

I guess I'll find out. ;)

@ATaco Done.

@Dennis well this is why I set up a GUI

if you want I can even spoon-feed you SQL queries to add to the DB. Which, by the way, I already set up... shifty eyes

but yeah this way we can offload the shortlinks infrastructure to something that isn't the TIO servers you're already using, and I can offload the data throughout multiple PostgreSQL databases in a cluster setup. Including a replication target with 1TB of data storage here at my location physically. (All the DB would need to store is a unique record identifier internally, a unique shortlink record, and the URL to redirect to... the frontend can be done however you want.)

(is 1000 characters of allocated maxlength too much for the TIO URL?)

@ThomasWard where do you get all these servers from

the Void. Or I pull the parts out of /dev/random and build them.

the one with the 1TB of data storage backend actually runs on site here at my apartment xD

the rest are VPSes or virtual dedicated servers (VPSes with dedicated CPU), or actual dedis, or colocated equipment of mine at a datacenter.

:)

and trust me I have a lot of resources available across all the VPSes, VDSes, Dedis, and colocated equipment.

It's insane :P

@Pavel I actually have a cluster of PostgreSQL servers serving as the backend for a few different things I have. Including an in-development Python listserv system.

@Dennis ^

I’ve never heard it pronounced “tea-o”, but I’m glad to see it represent.

@ThomasWard That's actually not a lot. Part of why I want to offer permalinks is that more complicated requests require too much space to store the state. For example, the link is this answer is 34,814 characters long, so it will not work Chrome or Edge/IE.

Unfortunately, a link-shortening service doesn't help with that. The client would have to directly read the state from the server, without going through URLs.

@Adám Cool! Thank you for the link.

@Dennis Thank you for TIO!

:)

@Dennis well, here's what my thought was:

store a shorturl ID, and have that be processed by a webserver with a 301 redirect to the fully-expanded link. But if it won't work with Chrome or Edge, then there's going to be bigger problems then.

if we're going to be handling data that long then we need to develop a tighter integration with TIO and a way to store the actual code to pass through to a page that can then be handed out. Which is doable, but a pain.

I know that there's multiple ways to do that via API, but... not sure if we can return an actual page with that information without the encoded link, as I haven't dug into the TIO API yet.

Unless you have a way to shorten the actual link (encoding + compression of TIO encoded data perhaps?)

@Dennis Do we know what the longest URL length Chrome and Edge/IE can support is?

because using zlib compression (and ending up with a painful Unicode string) on a 34814 byte long string yielded me 29288 characters.

@ThomasWard that challenge is basically designed to produce longer and longer permalinks over time.

@Pavel we're going to be limited ultimately by what browsers support.

unless we do a redesign of TIO to have code stored separately with unique identifiers used on TIO's generated URLs to then query a backend and restore the code.

The core issue there is we have to work around the browser limitation

It's clear that if we're having issues with 34814 character long links, that we need to rethink how we're going to design things.

if we want to have longer and longer permalinks

but that's a redesign of the entire TIO system.

The point is, mapping long permalinks to short ones isn't the solution

right... hmm...

That doesn't fix the issue

then your only option is a redesign of the TIO infrastructure to store data somewhere. With a corresponding shorter link ID on the frontend, even if it's a short string of ints.

but that also could introduce breakage from a reverse-compatibility perspective with older TIO links after that redesign.

(Basically we've just increased complexity 10-fold)

I can still provide a DB backend for you guys for this, though, if the redesign does happen.

From a code perspective, if we compress the data being stored in the DB first, and then decompress when we load the data into a page, we can cut back on the space issue slightly. Not the best approach, but it works half-decently.

@ThomasWard Chrome goes up to 32,768 characters, Edge to 2,048. Non-ASCII characters get escaped by browsers; my attempt to use them to shorten the permalinks failed miserably.

The state inside the URL is already DEFLATEd btw. Just the base64 encoding would have to be undone.

@Dennis I think the b64 encoding is what's screwing you

a compressed string of 50000 bytes ended up being 26k or something in my end

when I b64'd it, it explodified into about 36k in length

Well, for URLs, there's no other option. I wouldn't base64-encode it to store it in a database.

@ThomasWard I actually started working on that (using Google Cloud Storage), but the potentially fast-growing storage requirements made me re-think it.

I don't have a solution for that, but barring a TIO system rewrite to store data in a DB backend and then have tio.run/52b3F do a query to the DB backend to get the code, language, etc. there's no real other way around it.

@Dennis my question then is exactly how fast do we expect growth?

@ThomasWard The frontend would have to be able to handle both.

Storing the code in the DB, we can do it in straight compresson.

and then not care.

@ThomasWard That's the issue: absolutely no idea.

@Dennis have you tried better logging

^ that

I'd suggest we implement a logging function to determine average amount of data over a week that we'd have expansion

That's a good idea.

and my question is whether we'd want to set a long expiry (read: 3 years) or whether we just want permlinks

if we are OK with a very long expiry then we can get by with 3 years, and then offload anything older than 3 years to an 'archived' state, with its own datastorage.

if we must have it always perm links then we get into the inability to gauge expansion

I'd very much like them to be permanent. Nothing more frustrating than 30 permalinks on a PPCG page, and none of them works.

well, that I agree with. I have a certain hatred for that though, for obvious reasons

unless we implement the option to choose whether it's a permlink or not and just set a very high default expiry

like 3 years

or whatever we choose the upper bound for 'defaults' to be

@Dennis also people have requested logging for how often specific languages are run for a while

@Dennis in a non-base64-encoded state, what do we end up with for that TIO link you suggested? (Also that URL length appeared to work in Chrome here?)

i mean I could b64decode it in Python but... I'm lazy :)

@ThomasWard At least as far as TIO links on PPCG go, yes, absolutely. Let me start by adding a bit of logging, so we can estimate the storage requirements.

@ThomasWard Roughly 75%.

So 26 kB.

@ThomasWard You're right, it still works. It's just copying it from the address bar that's broken.

@Pavel Yeah, the new permalink format broke that.

I'm working on a new backend anyway. I'll add decent logging to the list of features.

@Dennis Since most TIO links are relatively small, would having server side permalinks only for links over a certain size and/or specifically requested to be server side make sense?

@Potato44 Yes, my intention is to offer both. Personally, I'd probably use server-side pretty much all the time though, as it makes it easier to edit answers.

If the link can be max 2000 characters, one can compress to base 64 the code solution store it in the link, written in the answer. When someone push the link the part of the link has the code could be decoded from base 64 to text in Tio site. But it would be ok with solution <2000 characters

@Dennis I asked this a while ago, but is there any plan to allow an option to toggle wrapping?

@DJMcMayhem Yes. I started working on that, but ran into an obstacle (don't longer remember what it was). Thanks for reminding me.

Ah, OK

@Dennis could you pull Deorst please?

@RosLuP To be URL safe you need it b64 encoded to use non-Unicode. You're right we could compress the b64 encoded item but then we have a headache with unicode multibyte adding to the problem.

we don't have that issue with b64 encoded

@Dennis how many people, then, actually copy from the address bar and not the link generation function that TIO has though?

if the issue is not as widespread as we think then, we can probably work around with just using the link generator for each link. That would permit us to straight short-link things, then. At least until the new backends are done.

@ThomasWard I do

@Adám That's exactly the Spanish pronunciation

Oh, I see now the comment is by Dennis. That explains it :-)