Stack Apps

10:15 AM

@rene I suppose so - still think it is broken-by-design, though :( Really, I know popular CRM APIs [that would be subject to DoS often] that have more generous limits - not to mention that they consistently return valid error objects saying that the throttle violation happened instead of just... dropping requests by not sending CORS headers at an unspecified point

10:40 AM

I've suggested an edit on Adams' post converting the code into a snippet and the number of pages it is able to fetch doesn't appear to be the same every time

sometimes it is even able to fetch all 300 pages

11:41 AM

the exact number of sequential requests before the drop occures also seems to be dependent on the exact endpoint one is hitting as some are clearly weighted more than others (with /search pretty consistently capping out at around 176-180 requests)

5:28 PM

IMO, when it happens is quite variable.

While I agree that the response from the SE API should be consistent, that's not how it was designed/deployed. They do document that what the response will be is undefined, which ... is sufficient, even if it's not what we'd like to see. I really doubt that we'll see them make changes here in any reasonable timeframe. Having it as an issue on MSE is good, but being indignant about it not being "right" doesn't help us.

From a "dealing with it" point of view, we just have to accept that anything other than one of the "normal" responses is, effectively, an indicator that we should slow down on requests.

@rene @double-beep Yes, at least the behavior which was the case when I investigated it for MS, SD, and FIRE was that revision_number did not exist or was invalid/null (e.g. in JavaScript Boolean(response.revision_number) was sufficient to determine if the entry was an actual revision). Code in all of the above projects currently relies on that behavior. Has it changed?

apparently

example | revisions of that post

@Makyen I don't know, really - from my POV, the more we complain about the state of things (hardly acceptable if you ask me) the higher the chances that someone notices - for all I know, the folks responsible for the API (if there are any left now - I think the Public Platform finally got the resources necessary to make changes in a reasonable timeframe, but know nothing of who is responsible for maintaining it nowadays) are sure that everything is working just right because we keep [1/2]

as you can see, the revision with comment "add link to video" is revision 6 according to the API, but 4 according to the site

5:46 PM

[2/2] working around these things. I know that the chances are slim at best, but while complaining about the state of things may not be very helpful given the treatment the API gets, it does not hurt (at least when done in a reasonable manner - and I only went on the harsher side in comments to compensate for the "well, this is how it works, no need to get upset about it" vibe your initial comment gave off [I understand this wasn't your intention but this is what will be a takeaway for many])

for now, maybe the most sensible thing would be creating and maintaining a sort of a "canonical" post on the real state of things with the API throttling here by bringing together what has been investigated by individual parties - I will likely look into it later (unless I missed something that can act as one)

6:10 PM

@double-beep Well, that's a pain in the rear, and will have resulted in a few months of potentially erroneous data in MS and SD chat messages. It looks like we now need to use something like:

$.get('https://api.stackexchange.com/2.3/posts/59189/revisions?key=U4DMV*8nvpm3EOpvf69Rxw((&site=politics&pagesize=100&filter=!3tlXYBI*OhFAozJLO')
    .done((response) => {
        const realRevisions = response.items
            .filter(({title, body, tags}) => title || body || tags)
            .sort((a,b) => a.revision_number - b.revision_number);
        console.log('revision count:', realRevisions.length, ':: realRevisions:', realRevisions);
    });

but that assumes that you have all the revision API items

@double-beep It means you need all of those properties in order to determine if an entry is a real revision, and need all of the items in order to determine a revision count.

6:41 PM

@OlegValter this might be relevant: meta.stackexchange.com/questions/249893/…

6:52 PM

@rene It's related, but it sounds like it's about rate limiting requests to regular SE sites, rather than the SE API. The two things are on separate rate limiters. Chat is also on a separate rate limiter (couple of different ones, IIRC). I would suggest using a different question if we're wanting to post something specifically about the SE API.

@Makyen it looks like having an thruty "content_license" can distinguish between edits and other reasons to have a revision. Only tag edits don't have a content_license so still not 100% :/

@Makyen yeah, there is definitely different configs for different routes.

@rene Yeah, I initially thought that might be able to be used, and also noted that it's not present for tag-only edits. Using it instead of both the title and body properties would save some bandwidth, if you don't want to fetch the body and title. Before actually using it, I'd want to double-check that it's present on some really old questions.

@rene looks like a good place to start, thank you - I concur with Makyen here, though, seems like it is about scraping rate limits rather than API rate-limiting so only tangentially relevant

@Makyen license was backfilled IIRC but better check to be sure.

wait... did main just go offline for maintenance again?

6:59 PM

@rene Yeah, it should work, but ...

...this is SE :)

@OlegValter maybe they are fixing the API while we talk ...

too fast for that :) already back up - seems to be just another hiccup (too many of them lately to not be worried, though)

@OlegValter I guess my primary issue was that your question was conflating two different issues, both of which are related to CORS. The questions you link from November 2019 (two of the questions in your table) are a "something's changed; the SE API is completely broken now" issue, which needed to be fixed for most things to work. The issue which you're re-reporting is an ongoing issue related to how SE has chosen to deploy the SE API and implement rate limiting.

I agree that it would be much better for SE to change how they handle this overall and adjust their deployment such that the SE API always returns a consistent response (e.g. either something valid, a response with backoff, or an actual indication of "you're blocked; you must wait X time"). I just think that it would be better to keep the issues separate and present the ongoing issue more accurately.

At a minimum it would be very helpful for the documentation to be updated to make it clear what might happen and that it might happen without any prior indicator that you should be backing off.

the posthistory does have contentlicense for all create / edit / rollback events: data.stackexchange.com/stackoverflow/query/1482894 so it is strange why the api doesn't have them.

that specific post is complete in SEDE: data.stackexchange.com/politics/query/1482896

7:18 PM

@Makyen oh, I included all 4 to drive the point of "similar issues happened before and some have even been fixed, so why not this one?" - I understand that this is an ongoing issue, although my expectation was that self-rate limiting to a measly 10 requests per second with proper throttling should cover the common usage, which turned out not be the case. It is that much more annoying that there is nothing to refer to officially as neither the docs nor the "complete rate-limiting guide" [1/2]

[2/2] have detailed information on the matter. I suspect that even if not actually fixing this, SE definitely could spare a bit of time to fixup the documentation, and in the meantime, we could provide community-sourced information (maybe by updating the "complete rate-limiting guide", come to think of it).

@double-beep you post a bug for that?

Q: revision_number is returned in every revision item, even if it's not an edit in the API

yes, I'm writing a post on MSE

0

As of recently, the Stack Exchange API v2.2 (and v2.3) returns revision_number in every revision item, even if it's not an actual edit, effectively invalidating the "may be absent" for comment in the official API docs. Consider this API call (revisions page of that post): { "items": [ {

@OlegValter Right, but "similar issues happened before and some have even been fixed, so why not this one?" implies that the issues are similar, when they really are not the same, other than they happen to involve CORS and rate limiting. The November 2019 issue was a "everything's broken" and things need to be fixed. What you're reporting is an issue that can be paraphrased as "I don't like the way this is done. It's not clean, more work, and unexpected from how I read the documentation".

Conflating the issues makes it appear that you're trying to say the issue you're reporting is "everything's broken" and things must be fixed, which just isn't the case for this issue.

@rene on that note, can the comment in these cases be relied on for filtering out those "bogus" revisions? as it seems like <b></b> is a pretty consistent property (not saying the issue is invalid, just what might be done in the meantime)

@OlegValter yes <b></b> is the signal here.

7:33 PM

@OlegValter that's (yet) another bug

I think I am going to switch off audio notifications on Stack Apps :) mentions are going to give me a heart attack here

Q: revision_number is returned in every revision item, even if it's not an edit

@OlegValter There's always SE Chat custom notification sound, which can be used to change the sound. Note: If interested in it, I suggest using the fork that another user created on GitHub (to which I've contributed).

Feeds

3

As of recently, the Stack Exchange API v2.2 (and v2.3) returns revision_number in every revision item, even if it's not an actual edit, effectively invalidating the "may be absent" for comment in the official API docs. Consider this API call (revisions page of that post): { "items": [ {

bug revisions-list api revision-number

@double-beep yay, great... missed that one, thank you

@Makyen I understand what you are saying but... unsure what would you think would be a better style of reporting this - it may be a good idea to rephrase how the list with issues is introduced or maybe add a footnote, I'll work on it a bit later. But as for the general sentiment of the report, here is where I disagree - I consider anything that differs from an expectation of a reasonable consumer of a service to be a bug and not a suggestion to change how things are or a feature request.

@Makyen yeah, I know about it, but thanks for the reminder anyway :) Unfortunately, right now I am on a pristine-clean browser in terms of userscripts - for some reason this "pinching a string" sound just makes me jump...

tempting

7:47 PM

am I that annoying that you want my early death? :)

:D

@rene I guess it can be relied on then - bugs seem to be the most constant thing around...

true that