Somehow I suspect manufacturing custom floor tiles wouldn't be cheap...

The big tile is perfectly level, but at least I "see" it's actually slightly tilted.

So if you have say 1000 web requests you need to make (say on average they take a half second to respond), and after each response you need to update a dynamic list, how would you do this for the best combination of speed and stability? It currently seems like launching a bunch of URLFetchAsynchronous calls is the fastest, but often as many as 1/3 of them get lost, and using ParallelSubmit (or of course even just looping through them on the main thread) is the most stable but a good deal slower.

I'm still experimenting though.

@MichaelHale Does it "get lost" without any message whatsoever? Otherwise maybe using the callback to initiate a new download might still be the fastest approach.

@Pickett I was just thinking about that. At grocery store now. When I get back I'll look into that. I do attempt re download on http error currently.

Would be easier to investigate with an EvaluateAsynchronous that uses same mechanism as URLFetchAsynchronous. But I think we can only mimic it with QueueRun and ParallelSubmit which might have different behavior.

I think I was also getting a fair number of bad results just from submitting too many URLFetchAsynchronous at the same time. Maybe something like this is the best hybrid:

activeRequests = 0; trafficList = {}; i = 0;
While[i < 1000,
 If[activeRequests < 10,
  activeRequests++; i++;
  getTraffic[pageList[[i]], {2016, 6, 17}, {2016, 7, 17}],
  [email protected]
  ]
 ]

So I limit the rate to only have 10 active asynchronous requests at a time. Still a good deal faster than the 4 I can get with the parallel kernels. And the number of valid results I get seems much higher.

Then getTraffic looks like:

getTraffic[page_, start_, end_] :=
 URLFetchAsynchronous[
  "https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.\
wikipedia/all-access/user/" <>
   URLEncode@StringReplace[page, " " -> "_"] <> "/daily/" <>
   DateString[start, {"Year", "Month", "Day"}] <> "00/" <>
   DateString[end, {"Year", "Month", "Day"}] <> "00",
  Which[
    #2 == "data",
    Module[{data =
       ImportString[FromCharacterCode[First@#3, "UTF-8"],
        "RawJSON"]}, (
      activeRequests--;
      If[KeyExistsQ[data, "items"],

Then I can just run that main loop on a subkernel in parallel and share the trafficList symbol, so it updates in the main kernel dynamically.

Nevermind. I give up. Every thing seems to block the main thread if it ends up working. WaitNext, QueueRun, etc.

Nope. lol

Maybe for all of the shared variable stuff to work properly it requires the main kernel to be active and blocked

Maybe something like RunScheduledTask[QueueRun[], .01] is OK. Each call to QueueRun only retrieves one item added to the list, so I have to call it frequently.

Best I've managed to get. Still not perfect, but better than it was.

pageList =
  WikipediaData["Category" -> "2016 songs", "CategoryMembers",
   "MaxItems" -> All];

getTraffic[page_, start_, end_] :=
 URLFetchAsynchronous[
  "https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.\
wikipedia/all-access/user/" <>
   URLEncode@StringReplace[page, " " -> "_"] <> "/daily/" <>
   DateString[start, {"Year", "Month", "Day"}] <> "00/" <>
   DateString[end, {"Year", "Month", "Day"}] <> "00",
  Which[#2 == "data",
    Module[{data =
       ImportString[FromCharacterCode[First@#3, "UTF-8"],