Wolfram Mathematica

« first day (1642 days earlier) ← previous day next day → last day (3140 days later) »

kirma

05:18

Somehow I suspect manufacturing custom floor tiles wouldn't be cheap...

The big tile is perfectly level, but at least I "see" it's actually slightly tilted.

10 hours later…

Michael Hale

15:30

So if you have say 1000 web requests you need to make (say on average they take a half second to respond), and after each response you need to update a dynamic list, how would you do this for the best combination of speed and stability? It currently seems like launching a bunch of URLFetchAsynchronous calls is the fastest, but often as many as 1/3 of them get lost, and using ParallelSubmit (or of course even just looping through them on the main thread) is the most stable but a good deal slower.

I'm still experimenting though.

Pickett

16:20

@MichaelHale Does it "get lost" without any message whatsoever? Otherwise maybe using the callback to initiate a new download might still be the fastest approach.

Michael Hale

@Pickett I was just thinking about that. At grocery store now. When I get back I'll look into that. I do attempt re download on http error currently.

Would be easier to investigate with an EvaluateAsynchronous that uses same mechanism as URLFetchAsynchronous. But I think we can only mimic it with QueueRun and ParallelSubmit which might have different behavior.

1 hour later…

Michael Hale

17:50

I think I was also getting a fair number of bad results just from submitting too many URLFetchAsynchronous at the same time. Maybe something like this is the best hybrid:

activeRequests = 0; trafficList = {}; i = 0;
While[i < 1000,
 If[activeRequests < 10,
  activeRequests++; i++;
  getTraffic[pageList[[i]], {2016, 6, 17}, {2016, 7, 17}],
  [email protected]
  ]
 ]

So I limit the rate to only have 10 active asynchronous requests at a time. Still a good deal faster than the 4 I can get with the parallel kernels. And the number of valid results I get seems much higher.

Then getTraffic looks like:

getTraffic[page_, start_, end_] :=
 URLFetchAsynchronous[
  "https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.\
wikipedia/all-access/user/" <>
   URLEncode@StringReplace[page, " " -> "_"] <> "/daily/" <>
   DateString[start, {"Year", "Month", "Day"}] <> "00/" <>
   DateString[end, {"Year", "Month", "Day"}] <> "00",
  Which[
    #2 == "data",
    Module[{data =
       ImportString[FromCharacterCode[First@#3, "UTF-8"],
        "RawJSON"]}, (
      activeRequests--;
      If[KeyExistsQ[data, "items"],

Then I can just run that main loop on a subkernel in parallel and share the trafficList symbol, so it updates in the main kernel dynamically.

Michael Hale