SEBotics

« first day (34 days earlier) ← previous day next day → last day (2885 days later) »

Bhargav Rao

10:15

@jam alive

Jammy

@BhargavRao Nope

Jammy

10:29

[ Natty | Sentinel ] [tag:lenovo] Link to Post BodyLength: 210 Rep: 1 Ends with ?; Low Length; No Code Block; One Line only; Low Rep; Body Starts With: I'm looking here for help. I did follow steps above but on step 4 after "Try Ubuntu without installing" nothing happens. Screen stays black. Can someone please support me on setting up linux on Miix

5.0; @BhargavRao

Bhargav Rao

@Jammy tp

Yup, it's working .... woohoo!

jokerdino

11:23

\o/

Jammy

12:08

[ Natty | Sentinel ] [tag:software-installation] Link to Post BodyLength: 167 Rep: 1 Low Length; No Code Block; One Line only; Low Rep; Unregistered User; Body Starts With: This is helpful discussion. I enable/evoke python 3.x when I write python3 on the terminal. And it is the default version. So how to upgrade to upper version of python 3.5;

@BhargavRao

Bhargav Rao

@Jammy tp

@Art there?

ArtOfCode

aye

Bhargav Rao

Sentinel's working perfectly. Can you remove the first 3 posts on AU sentinel?

ArtOfCode

when I get a chance, yeah

Bhargav Rao

awesome, I'll push it to server today and be done!

ArtOfCode

12:20

there is a little bit more work to do to make Sentinel function properly - I need to set it up with SA credentials - but it's mostly there

Bhargav Rao

Ah, so should I wait before a deployment?

ArtOfCode

nah, you can deploy whenever, and I'll do all that when I have the chance

Bhargav Rao

And this will all be managed by you and not Undo?

ArtOfCode

aye

Bhargav Rao

Okie! Allz well. I think we can expect around 30 posts per day max.

But it'd still need around 6k reqs. Need to work on that part.

ArtOfCode

12:25

queue 'em up and fire off a bunch in one go

Bhargav Rao

Yeah, I waited for 5 minutes instead of the normal 1. In that case, the number of requests reduced quite drastically. (to 3k~)

ArtOfCode

@BhargavRao You could do something like Smokey does with bodyfetcher

Bhargav Rao

I'm thinking of that, at the moment, 3k of AU + 6k of SO, should be somewhere near by to the 10k

@ArtOfCode what does it do?

ArtOfCode

instead of firing an API request as soon as you get the post, store the post ID in a queue. Fire the API request when you have 5 IDs queued up, and your requests decrease by 5x.

Bhargav Rao

Hmm, that's quite interesting.

ArtOfCode

12:29

smokey wouldn't have anywhere near enough quota to fire one request per post if we didn't have bodyfetcher queueing - we have 20k quota and 100k+ posts per day

Bhargav Rao

What I do is, wait for 5 minutes, collect the answer IDs in the past 5 mins, then batch all and fire. (No web-sockeys). But the issue is, for every answer, I need to fire another batch for the questions. So it is 3 calls per 5 minutes.

In a few cases, the number of posts increases above 100, so I'll need more

ArtOfCode

@BhargavRao Hm. Yeah, that's a pain

Bhargav Rao

For SO, it comes up to 6k requests (there it is per minute + reportings + deleted stuff). I think I should be able to do some stuff to reduce it to 5k.

ArtOfCode

What you could do is dynamic intervals, though. In periods of low traffic, wait 15 minutes between captures, in high traffic wait 3.

Just have some code to calculate the next interval based on how many posts you had in the last request - i.e. if you had 120 posts in the last request over an interval of 5 minutes, the next interval should be (120 / 5) * 5 minutes

Bhargav Rao

I actually planned that, but did not go far. I guess, that should work out. The other day, we ran out of requests around 1 hr before the rollover.

ArtOfCode

12:35

no that's not right

int nextIntervalSeconds = (100 / postsInLastRequest) * lastIntervalSeconds;

Bhargav Rao

I actually thought of another idea. If the number of posts this minute is less than 20, skip and wait for the next minute.

ArtOfCode

Similar kind of thing

I'd probably go with the mathematical approach, just because it's more accurate about how long you need to wait

make it 95 / posts for a safety factor

Bhargav Rao

Yeah. I'll try to get some code for that.

It's like just 500~ more requests needed.

Bhargav Rao

12:51

Hmm, @Art, The mathematical approach needs some refactoring. At the moment both AU and SO process occur side by side. So I can't individually change the rates of each. :|

Should have planned before

I'll try the crude approach.

ArtOfCode

@BhargavRao Oh, is it all in one bot? I thought you were running two separate instances.

Bhargav Rao

Yep, I refactored the code completely to make it easier to add another site.

ArtOfCode

Ah. Yeah, probably want to separate the interval timings as well, then

Bhargav Rao

Now, anyone from anysite can use it, but only one condition, they must host it (and not in my IP)

I actually had separate threads for the 4 different rooms earlier. Then it all screwed up big time. So I removed everything and put it in one thread.

2 hours later…

Jammy

15:17

[ Natty | Sentinel ] [tag:12.04] Link to Post BodyLength: 1644 Rep: 1 Contains Blacklisted Word - How do I; Ends with ?; No Code Block; Low Rep; Unregistered User; Body Starts With: "\[14263.876595] Buffer I/O error on dev sr0, logical block 38192, async page read 6.0; @BhargavRao

Bhargav Rao

@Jammy tp

« first day (34 days earlier) ← previous day next day → last day (2885 days later) »

all rooms

Transcript for

Apr2

Apr '173

Apr4

A side expansion of the SOBotics project to the SE network. Fo...

bots sea waffles

join this room
about this room

00:00

06:00

12:00

18:00

all times are UTC