sdc report stackoverflow.com/a/74573964 "Undisclosed affiliation with Seasoned: See their LinkedIn and Apollo profiles; the usernames and company logo match the info on Seasoned's GitHub repository exactly. In addition, post is VLQ (link-only). Custom flagged."
@SmokeDetector Custom flagged: This may look like an off-topic question, but it's really spam. It's part of a campaign of spam for learnshareit.com, posted on multiple SE sites using multiple accounts/usernames over the last couple of weeks. The spam has varied from straight spam, to plagiarized + spam link, to apparently valid questions or answers with a spam link, which are commonly cross-posted on multiple sites and/or off-topic with the appearance of being on the wrong site.
@JeffSchaller That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer Append -force to the command word(s) if you really want to add the pattern you provided.
@SmokeDetector While the views on the linked video are low and the number of followers is low, the GitHub link in the YouTube channel's "about" page doesn't appear to match the account from which this was posted on SO. So, I'm going with NAA, for now.
@Makyen maybe we're looking at different things; I saw an email address on YouTube with "monud" which was close enough (for me) to "Monu Dahiya". I'm not trying to change your mind so much as double-check that we're seeing the same things
@JeffSchaller Getting the email address from the YouTube channel's about page requires accessing the page while logged in. Accessing it will be recorded by YouTube/Google. I try to limit such tracking when investigating potential spam, so don't log into a Google account in the sandbox I use for such investigations. As a result, I don't see the YouTube channel's email address which is "For business inquiries" and hidden behind a click.
The GitHub account linked for the YouTube channel is Navdeep, which didn't appear connected to the SO user, at least from a relatively cursory check. That GitHub account name is also consistent with the name used for the LinkedIn account for the YouTube channel.
Given that you've said that's the email address for the YouTube channel, that seems enough to connect the two.
@Makyen i may steal you if you know how to make a new thread launch from SD and do something on a regular time basis - or if not you someone else. Still working on the mongo approach i had in mind for pickle sync but need at some point a quick how to for a timed thread so when it triggers every X minutes it does the stuff
@ThomasWard How to do this is going to depend on what, exactly, you're wanting to do and how much coordination there is between this sync and the dumping of pickles. There should probably be some coordination which results in pickles being dumped shortly prior to syncing. For that, I'd probably pull out the part of helpers.exit_mode() which is the five lines after and including # Flush any buffered queue timing data into its own function, maybe dump_pickles(), or something similar.
The line in ws.py which is Tasks.later(restart_automatically, after=21600) shows a way to run a task at a later time. One of the ways which you could use to periodically sync is to have a function which performs the sync and then sets itself to be called later, again. That would rely on the existing Tasks thread. Alternately, you could start your own separate thread, which could be similar to the lines in ws.py which are:
@Makyen well by 'sync' i mean it'd be a call which every X minutes takes the corresponding running globalvar/dictionary/item/etc. and dumps it to a MongoDB object by id. Then on a restart during the init of globalvars compare the value from the pickles to the value in MongoDB and if MongoDB is newer pull from Mongo for the data
that way when it dumps to local pickle it's accurate to within X-time based on the last sync
@ThomasWard OK. I wouldn't go directly to the variables in GlobalVars. I'd stick with syncing the actual pickles. If you go directly to the variables in GlobalVars, that introduces an additional place where what's getting saved has to be defined and those low level definitions of what to save have to be kept in sync when things are changed. What gets saved already has to be explicitly defined for the pickles, so I'd just stick with those.
It wouldn't be unreasonable to have part of the datahandling.dump_pickle() and datahandling.load_pickle() utility functions wrap the data being saved within a structure which also allows storing and recovering a timestamp of when the data was stored.
@Makyen this makes sense. but we also would need to deal with the issue of actually syncing it more regularly than normal, unless we dump to the .pickle regularly
@ThomasWard Yes, which was why I mentioned pulling out the syncing which is done in exit_mode() into its own function, which you could then call when you needed a current sync. Currently, most of the pickles are stored onto disk substantially more often than they really should be (e.g. at every scan), rather than when the data needs to be saved prior to reboot/crash.
You'll also need code which will add thread locks for each/all of the pickle files, so you're not syncing something which is in the process of being saved (or only perform the sync upon reboot, prior to reading all the files). I have code which adds locks for all of these, which we also need for other reasons. I'll see if I can finish testing it (and at least re-re-reviewing it, and verify that it's really written in a way which is reasonably extensible).
@Makyen what i should be saying though is, the data can be dumped easily, but we need to implement the 'on load, check if we're really out of date' based on timestamp, etc.
to actually sync the pickles when we first start SmokeDetector or such
that's the part where we'd have to implement some 'on load' stuff
i'm still working the Mongo side to fine tune things - like how Python loads the data, dumps it to Mongo, etc. and whether we dump it as straight bytes or depickle it and dump that into Mongo first.
[Ahh, yes, more caffeine, that's a good idea...] :;
Yes, the on load logic would need to change to account for such things. We'd also need to make choice of what to do when things get out of sync (e.g. changes stored to the database, but unable to be retrieved by a newly starting SD, but then the new SD has changes, etc.).
@Makyen we'd also have the same problem if SD can't connect to MS, then we have some desync there
given that I'm probably putting the Mongo interface if this goes live on MS's server, that makes a little more sense for things to get online. But that's implementation logic once we get stuff done there
and probably it's just you and I spinning 'new' SDs nowadays
since the infrastructure is stable as long as my internet doesn't asplode
and usually if it does we're only a few hours out of sync at most which isn't "the end of the world" but that's something to poke at still yes
@Makyen when we get closer to a finished product and implementation in prod we can start brainstorming those cases. Base implementation will be simple, but then deciding the logic for those cases will be something to consider
@Makyen DB will either live on the MS server or on one of my cloud servers in encrypted form, so 'not reaching' the instance is actually harder said than done ;)
(assuming someone has access to the base config, it's just an auth key we use)
