(@Makyen) I am currently performing a major refactoring of globalvars.py according to this message, by using inner classes and getters/setters, while also improving documentation by the way. However, there are some problems: 0. Is it a good idea? (or I should forget about it) 1. Definitely conflict with existing PRs. (to be continued)
(continued) 2. Would it be preferable to have one big PR after the entire refactoring, or have individual PRs for every inner classes implemented? 3. Is it a concern of the performance cost of locks? I have implemented a rw_lock to solve the first reader/writer problem, and if there is a major performance cost by one reader locking out another reader, maybe I can use it in SD. (to be continued)
(continued) 4. I am current using docstring + documentation that looks like this, and is that the ones SD wants? (I suppose (panic driven development) == (rush through things) && (don't ask, just implement)) (end)
@user12986714 Matched by ^\s*<p>\s*(?:</?\w+/?>\s*)*<a href="(?!(?:[a-z]+:)?//(?:[^" >/.]*\.)*(?:(?:quora|medium|googleusercontent|youtube|microsoft|unity3d|wso|merriam-webster|oracle|magento|example|apple|google|github|imgur|stackexchange|stackoverflow|serverfault|superuser|askubuntu)\.com|(?:(?:lvcharts|php|jsfiddle|mathoverflow)\.net)|github\.io|youtu\.be|edu|(?:(?:drupal|python|isc|khronos|mongodb|open-std|dartlang|apache|pydata|gnu|js|wordpress|wikipedia)\.org))[/"])\W*(?![\W\w]*?</(?:code|blockquote)>) on line 6960 of watched_keywords.txt
@user12986714 this is a fairly common source of FPs but it also catches a fair number of posts which would not otherwise be reported, so it is regarded as useful, albeit noisy
@user12986714 It reports all posts that begin with a link, but not a link from specific domains, and the post must not contain <code> or <blockquote>. As mentioned, it does get a lot of FP, but it's not actually all that bad percentage wise, at least the last time I ran the numbers (a quick search for the last time I ran them looks like it was 59%TP at that point).
@tripleee What we really need to tackle that is a plug-in that lets us run against the entire MS database (I assume we'd have a copy of the database stored local to the machine running the tests) and know the # detections for each reason, which ones, etc, so we can see what the overall effect is on detections.
@user12986714 It's currently a watched regex. I've owed moving it into a separate detection and additional FP reduction for quite a while now, but 59%TP (if accurate) isn't all that bad. As any detection does, it would just be one more detection with a weight based on its %TP.
@JohnDvorak I don't think we have good visibility into that; back when I experimented with running the whole database of detections I gave up because it seemed to overload MS but the symptoms were vague and inconclusive
@user12986714 I haven't forgotten about getting back to reviewing your PRs. I've just been doing other things today. I'll probably be able to get back to them in an hour, or so.
@tripleee should be able to start working once the five problems mentioned is solved. Currently I assume that for problem 0 at least you agree refactoring is a good idea
@Makyen thank you. Though I thought it has cost you disproportionate amount of time to review them w.r.t. my contribution (less than 10 lines) :-(
I've been hoping to refactor the blacklists to the point where !!/watch-ip and !!/watch-ns would be feasible to implement but it's a bit of a mess ... I have a PR in progress so probably we should coordinate if you want to tackle that
@Makyen Also please consider rejecting #3853, as it adds negligible amount of value to SD with added complexity. I cannot close it because I am using a phone now and don't have access to my password database on my computer
@Makyen 'yogateachertrainingrishikesh.com' is not caught by a blacklist or watchlist item.
> Bad keyword in body, bad keyword in title, bad keyword in username, pattern-matching website in body, pattern-matching website in title, potentially bad ip for hostname in body, potentially bad ip for hostname in title ---------- Bad keyword in body - Position 11-28: trainingrishikesh Bad keyword in title - Position 11-28: trainingrishikesh Bad keyword in username - Position 11-28: trainingrishikesh Pattern-matching website in body - Position 11-32: trainingrishikesh.com Pattern-matching website in title - Position 11-32: trainingrishikesh.com
@tripleee the entire codebase, for now, lacks applicable developer's documentation, which is also a major reason I am refactoring the codebase. Since there is few development activities recently, I don't really consider the codebase as terribly maintainable...
@user12986714 I agree that more documentation is desirable. While I'm not a particular fan of formally structured documentation, the code could certainly use more comments to explain what the intent is behind what's being done, what data structures look like, organization, various more descriptive variable names, etc.
(continued) My hope is to use inner classes & proper(?) documentation and refactor the entire codebase over time. Example: now we has a huge pot in metasmoke.py in which various unrelated functionalities are implemented and undocumented, all under class metasmoke... But it can be a long way and we'd better start from a small step.
@Makyen OTOH, formally structured documentation is good for projects where the size outstrips the few developer model, and SD is a bit past that point.
@user12986714 I agree with much of that, but I'm not sure that metasmoke.py is a good example of things that are "unrelated functionalities". It encapsulate things dealing with communicating with metasmoke, which does make them related.
@Makyen my thoughts are that we get three inner classes: one for status ping & statistics, one for ms status (up/down/tracking connection failures), and one as an interface for the rest of SD to communicate with ms
@user12986714 I don't mind additional structure, and I can see where those divisions could be useful. Maybe I was being a bit too pedantic, but grouping everything which is communicating with metasmoke isn't what I'd call "unrelated". It's related at one level. Whether it's the most appropriate grouping to use, is another question.
also, if the code is fairly stable, refactoring it is probably less urgent -- the parts we update the most seem to be findspam (for obvious reasons), chatcommands, and blacklists (when new blacklist types come up)
though that's from my personal memory -- perhaps it would be useful to run some numbers against our git commits
@Makyen Ah get to update my dictionary. I tried to express that the code implementing those three functionalities is mixed together. We have send_status_ping then update_code_privileged_users_list then determine_if_autoflagged...
@tripleee the problem is that we are experiencing random issues recently (example: ms was not considered down even when there is 28 failures) & being a new developer in this project, I get little idea what does what (for lack of doc). Since there is few recent dev activities, I doubt if people still remember what does what... Which makes implementing new features incredibly difficult
@user12986714 a manual redirect might work better, so as not to duplicate information (which tends to lead to at least one of them falling out of sync whenever something needs updating)
but I haven't looked into what Github actually wants
@SmokeDetector Google translation: "Orifice of work clothes is called clothing, which can be selected according to the tastes, type of institution, position of each person in the organization, type of job activity and weather conditions. Workwear or uniforms are an important issue in creating order in the coverage of organizations. Clothing is actually the same cover for employees of organizations, offices and institutions."
[ SmokeDetector | MS ] Bad ip for hostname in body, bad keyword in body, blacklisted website in body, pattern-matching product name in body, potentially bad asn for hostname in body, +2 more (396): Do you know about this product Veyomax RX? by Veyomax RX on askubuntu.com
[ SmokeDetector | MS ] Link at end of body, potentially bad ip for hostname in body, potentially bad ns for domain in body, potentially bad keyword in body, blacklisted user (173): simple definition of technology by soknopurza on superuser.com
Failed to open TCP connection to api.stackexchange.com/2.2/posts/1552836/…*4m9:80 (no implicit conversion of URI::HTTPS into String): Failed to open TCP connection to api.stackexchange.com/2.2/posts/1552836/…*4m9:80 (no implicit conversion of URI::HTTPS into String) | /home/undo/.rvm/rubies/ruby-2.5.7/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect'
@SmokeDetector Google translation: "I think it is not possible to do server-side execution (Google-side) HTML code to replace js by utilizing newly recaptcha images"
Last week I got this phishing mail claiming to be from Amazon.
Is this really from Stack Exchange?
Where would I report this?
Update:
It really seems not to be sent by stackexchange.
[ SmokeDetector | MS ] Bad ip for hostname in body, bad ns for domain in body, blacklisted website in body, link at end of body, pattern-matching product name in body, +3 more (694): Peak Bioboost Reviews by tammy bryan on askubuntu.com
Last week I got this phishing mail claiming to be from Amazon.
Is this really from Stack Exchange?
Where would I report this?
Update:
It really seems not to be sent by stackexchange.
