:66252154 Matched by (?:91|0)?[6-9]\d{4}(?<!\d\D\d{5})[^A-Za-z\d]*+\d{5}(?![\s,]++\d)[^A-Za-z\d]++(?:91|0)?[6-9]\d{4}[^A-Za-z\d]*+\d{5}(?![^A-Za-z\d]*+\d)(?=.{0,200}+\Z) on line 48813 of watched_keywords.txt
Yeah. The full expression would be something to the effect of (?s)(?:^|\b)(?:91|0)?[6-9]\d{4}(?<!\d\D\d{5})[^A-Za-z\d]*+\d{5}(?![\s,]++\d)[^A-Za-z\d]++(?:91|0)?[6-9]\d{4}[^A-Za-z\d]*+\d{5}(?![^A-Za-z\d]*+\d)(?=.{0,200}+\Z)(?:\b|$)
(?s)(?:^|\b)(?=\d(?!\d:|\d{4}-\d{5}:))(?:91|0)?[6-9]\d{4}(?<!\d\D\d{5})[^A-Za-z\d]*+\d{5}(?![\s,]++\d)[^A-Za-z\d]++(?:91|0)?[6-9]\d{4}[^A-Za-z\d]*+\d{5}(?![^A-Za-z\d]*+\d)(?=.{0,200}+\Z)(?:\b|$) @Spevacus any suggestions?
@Makyen Im looking into adding a way to watch both versions with one command for numbers with leading zeroes. should that be an auto detect, or something that is done through a comment?
@Dragonrage I was planning on having it as part of an auto detection, but the numbers that have 0 prepended are also NANP numbers, or at least the ones I've been noticing are. I needed to look more into what the actual formats are for the Indian numbers which are like that to see if there are exclusions, such as exist in NANP. My general concept was to have all of the 1, 91, and 0 permutations used unless the user specified in a comment not to use one or more or to only use NANP vs Indian.
It's also possible that I'm not thinking of the same numbers you are, so could be off-base.
@Dragonrage All of the number watches/blacklists will need to be looked at and, potentially adjusted. At a minimum, there will be significant overlap. When I did the NANP detection and moving the numbers from the watchlist/keyword blacklist to the numbers lists I used the CI testing to find the duplicates. IIRC, I had it generate the command I'd need in the editor I use to remove the appropriate lines that were duplicates. If you use that part, it's very much a use at own risk and double-check.
im tempted to write up a small script to parse the watch/blacklist to cleanup the duplicates. I.e. remove all numbers that dont have a leading zero but are a match for another watch that has it
Ill say, I am definitely working much slower in python than I would in C#. but hopefully my python will be better after i get through it
@Dragonrage The CI testing already does this, assuming that you're adding the "0" version as an alternate normalized version.
Well, more accurately, it explicitly tells you which ones result in duplicate normalizations.
I generally just ran the CI tests against my local SD git repository to get that information.
The run_tests.sh script does that for you on a local repository.
For just the blacklists/watchlists, I usually modify that to just python3 -W default::Warning -m pytest test/test_blacklists.py, which I have in a different script.
@SmokeDetector appears to be self promotion of their website. website appears in their bio, and on every site except travel, they advertise they are a "Vietnam based tour operator, web developer, blogger, writer and photographer"
np To be clear, my intent was that the default for appropriate numbers (potentially all/most 10 digit numbers) that they would be normalized to each of "1", "91", "0", and "", unless there was an indicator that some of those should not be used.
And excluding the "1" combination if it can't be a NANP number.
@Makyen so if i watch a 10 digit number, it should also watch the normalized versions of said number with a prefix of 1, 91, or 0 as well, unless there is a comment saying not to?
also i was checking back on the 0 prefix numbers, and a lot i see are 0####-###-### format. is that still considered NANP? typically a north american number would be 3-3-4 format, but i guess it could be just people messing with the format to try to avoid spam detectors
@Dragonrage That was what I was going to go with. The chances of FP are relatively low and having that as the assumed default would substantially reduce the number of times people get the "is that NANP" reply from SD. Alternately, such a reply would now need to be for nearly every 10 digit number asking for more specificity. Basically, the observed FP rate for the entries we do have doesn't really justify forcing people to be more specific all the time.
Although, I'd probably have the reply remind them that they can be more specific.
@Dragonrage Yeah, we can't assume that's not some type of obfuscation of a NANP number.
Yes, it makes it likely it's not a North American number, but the digits probably comply with NANP. Currently, we only assume that 1) a 10 digit 3-3-4 number is NANP or 2) that an 11 digit number that starts with "1" is NANP (for both, the number also has to be a possible NANP number).
I'd leave those two assumptions.
I'd assume an 11 digit number that starts with "0" is Indian ("0", "91", and "" combinations).
I'd assume a 12 digit number that starts with "91" is Indian ("0", "91", and "" combinations).
I'd assume that a 10 digit number is Indian and NANP, if it also complies with all other NANP requirements. So, "1", "91", "0", and "" combinations.
The entry should also be able to have either or both of the "(?#NO NorAm)" directive and a new directive such as "(?#NO Indian)".
@Makyen Can I have a bit of advice at some point? I made a (draft) PR to address multiline chat watches. However, there's also this GH issue, which is for SD ignoring short multiline messages. Ideally, I'd like the GH issue fixed so I can test the draft PR with the fix.
While I think I can/know how to fix the issue of SD ignoring short multiline messages, should that be a separate PR? If so, is the right way to do that to make a new branch in my fork of SD based on master and send a second PR from there?
@Makyen the only thing here I wonder about is if we should force a command of one of the NO directives or some sort of affirmative of yes watch all 4 combinations
@Makyen Ok, I'll make that a separate PR and make a separate branch going forward. For the multiline watch item, I already put that commit in master (on my fork), and then made the PR based on that. Should I make a new branch, add the change there, close my PR, and make a new one from the new branch? Or is there a better way to fix that one...?
@cocomac I'm wondering how you are proposing to address them, other than just ignore the command or reply with an error that multi-line watches aren't supported.
basically expanding the current behavior of watching a 10 digit number that could be a NANP number. currently it requires you to either force, or add one of the two flags/directives (IS NorAm or NO NorAm). should that instead now ask for (NO NorAm, NO Indian, YES All), or should it just accept it as all?
@Dragonrage My intent was to just accept it as all and reply that that was what SD was doing, along with reminding that there's the option to tell SD not to use the different combinations. Basically, the current operation appears to be a pain in the rear for people a lot of the time, while the downside of looking for the other combinations is quite small (i.e., too much pain for not enough gain :) ).
@micsthepick They are, and always have been, visible to users with the Core role.
@Makyen Appears resolved. Initial problems were probably that my Firefox needed to be restarted. However, I broke the code in trying to figure out what was going on/test prior to restarting Firefox, so I still saw the problem after restarting. Stopping, doing other things, then going back to it made the issue introduced in testing obvious.
@CodyGray Yes, it would be. It's something I've desired for years. I usually end up using a complex regular expression to accomplish that.
@micsthepick That's equivalent to aspadol(?![\W_]*+\d{3}). given that there's nothing after your \d{3,10} it's not actually limited to 10, so it's really just checking to see if there are >= 3 digits.
[ SmokeDetector | MS ] Bad keyword in link text in answer, potentially bad asn for hostname in answer, potentially bad keyword in answer, potentially bad keyword in username, username similar to website in answer (157): Can I solve this with a static route? by Vins Fertility on serverfault.com
what we sometimes do is !!/watch keyword\d*+ which will match keyword with or without numbers at the end. If the spammer writes a space before the numbers, we simply ignore them
if you want 100 to be included in the match if it's there, that's fine as such, but probably not what we would usually do; if it's aspadol anything it's aspadol and we are done
but why do we care? again, if it's aspadol it's aspadol
aspadol(!?[\W_]*+\d{3]) will also match if it's followed by one or two digits
@Makyen tangentially related, if the number begins with a plus we already know it's an international number; this is something I've been planning to fix but if you update the code, this would be nice to include
@micsthepick dan.com is a domain squatter, they own a large number of BS domains like all.zip and also occasionally post spam advertising a domain for sale. I know the watch is noisy; my plan is to collect the common mydomain.com etc FPs and probably add them to the domain whitelist, or perhaps create a separate whitelist
@tripleee No, that's not an accurate assumption. I've definitely seen 10 digit NANP numbers with a plus in front of them without the "1" country code. I've also seen Indian numbers that start with a "+", but without the country code.
@micsthepick No, it's a lookahead, negative lookahead to be accurate, so the text isn't part of the match.
@micsthepick Messages/reports from SmokeDetector in Charcoal HQ are generally kept as records. If you really need to delete a message, please use sd delete-force. See this note on message deletion for more details.
@micsthepick Yes, I'm aware that aspadol[\W_]*+\d{3,10} and aspadol(?![\W_]*+\d{3}) are in the watchlist. The combined effect is that aspadol[\W_]*+\d{11,} will not be caught. Overall, it would probably be more efficient as a single regular expression. Previously, I was only stating that aspadol(?![\W_]*+\d{3,10}) and aspadol(?![\W_]*+\d{3}) are equivalent regular expressions that will match exactly the same things.
@micsthepick Go be with your friends. Real Life is more important.
@cocomac Not sure if Makyen already answered this one, but here goes anyway... There's nothing technically wrong with creating a PR based on the master branch in your fork. However, it is a potential accident waiting to happen. See, when you have an open PR based on a branch, any changes you later make to that branch will be reflected in that PR. So, if you base the PR on your master branch, then go on and do some other edits, those edits will be invisibly added to your PR, which could result in your PR containing some stuff that you did not intend. This is why it's "best practice" to do each PR from a separate branch dedicated specifically to that PR. Since you didn't know, it's fine to keep master as the branch dedicated to your existing, open PR. Just make sure not to use it as a "working" branch.
@CodyGray I see, that makes sense, thanks! I think it'll be an issue, so is it fine to close the PR (with a note)? I did the work in master, but when I tried to make a new branch for fixing short multiline messages, that included my changes on master, and I think the easiest thing is just for me to delete my fork & do it again the way you & Makyen described.
Creating a new branch doesn't have to include your changes on master. To do it correctly, you need to switch to a non-head commit (i.e., one commit back from your master head commit, which is not associated with a branch). Then make your changes.
You don't have to close the PR. You can rebase master and that'll fix the PR (as long as it wasn't merged, obviously).
@Dragonrage That pattern looks like it's already caught by Potentially bad keyword in body and Potentially bad keyword in answer Append -force to the command word(s) if you really want to add the pattern you provided.
These will, undoubtedly, be FP-prone, but we missed that last one, and I've seen other spam like this that was missed, so hopefully this will help address that gap.
[ SmokeDetector | MS ] Bad keyword in body, blacklisted website in body, potentially bad ip for hostname in body, potentially bad keyword in body, potentially bad keyword in title (195): BTC RECOVERY EXPERT by Michael on bitcoin.SE
