!!/watch- econedge\.org

[ SmokeDetector ] Blacklists reloaded at rev a3e2129 (SmokeDetector: Auto watch of econedge\.org by Makyen) (running on teward/Osiris)

[ SmokeDetector | MS ] Bad keyword with email in body (57): Powershell ConvertFrom-Json output json format not correct by satya Narayan mohanty on stackoverflow.com

fp- feedback received

Looks like someone disabled circleci on Metasmoke PRs

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): Klotski Puzzle; Solution or Program needed by Mario on puzzling.SE

iBug/Coral: Executing automatic scheduled reboot.

[ SmokeDetector ] SmokeDetector started in standby mode at rev 1f80a75 (SmokeDetector: Auto watch of myxml\.in/xml-utils\.html by Daniil) (running on iBug/Coral)

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Sum with decimal numbers returns NaN by Milly on stackoverflow.com

fp- feedback received

@ThomasWard Do you want to disable MS trying to dump the DB?

@thesecretmaster we already disabled MS DB dumping

for now

somewhere we disabled that

@thesecretmaster but yes, we're going to be implementing a MariaDB native solution that offloads the sanitization etc. from MS to another machine so as not to drag down MS availability

which was a primary headache we've had every time the DB backup ran, even in public environments.

I also demanded DB dump disable because of the security issue I found and then fixed in teh SQL dump scripts and sanitizer scripts

I don't think it's been enabled since, pending me finishing the replacement system

@thesecretmaster ULTIMATELY with the separate mechanism in place MS will only have to index a single file URL to serve from rather than do the dumping itself

just gotta balance the cronjobs

Looks like charcoal-se.org won't auto deploy from GH

we know this

@user12986714 where have you been the past 3 hours lol

Cool, just wanted to ask if MS code changes are needed, and if so I'd appreciate an issue so that can happen.

@thesecretmaster i'll request DB dumps be deactivated

and I don't think we've done a redis dump in $ETERNITY

otherwise i'd see it (core+admin+dev+...)

@ThomasWard looking through before_action of all controllers...

They existed... at some point

@user12986714 Thank you for doing that, by the way. I'm sure there are more places where we've missed access checks.

@thesecretmaster yeah once I get the prototype system properly dropping SQL dumps to the right location on server then we won't need to use Metasmoke+Rails to actually execute the dump

@thesecretmaster maybe; can't find any though :-P

it'll just be "Done" and MS only has to update the URLs

unless of course Undo wants to use S3 still

but since we've moved the infra to my systems and have 250GB of space and no current bandwidth transfer limits I can find... :p

@user12986714 Wow, that's surprising, although I guess it's a good thing :P

the unsanitized backups are kept on the processor system which is backed up weekly to my cloud underneath a PGP encryption key

@user12986714 you found a couple, and we fixed a couple, that last one's By Design since we're less dependent on S3 now and more dependent on my infra being Online :P

@thesecretmaster ^ the only issue is how we configure MS to configure the URL proper for a local file pickup, but that's probably trivial

assuming I get the SFTP problem between the two machines fixed.

though, with the move to SSD the DB dumps probably don't take long anyways

@thesecretmaster regarding the Redis dumps, are there things in the redis dumps that need sanitized before being made available or no?

Nope, it's all public IIRC

Pretty sure we just copied the rdb file

that could have been disabled. I'm going to open a PR and tag it 'security' and 'db' indicating we should disable all DB dumps done by Ruby for now to make sure it's disabled

and since MS backups didn't exist in 'easily restorable' form previously, the second machine involved makes sure we have restorable SQL DB dumps in the event of Catastrophic DB Failure

or data loss like we did the logs for a few months

@thesecretmaster github.com/Charcoal-SE/metasmoke/issues/759 for your reference

in case you want to make the changes :P

though at the system level I think I chmod 000'd the script... heh

@SmokeDetector Unsanitized HTML in this one (<button> seems to trigger it)

@thesecretmaster also in case you're curious it's up to @undo if we run it in S3 still or just straight off the MS box. We're not bound by costs for data transfer anymore, so we have the bandwidth to spare.

we could share the dumps right off the MS machine directly

and then not have to rely on S3

the only thing we'd still need is the ancillary VM I'm working on so taht we keep the private backups private AND keep those over time for recovery/compare purposes.

@ThomasWard Fine by me

[ SmokeDetector | MS ] Bad keyword in title (98): Need to collect the list of colleges in India by Chandru Raj on stackoverflow.com

cool then we'll move forward with sharing straight off the same machine, no need to have S3 costs if we don't need them right?

@Undo that also reduces the requirement to have Core to get the public sanitized dumps, in theory we could then let anyone who wants them (with the proper URL) to get them

Ye

yep

.... oh FOR THE LOVE OF... stupid Microsoft Exchange at work being stupid...

goes to fix

The reason before was to keep some idiot from downloading it nine million times and ballooning my S3 bandwidth out cost

(Some idiot meaning someone we irritate)

lol

well THANKFULLY I'm not charged per GB of data that goes out the door

@JohannesKuhn I'm assuming you're talking about the view in FIRE. Correct? Ironically, it's de-sanitized HTML, where FIRE needs to reverse the transformation which SD performs on the HTML prior to running most tests. I have code to fix it. If I can get some time, I was about 80% through bundling up a new version of FIRE with that fixed. At this point, there are a couple of additional features I'll probably squeeze in prior to release.

[ metasmoke ] ci/circleci success on e520f6d: Your tests passed on CircleCI!

@Undo yeah, well I'm suggesting we leave the URLs and URL access structure the same as is

whether or not anonymous can get to the page or not

because that way "idiots" can't get to it without the URL anyways

AND if we have more data on region of the baddies, I can even drop in some GeoIP blocks :p

@Makyen Yeah, FIRE. Not sure how exploitable that is.

Hope I will see more blue usernames here in a few days :)

@JohannesKuhn That is a concern.

[ SmokeDetector | MS ] Potentially bad keyword in answer, blacklisted user (72): How can I insert a cross-reference to a figure in Google Docs? by neocat on webapps.SE

fp- feedback received

Mhh, testablilty is a problem. I have to create a post with a bad keyword and a <button onload="..." />.

@JohannesKuhn Hopefully. :) We'll find out in a week. :)

@user12986714 That user posted the exact same answer to that question before.

@SmokeDetector tpu-

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Are the teeth of this machine iron or steel? by Lonelyman40 on crafts.SE

[ metasmoke ] ci/circleci success on e46972f: Your tests passed on CircleCI!

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): Implementing Search function in flutter by OmkarKhilari on stackoverflow.com

fp- feedback received on Are the teeth of this machine iron or steel? [MS]

naa- feedback received

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): React y GraphQL en tiempo real by David Arevalo on es.stackoverflow.com

[ SmokeDetector | MS ] Few unique characters in answer, no whitespace in answer, potentially bad keyword in answer, repeating characters in answer (263): forbid her anything and it became her desire by Edwin Villanueva on english.SE

tpu- feedback received

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Will a Xiaomi Redmi Note 9 (international version) work reliably in the United States (comparable to a domestic phone)? by DrJuiceBoy on android.SE (@AndrewT.)

fp- feedback received

[ SmokeDetector | MS ] Blacklisted username, blacklisted website in answer, pattern-matching website in answer, potentially bad ip for hostname in answer, blacklisted user (333): Atom is running slow when edit a file which is over 500 lines by techmartin on stackoverflow.com

tpu- feedback received

naa- feedback received on React y GraphQL en tiempo real [MS]

[ metasmoke ] ci/circleci success on f879c65: Your tests passed on CircleCI!

[ metasmoke ] ci/circleci success on 591cc80: Your tests passed on CircleCI!

@thesecretmaster are you able to see if the DB update is actually disabled or not via code? If not, I can just chmod 000 the db dump script and then it'd do nothing. Or just exit 0 before doing anything.

[ metasmoke ] ci/circleci success on c302ef3: Your tests passed on CircleCI!

[ SmokeDetector | MS ] Bad keyword with email in body, potentially bad keyword in body (58): How will I be able to display keys? by Mo Abbas on stackoverflow.com

fp- feedback received

naa- feedback received on Why can rolled up capacitors be modelled as flat parallel plates? [MS]

@ThomasWard github.com/Charcoal-SE/metasmoke/blob/master/config/…

yeah i should disable this for now...

and github.com/Charcoal-SE/metasmoke/blob/…

@thesecretmaster yeah not touching that one because I don't want to explode it. Just gonna disable the cronjob for now. And then when it DOES run it just has to pick up the file from its directory location and handle the URL locally

and github.com/Charcoal-SE/metasmoke/blob/…

but that's going to be someone else needing to do that because I'm not a Ruby pro

[ metasmoke ] ci/circleci: rubocop failure on ffd0917: Your tests failed on CircleCI

ooops

[ metasmoke ] ci/circleci: rails-tests failure on ffd0917: Your tests failed on CircleCI

[ metasmoke ] ci/circleci success on 796d293: Your tests passed on CircleCI!

@thesecretmaster This can get nuked

@thesecretmaster This needs to stay until DumpController gets revamped

@thesecretmaster that can be nuked only while we have the cronjob disabled

if we revamp the job then we might not even need a dump controller (I think?)

[ SmokeDetector | MS ] Pattern-matching website in answer (78): Undo command in Minecraft by nijo on gaming.SE

[ SmokeDetector | MS ] Potentially bad keyword in answer (1): Can sex with a prostitute be permissible if they become a temporary slave? by Zubair on islam.SE

@ThomasWard Dump controller is an MVC controller for the web side

ah OK

[ SmokeDetector | MS ] Few unique characters in answer, mostly non-latin answer, mostly punctuation marks in answer (171): Choosing a Random Background - HTML/CSS by USer bOi on stackoverflow.com

@Undo I disabled the DB dump scheduler for now just an FYI. My prototype ancillary support box for the backups processing/storage has a backup from yesterday and in... 3 hours... will have a new backup from the day (~00:00 UTC)

tpu- feedback received

at least getting the dumps done works but moving them back it doesn't like it so i'mma have to futz with perms to make it work right

[ SmokeDetector | MS ] Link at beginning of answer (35): Starting pentaho after first installation by a deleted user on stackoverflow.com

naa- feedback received

The message about GitHub issues can be unpinned now that those issues are resolved.

Done

[ SmokeDetector | MS ] Few unique characters in answer (87): What's the etymology of the word "zilch"? by David K Foat on english.SE

[ SmokeDetector | MS ] Offensive answer detected, toxic answer detected (158): Is there more to Whitehead’s “Age-Spirit” than metaphor? by Philosophy.s on philosophy.SE

tpu- feedback received

tpu- feedback received on What's the etymology of the word "zilch"? [MS]

[ SmokeDetector | MS ] Offensive body detected, potentially bad keyword in body, toxic body detected (104): Calculate the character of the element g ✏️ by Albert Haimmends on math.SE

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Firstly, Spider ID? Also possible relation to ~100+ serious bite reactions in past 1-2yrs by Custom Creations on biology.SE

tpu- feedback received on Calculate the character of the element g [MS]

[ SmokeDetector | MS ] Potentially bad ns for domain in body (1): How to get a quick abs six pack? by CAPS LOCK on meta.SE

tpu- feedback received

[ SmokeDetector | MS ] Potentially bad keyword in body (1): Shared library symbol conflicts and static linking (on Linux) by xnervwang on stackoverflow.com

[ SmokeDetector | MS ] Potentially bad ip for hostname in body (1): Permalink Rewrite to include Custom Taxonomy Term ✏️ by user1729819 on wordpress.SE

fp- feedback received on Shared library symbol conflicts and static linking (on Linux) [MS]

fp- feedback received

[ SmokeDetector | MS ] Potentially bad ns for domain in body (1): How do I add a link with PHP? by Shreesh Agarwal on wordpress.SE

[ SmokeDetector | MS ] Potentially problematic ns configuration in body (1): What does the number after each command that listed by the apropos command mean? ✏️ by S M Fahim on unix.SE

fp- feedback received

fp- feedback received on How do I add a link with PHP? [MS]

!!/watch- fitwirr\.com

[ SmokeDetector ] Blacklists reloaded at rev 2035be1 (SmokeDetector: Auto watch of fitwirr\.com by Makyen) (running on teward/Osiris)

@Undo how hard is it to get MS to link to / index the local file? Since we won't need to dump to S3 anymore

[ SmokeDetector | MS ] Link at beginning of answer (35): A Vampire is still alive after mankind is gone by Bassoe on scifi.SE

[ SmokeDetector | MS ] Potentially bad keyword in body (1): How to write a message from a console app opened from a windows app c++98 by 3pa2 on stackoverflow.com

@SmokeDetector naa- I'm going to give this one the benefit of the doubt. They don't seem to be affiliated with it, and it's a drawing remotely relevant to the question.

iBug/Sandy: Executing automatic scheduled reboot.

[ SmokeDetector ] SmokeDetector started in standby mode at rev 129c8d2 (Makyen: Merge pull request #4144 from user12986714/patch-44) (running on iBug/Sandy)

!!/scan stackoverflow.com/a/62885256

@IanCampbell Post 1: This does not look like spam

[ SmokeDetector | MS ] Link at beginning of answer (35): How to compare differences between directories (linux) by a deleted user on serverfault.com

fp- feedback received on How to write a message from a console app opened from a windows app c++98 [MS]

@ThomasWard Should be easy

[ SmokeDetector | MS ] Offensive answer detected, potentially bad keyword in answer (79): Area bounded by sin(x)sin(y)=k by asp211's mother on math.SE

tpu- feedback received

Is it possible to make most of MS pages static?

No?

@user12986714 no

it's all generated via Rails AFAICT

and a lot of them have dynamic content, etc. so they can't be static

Yeah, I think it would be possible with a lot of javascript

it would, but that's a lot extra JS and would make some other headaches

But for the most part every page on MS is dynamic to some degree

Yeah, but it does not make sense to, for example, perform db queries for a list of posts related to a specific domain.

Why do you think that doesn't make sense? I'm a little confused.

That would be a lot of queries; it can be easily generated/updated each time a new Post object is created

And the query generally takes a lot of time

Oh, I see what you're saying

Yes, there is a way to make that faster without fully static content

But it gets complicated fast

Or... perhaps cache the page generated so no need to regenerate until a new Post is created by SmokeDetector

Yep, I did some similar work caching various parts of MS via redis

The problem is that a cache can be kind of brittle

so I guess something like modified=false; after_create: modified=true; retrieve(): if modified: generate() else display(cache.html)

Basically cache + lazy eval

So, for example, if we needed to redact a post from the MS database, somehow we'd also need to remember to remove it from every domain tag cache it was a part of.

Yeah, but the static pages will be way faster (likely ~0.1s vs ~10s on my side)

A static page and a fully cached paged should be about the same speed

True; almost equally faster than generate every page from scratch

The same applies to check historical accuracy part; we only need to do the diff every time a new post is created, not from scratch every time the check is performed

The problem is, if you were to regenerate every page that has a list of posts on it every time a new post hits MS, adding a new post would become incredibly expensive

fp- feedback received on How to compare differences between directories (linux) [MS]

@thesecretmaster Lazy eval

Just have a "modified" variable and set it upon new post creation

Well, sure, kind of

fp- feedback received on Firstly, Spider ID? Also possible relation to ~100+ serious bite reactions in past 1-2yrs [MS]

First of all, managing modified flags on every single page is non-trivial, and second of all, let's say we get a new post every 2 minutes. Unless all our pages are getting hits more than once every 2 minutes, there's no gain.

When a new post is created, the majority of pages do not need to be changed

naa- feedback received on Undo command in Minecraft [MS]

Can you give me an example of some pages that you think would benefit from this system then?

Sorry, I think I'm mostly following what you're saying, but not entirely

And it would be useful to talk about some specific examples

@thesecretmaster mainly https://metasmoke.erwaysoftware.com/(?:domains|reasons)/*

naa- feedback received on Can sex with a prostitute be permissible if they become a temporary slave? [MS]

Ooooo can we talk about reasons?

I did a big thing where I tried to make reasons faster a while back

It was successful, to an extent.

[ SmokeDetector | MS ] Link at beginning of answer (35): How can I make a contact us page ? but by Nischal Malla on stackoverflow.com

The idea beside a cache is to do the diff rather than from scratch; basically if a new post with reason X is created, we only need to add the link to X, rather than to do another query of all posts with reason X

naa- feedback received

@user12986714 Also /spammers/

@user12986714 Yep, IIRC I mostly did that, give me a second to grab the code and explain why it actually wasn't that effective.

Does this link work for you? metasmoke.erwaysoftware.com/mini-profiler-resources/…

Nope

:/ Ok, give me a sec to screencap + remember what I was thinking when I wrote this

So, this is the breakdown of timings for my load of /reasons on production

0.1% of the time on that page was spent on SQL queries, about 2ms, total.

For a practical example of cache + diff, say for a given post, it can be cached on disk until it receives a new feedback/comment, at which point we take ~0.1s to add the feedback/comment to the on disk cache, then we can expect any of /post/* to load in ~0.1s rather than 2s

@thesecretmaster That would be great

That's exactly what's happening, on production, now

The majority of that time is actually spent checking caches

2.2 out of the 2.4 seconds are spent in "Redis queries"

Which are queries to redis, the engine we use for cacheing

The problem is that the cache has to be kicked super super often and so is very granular to make sure that those kicks don't become too expensive

It has to be kicked every time a feedback is added

[ SmokeDetector | MS ] Phone number detected in title (65): Were the 1400s-1600s really not part of the "Middle Ages"? by Adesh on history.SE

fp- feedback received

Although, actually now that I look at this code again, I could prolly speed it up a bit

@thesecretmaster was this taken before the move to my infra? Or after? Also, does this reflect any kind of SSD data?

Mon, 13 Jul 2020 23:30:10 GMT

ah so today then

cool

SSDs don't impact redis much, since all its data is in memory

(although it does also persist to disk)

@thesecretmaster It still appears to be a liiitle bit expensive to spend ~2s rendering each /post/* page from redis though

@thesecretmaster the NVRAM occasionally gets cached to disk by VMware for some odd reason so :P

Yes, it is. Faster than querying a DB was, but still quite slow

That was the trade off I came up with to minimize the cost of adding new feedbacks while speeding up /reasons

So... I think there might be some way to cache the rendered page or something and directly serve it, which should be faster, because it is still somewhat inefficient to generate from redis every time even if the page is not changed.

Yes, that's true

Let's look at the idea of setting a modified flag every time a new feedback is added.

And then re-rendering if the modified flag is set

And potentially save the rendered page to the disk

Yep of course

Give me one sec to grab some data for you

So, in the last 24 hours, we've gotten about 400 feedbacks. That averages out to about 1 every 5 minutes.

That means that your caching strategy would only save computation if that page gets access more than once every 5 minutes.

A while back I stuck some code in MS to log how frequently pages got hit

Let me see if it's still around

Ah nope looks gone

But I would be very surprised if reasons got hit that frequently

So, in my view, it makes sense to try cache in a more fine-grained way, so that you can speed up the page even for people who don't luckily end up in the right 5 minute interval.

Now, that's not to say they're mutually exclusive. We could absolutely do both, and get the benefits of both.

But in my view, the benefits of caching the whole page are near zero because we'd be rebuilding the cached page almost every time it gets hit.

@thesecretmaster So the idea is not to rebuild from scratch, but to only build the diff

So how are you defining the diff?

Like instead of making another query about all post wrt. a specific reason, do load_from_disk(); post_list.append(new_post); save_to_disk()

So, for /reasons, only recalculate the reasons that have had feedbacks on posts in them since the last load?

Yes; same for /domain and /post; and potentially also /posts

@Daniil OK. np. Thanks for investigating further.

Let's stick with /reasons, since talking about one page makes this a bit easier

So, how would we determine which reasons need to be recalculated?

@thesecretmaster Not to recalculate reasons, but to, say for a reason X, append a new post to X.post_list rather than X.post_list = sql_query("blah")

So we're persisting post_list to disk for each reason?

@thesecretmaster For every new post there is a list of reasons; then for every reason in reasons do reasons/X -> post_list.append(new_post)

Yes

Ok, cool

As disk io is faster than db query

@user12986714 Yes, agreed, although as a side note, a redis query is usually faster then disk since redis lives in memory :P

(depending on network latencies, but it very frequently is)

Well, if feedbacks (& comments?) are really the showstoppers, it's fairly easy to have JavaScript render those. We already have JavaScript that renders new ones (although there's at least one bug). Even if the JS is fetching the whole feedback info from the API and rendering it, that's only between 150 to 350ms.

So, we have all these post lists on disk for each reason. Then how do we calculate the bars at the right of the reason list for each reasons tp/fp/naa rate? We have to go to the DB, no?

@thesecretmaster Upon updating of a reason's post_list, do either reasonX->tp_count++ or reasonX->fp_count++ or others, and calculate from reasonX->*_count