Charcoal HQ

12:00 PM

We do a lot of mostly-invisible stuff, and when we do something big and visible it's often like this meta post where we deal with a bunch of criticism too. Figured it'd be worth a reminder that y'all are awesome useful people doing awesome useful work :)

SmokeDetector

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, bad keyword in username, blacklisted website in body, pattern-matching website in body: Revolyn Ultra: Does this work effectively to Loss Weight? by Revolyn Ultra on superuser.com

M.A.R.

@SmokeDetector k

Ferrybig

sd k

M.A.R.

We're the cool and nice league of shadows

Unfortunately, I forgot my assassin's cowl

SmokeDetector

tpu- by tripleee

Mithrandir

12:02 PM

Assassin cows?

Henders

@ArtOfCode In which case, I hope you sent yourself one too.

2

ArtOfCode

heh

M.A.R.

@Henders but is he an awesome useful person doing awesome useful work :)?

All he does is say aye

And wreck Smokey

Everything else is probably Smokey telling him what to do

Henders

shrug

M.A.R.

Essentially turning every one of us into puppets

And contributing to Illuminati's world dominance

@Henders wake up, Sheeple

Henders

12:08 PM

-1 point. I expected you to reply with gurhs

A J

@Henders yhw?

M.A.R.

@Henders since someone stole that from me, I boycotted it

Henders

@AJ +1 point.

SmokeDetector

[ SmokeDetector | MS ] Pattern-matching website in answer: Billing module (Fee) for School Management Software by Sobia Waqar on stackoverflow.com

Henders

@M.A.R. ti dettocyob I, em mor.... Bah, I can't be bothered :P

SmokeDetector

12:10 PM

[ SmokeDetector | MS ] Bad NS for domain in body: Site Indexed but Not Showing When I Search Using Exact Domain Name by Muhibul Haque on webmasters.SE

tripleee

@M.A.R. wrong

M.A.R.

@tripleee that's just him mispronouncing "aye"

@Henders you're not a true master

I am, for I am even reversing the reversed

Henders

@tripleee Maybe I over-use this emoticon

SmokeDetector

[ SmokeDetector | MS ] Bad keyword in body: nutritionextract com/testo-ram-uk/ by zkiorbajva on askubuntu.com

tpu- by Tinkeringbell

[ SmokeDetector | MS ] Manually reported answer: No module named 'x'; 'x' is not a package, using tkinter in python by Callum on stackoverflow.com

tpu- by suraj

Mithrandir

@Henders You've used it ~1,000 times all time.

I've used it ~3,200 times all time.

Henders

12:18 PM

So just over 12% of all of my messages include it.

Yours is like 4% usage

SmokeDetector

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, link following arrow in body, +2 more: Tell Me More About provitazol? by Saked 19 on drupal.SE

Mithrandir

You need to chat more ;)

SmokeDetector

tpu- by suraj

M.A.R.

@Henders oh wow, you're really shy

I was expecting 80k chat messages

Henders

I tend to read rather than write :P

Magisch

12:30 PM

@ArtOfCode nice work

Henders

Oh! and look another smiley

Magisch

tis the perpetual smiley

rene

@M.A.R. You're not shy at all, right?

SmokeDetector

[ SmokeDetector | MS ] Repeating characters in answer: Convert string to binary in python by gabagaba on stackoverflow.com

JAD

@SmokeDetector rude

SmokeDetector

12:33 PM

New metasmoke user 'Penn' created

Hello, @rene :)

still alive

@AJ gone

@Henders \o

@rene I trust you're blooming?

Federico

12:35 PM

!!/report aviation.stackexchange.com/questions/49268/…

SmokeDetector

[ SmokeDetector | MS ] Manually reported question: PETROLEUM PRODUCTS ON FOB CI DIP AND PAY, CIF AND TTM PROCEDURE by Maksim Yaroslav on aviation.SE (@Federico)

tpu- by WELZ

A J

@SmokeDetector huh? already gone before I could hit Flag button.

Federico

@AJ yeah, I did not want to wait :P

A J

@Federico lol. no problem

WELZ

@SmokeDetector Why wasn't that caught automatically?

SmokeDetector

12:36 PM

@WELZ Post manually reported by user Federico in room Charcoal HQ.

A J

lol

rene

@Henders as never before

tripleee

!!/watch (?:7\W*)?926\W*503\W*65\W*51

SmokeDetector

@tripleee Added (?:7\W*)?926\W*503\W*65\W*51 to watchlist

DavidPostill

!!/report superuser.com/a/1301227/337631

SmokeDetector

12:40 PM

[ SmokeDetector | MS ] Manually reported answer: I can't find the right driver for ASUS laptop webcam on Windows 7 x64 by steve on superuser.com

Federico

!!/test neftegazconsultant@yandex.ru

SmokeDetector

> Potentially bad keyword in body, potentially bad keyword in title, potentially bad keyword in username
----------
Title - Position 1-29: neftegazconsultant@yandex.ru
Body - Position 1-29: neftegazconsultant@yandex.ru
Username - Position 1-29: neftegazconsultant@yandex.ru

Federico

@WELZ it should have been caught, but maybe I was too quick (2 minutes between posting and my report)

SmokeDetector

CI on 8737b91 succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev 8737b91 (SmokeDetector: Auto watch of (?:7\W)?926\W*503\W*65\W*51 by tripleee --autopull*) (running on teward/Lunar Eclipse)

Restart: API quota is 16059.

Mithrandir

@Federico next time, try !!/scan

WELZ

12:43 PM

!!/watch briskrange\.com

SmokeDetector

@WELZ Added briskrange\.com to watchlist

Federico

@Mithrandir I did earlier today, but force of habit takes over when in a rush :P I'll try to remember it next time

SmokeDetector

CI on aedb870 succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev aedb870 (SmokeDetector: Auto watch of briskrange\.com by WELZ --autopull) (running on teward/Lunar Eclipse)

Restart: API quota is 16027.

WELZ

!!/watch threadsphere\.bid

SmokeDetector

@WELZ Added threadsphere\.bid to watchlist

CI on f8bc01d succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev f8bc01d (SmokeDetector: Auto watch of threadsphere\.bid by WELZ --autopull) (running on teward/Lunar Eclipse)

Restart: API quota is 16002.

New metasmoke user 'Mitja' created

Tschallacka

12:56 PM

we stuff em chock full of bullets

DavidPostill

@thesecretmaster > There have been 656 attempts to cast flags automatically on Stack Overflow in the last 3 months.
We were able to cast automatic flags 653 times (99.54%) and we were unable to cast automatic flags 3 times (0.46%).
The flags were 622 (95.25%) true positives (TPs) and 22 (3.37%) false positives (FPs).

622+22=644, what happened to the other 12 flags?

Tschallacka

95.25 + 3.37 = 98.62

this mystery must be uncovered.

DavidPostill

And There have been 1 688 attempts to cast flags automatically on Super User in the last 3 months.
We were able to cast automatic flags 1 682 times (99.64%) and we were unable to cast automatic flags 6 times (0.36%).
The flags were 1 681 (99.94%) true positives (TPs) and 3 (0.18%) false positives (FPs).

1681+3=1684 but we only cast 1682 flags!

Tschallacka

99.94 + 0.18 = 100.12

math just exploded. it became more than the whole

WELZ

@Tschallacka rounding?

Tschallacka

1:05 PM

I chalck it up to referring to a wrong field in excel

PeterJ

Not 100% sure but think the unable to cast might refer to individual flags - so an individual post that should have been flagged might have 3 unable to flags per post.

Mad Scientist

metasmoke.erwaysoftware.com/post/62958

Mithrandir

@DavidPostill conflicting feedback?

Mad Scientist

I'm wondering why this one was autoflagged. It doesn't seem like it should, especially as the two reasons are strongly correlated

DavidPostill

@Mithrandir Hmm. Could be ...

Mithrandir

1:13 PM

metasmoke.erwaysoftware.com/search?utf8=✓&title=&body=&username=&why=&si‌te=3&post_type=&feedback=conflicted&autoflagged=Yes&reason=&user_rep_direction=>%‌3D&user_reputation=0&commit=Search

SmokeDetector

[ SmokeDetector | MS ] URL-only title, bad NS for domain in body, bad NS for domain in title, blacklisted website in body, blacklisted website in title, +4 more: healthpurelives.com/nitridex/ by gtukrkwmx on askubuntu.com

doppelgreener

@SmokeDetector k

SmokeDetector

tpu- by Floern

Loong

@MadScientist "Palmitate" might not be a useful bad keyword on Chemistry.

Mithrandir

@MadScientist some people cough had their settings a little low

Federico

1:16 PM

example of why palmitate is on the list: metasmoke.erwaysoftware.com/post/62278

Henders

@Mithrandir But that has now been resolved by enforcing a minimum value for the setting, right?

Mithrandir

(I think I changed my settings after that one.)

@Henders well...

Federico

@Henders minimum value was already enforced. It has been raised

Mithrandir

Click "eligible flaggers"

Magisch

pretty sure 197 would be flagged by me

Henders

1:18 PM

@Mithrandir Ah, thanks

Magisch

That's already 99,75% though

tripleee

!!/blacklist-keyword nitridex

Henders

@Federico I hadn't realised that there was a minimum figure before it was increased.

SmokeDetector

@tripleee Blacklisted nitridex

Loong

@Federico I see. However, some examples why it probably shouldn't be autoflagged on Chemistry: chemistry.stackexchange.com/search?q=palmitate

Mithrandir

1:19 PM

@Henders it was 99.50%, and it got raised to 99.75%

Mad Scientist

these rules are essentially boiled down to the presence of a single word, a legitimate chemical name. That's far too aggressive in my opinion

Henders

@Mithrandir Ah I see, thanks

JAD

@MadScientist the single word isn't enough to trigger autoflagging (anymore) though

Federico

@Loong and out of all of those, the post in question is the only one that got detected by smokey

Mithrandir

@Loong we can probably tweak the code to not detect it on Chemistry.SE

Magisch

1:20 PM

@MadScientist Yeah, it could never be a candidate for 5 flag autoflagging unless it hits 4 filters at once

So this is just one of the 0,25% of posts that got hit with FP 3 flags

SmokeDetector

CI on 47dc2e8 succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev 47dc2e8 (SmokeDetector: Auto blacklist of nitridex by tripleee --autopull) (running on teward/Lunar Eclipse)

Restart: API quota is 15676.

terdon

@Magisch The problem is that your stats are network wide. On Chemistry, the stats are far far worse. That's one of the problems of looking at this as a network-wide issue: you loose granularity.

Magisch

@terdon Yeah we can take it out for chemistry

We already do that for several dozen other things

like "muscle" on health.SE

etc

We do fine tune keywords just for specific sites due to content

Henders

@terdon Agreed for chemistry

Federico

@terdon we need a person from statistics, because I am pretty sure that the sample size for Chemistry and Aviation is not significative enough

Mad Scientist

1:22 PM

@JAD it's not very far away from the new autoflagging threshold as far as I understand. And it's not a reasonable false positive, this indicates a fundamental issue with these particular rules and how they're applied on certain sites

Mithrandir

github.com/Charcoal-SE/SmokeDetector/blob/master/…

Magisch

@MadScientist 196 vs 300 is quite a ways away

>For one, 300 is unreachable unless it trips 4 reasons at once rather then 2

So single keyword FPs can never ever be eligible for 5 flag autoflagging

Mithrandir

And yeah, some of our regexes are a bit too greedy

Mad Scientist

do multiple different keywords for the same rule count as different rules here?

Mithrandir

@MadScientist nope

Magisch

1:24 PM

@MadScientist no

"bad keyword" is a rule

for title and body they count seperately, but keywords alone will never make a post eligible for 5 flags autoflagging

JAD

@Federico uh, there are plenty enough data points to tell that chemistry more fps than 99.5%

terdon

@Federico It isn't, that's the problem. That's why it's hard to use network-wide stats for the smaller sites.

Federico

@JAD 14 data points is significant? that's why I am invoking a statistician, I personally would not know

JAD

hmm, what filter are we looking at exactly? I was looking at the 300 odd total reports on chemSE

terdon

@Federico You can't say from that. You would also need to take into account the full number of posts scanned. Which can be done, of course.

Mad Scientist

1:26 PM

I wouldn't even argue statistics here. Rules based on chemical or biological terms simply should never run on sites where those topics are on-topic. They're just harmful there

Mithrandir

@JAD autoflags

terdon

(and no, I'm not a statistician, I've just been bitten by enough of them that I've caught a little of what they have)

Federico

@JAD the autoflagged ones only

terdon

@MadScientist Yes.

Mithrandir

@MadScientist Sure, we can exempt certain sites from keywords.

We could also downgrade Palmitate to a potentially bad keyword (reason locked at 1 weight).

JAD

1:28 PM

Hmm, so there you have a 95% CI between 72.7% and 98.6% TP rate

which is wide, that's for sure

Federico

@MadScientist to add to the other nopes, you can see it in the example you linked: metasmoke.erwaysoftware.com/post/62958 194 = 97 + 97 (one per rule, even if the second one got triggered twice)

Mithrandir

"alpha male" is also blacklisted, which IIRC caused an autoflag FP once.

Mad Scientist

@Federico that would also happen if the keywords are different?

terdon

@Mithrandir Yes, on biology.

On a crappy, off topic question, mind you, but not spam.

JAD

@MadScientist well, if blacklist in title triggers on another word than blacklist in body, sure

Magisch

1:30 PM

@MadScientist depends. Some chemical sounding supplement names should be blacklisted on say chemistry.se

Federico

@MadScientist metasmoke.erwaysoftware.com/post/109284

Magisch

since they're not actual chemicals, just products used in spam

Mithrandir

Actually, twice on Biology.SE

MS search

terdon

@Magisch There can also be valid questions though: "What does EnhanceYourStamina® actually contain? The label states: blah blah" or something

Magisch

@terdon cross that bridge when we get there

Mad Scientist

1:32 PM

@terdon that's something that could e.g. happen on Skeptics

Magisch

don't think random product names of uncommon alex jones tier supplements are ever going to be interesting enough for that

Mithrandir

@terdon and unless they link to the site also there's no chance of it triggering 5 flags

terdon

@MadScientist Yes, there too.

Magisch

@MadScientist Health site exemptions include skeptics.SE

ref: github.com/Charcoal-SE/SmokeDetector/blob/master/…

Mithrandir

Wait, why is Robotics.SE a health site? O_o

Magisch

1:34 PM

Because some health related keywords come up on robotics

Mithrandir

Smokey must've stuck it in there.

JAD

@Magisch like steroids for robots

Hmm, I wonder what our false negative rate is due to the exception for health sites on some keywords

WELZ

@tripleee pictures, or it never happened.

tripleee

@WELZ they're smaller than I expected from other pictures (this is a 15" laptop)

WELZ

[awkwardly-waits]

tripleee

1:40 PM

2

WELZ

they do look slightly smaller.

terdon

@tripleee Oh hey, ultimate spam protection! Just stick one of these on your screen and you'll never see spam again!

WELZ

Lmao

M.A.R.

Chem has more than enough users. Death to Charcoal

Tschallacka

Hack the world!

SmokeDetector

1:51 PM

New metasmoke user 'James Hughes' created

M.A.R.

@Mithrandir my impression is Bio is more FP prone than us

OTOH, most Chem spam hasn't be caught by Smokey, I think

They're often too weird

Tschallacka

Excelent glass quality beakers for sale, buy now at excelentglassbeakersnotascam.com

M.A.R.

I wish

It's usually alloys. Some idiot thinking chemists sit on platinum so they can buy gold

Or somesuch

Federico

@M.A.R. Smokey was not catching fuel "resellers" on aviation until I started reporting them here. Now they get caught ;)

M.A.R.

@Federico hah, fuel resellers? They sell water vapor and carbon dioxide?

Federico

1:58 PM

@M.A.R. -_-'

JAD

@M.A.R. if only

Federico

!!/tea

SmokeDetector

@Federico brews a cup of green tea for @Federico

Federico

!!/test dentoaviation.com

SmokeDetector

> Potentially bad keyword in body, potentially bad keyword in title, potentially bad keyword in username
----------
Title - Position 1-18: dentoaviation.com
Body - Position 1-18: dentoaviation.com
Username - Position 1-18: dentoaviation.com

Federico

2:01 PM

ok, it got added, I forgot

M.A.R.

!!/teast

SmokeDetector

@M.A.R. No such command 'teast'.

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, pattern-matching website in body, repeated URL at end of long post: Andras Fiber Reviews by robertmazon on astronomy.SE

M.A.R.

@SmokeDetector I miss the times when a poor soul commanded something that wasn't a command and you just stood silent, mocking and giggling

@SmokeDetector K

SmokeDetector

New metasmoke user 'Eldho' created

M.A.R.

@SmokeDetector is it just me, Or are we getting more users than yesterday?

WELZ

2:06 PM

@M.A.R. both

M.A.R.

@WELZ I am the lone supervisor

SmokeDetector

[ SmokeDetector | MS ] URL in title, bad NS for domain in body, bad NS for domain in title, bad keyword in body, bad keyword in title, +5 more: healthpurelives.com/nitridex/ by user279040 on apple.SE

tpu- by WELZ

[ SmokeDetector | MS ] Few unique characters in answer, repeating characters in answer: How is centripetal force related to radius? by fahad ali on physics.SE

JAD

@SmokeDetector k

M.A.R.

@SmokeDetector K

SmokeDetector

2:27 PM

New metasmoke user 'Matthieu M.' created

Suraj Rao

Is this spam? User has one more answer linking to one of the urls.

> You can read an eye-popping comparison between Twilio & TheTexting

marketing terms at play

SmokeDetector

[ SmokeDetector | MS ] Bad pattern in URL answer, pattern-matching website in answer: skullcandy earbuds with mic by nidhi on android.SE (@iBug)

[ SmokeDetector | MS ] Offensive answer detected: How can I bypass the Oauth2 authentication with predefined tokens in Spring Boot? by Faggot on stackoverflow.com

tpu- by Magisch

tpu- by Tinkeringbell on skullcandy earbuds with mic [MS]

WELZ

2:47 PM

@angussidney oh yes, all kinda fp come up with them

SmokeDetector

[ SmokeDetector | MS ] Bad keyword with email in body: Can "we'll get back to you" be an independent clause? by Stacy on english.SE

[ SmokeDetector | MS ] Few unique characters in body, repeating characters in body: C - function that compresses characters by George P. on stackoverflow.com

tp- by WELZ

fp- by WELZ on Can "we'll get back to you" be an independent clause? [MS]

M.A.R.

@SmokeDetector F

@SurajRao very suspicious. Can you ask a mod like Bhargav?

Cerbrus

3:04 PM

@SurajRao Looks like spam to me. 2 answers promoting the same product

SmokeDetector

3:16 PM

[ SmokeDetector | MS ] Bad keyword with a link in answer, bad pattern in URL answer: Android - How to run full method before triggering TextWatcher? by anderson on stackoverflow.com

Undo

@Seth There's nothing there anyway IIRC

Or at least nothing anyone here would care about

@Mithrandir Nothing, because I'd be watching it and shut it off when I leave.

Seth

@Undo true

Undo

@Henders 3 gigabytes. Ya'll have been busy :P

Henders

Wow :o

M.A.R.

@Undo Hooooowly . . .

Undo

3:20 PM

I have no idea why s3cmd doesn't work in a cron job.

> [part 2 of 200, 15MB]

Mad Scientist

@Undo different environment? There could be error messages in the cron log that help.

Undo

@MadScientist That's what I was thinking. I found the cron log - no error messages, and modifying the job to print s3cmd's config shows that it's identical to that of the CLI environment.

Mad Scientist

I assume all paths are absolute in the cron job?

Undo

Should be, let me check

#!/bin/bash

HOME=/home/ubuntu

bucket="s3://erwaysoftware.mysqlbackups"

# Timestamp (sortable AND readable)
stamp=`date +"%s - %A %d %B %Y @ %H%M"`

for filename in /var/railsapps/*/shared/log/production.log; do
	APPNAME=`echo $filename | cut -d \/ -f 4`
	echo Backing up $APPNAME

	object="$bucket/$stamp/$APPNAME.log.gz"

	gzip -c $filename | s3cmd put - "$object" --multipart-chunk-size-mb=100
	s3cmd --dump-config
	echo Done
	# rm $filename
done

terdon

Yeah, use the full path to s3cmd

SmokeDetector

3:27 PM

[ SmokeDetector | MS ] Bad keyword in body, bad keyword in title, blacklisted website in body, link at end of body, link following arrow in body, +1 more: Working of Total Radiance Teeth Whitening Pen Teeth Whitening Kit by Thicest 31 on drupal.SE

terdon

Cron uses its own (hardcoded) PATH

SmokeDetector

tpu- by K.Dᴀᴠɪs

terdon

You may as well quote $filename while you're at it, this will break on file names with spaces.

Undo

@terdon Will try it, but it's working with --dump-config so it definitely finds the binary

K.Dᴀᴠɪs

is MS down?

Undo

3:28 PM

Good point. No spaces to the path right now, but good fix.

@K.Dᴀᴠɪs probably

terdon

@Undo Oh. That's odd.

SmokeDetector

New metasmoke user 'anothernode' created

terdon

Add a 2> /tmp/some.log to the cron line and see if that gives you anything useful.

Undo

Do I have to touch that file first, or will it auto-create?

177/200, almost there

terdon

It will autocreate

Another possibility is that it doesn't find gzip. It should, IRC cron uses /bin and /usr/bin as its path but you never know.

Undo

3:30 PM

This'd probably be way faster in 500MB chunks - EC2 has a stable connection to S3, I would hope.

which gzip
/bin/gzip

Will concrete-ify that one, too

terdon

I'd asked a question about this, I was really surprised that it was hardcoded: unix.stackexchange.com/q/401621/22222

WELZ

@Undo yep, it's down alright.

Undo

It is right now, I'm doing some heavy stuff

Rebooting, then should be all happy

@terdon Wow, that's a little strange.

Okay, updated job:

#!/bin/bash

HOME=/home/ubuntu

bucket="s3://erwaysoftware.mysqlbackups"

# Timestamp (sortable AND readable)
stamp=`date +"%s - %A %d %B %Y @ %H%M"`

for filename in /var/railsapps/*/shared/log/production.log; do
	APPNAME=`echo $filename | cut -d \/ -f 4`
	echo Backing up $APPNAME

	object="$bucket/$stamp/$APPNAME.log.gz"

	/bin/gzip -c $filename | /usr/local/bin/s3cmd put - "$object" --multipart-chunk-size-mb=100
	/usr/local/bin/s3cmd --dump-config
	echo Done
	# rm $filename
done

terdon

@Undo Innit though?

Undo

And crontab line:

03 20 * * * /bin/bash /home/ubuntu/rails-log-backup/run.sh 2> /tmp/railslogbackup.log

Verified that running it with sh run.sh works

terdon

3:36 PM

Looks about right.

SmokeDetector

[ SmokeDetector | MS ] Potentially bad keyword in title: Could hacker change data in one smart contract with out call contract function? by Yudao Yan on stackoverflow.com

Undo

Adjusted cron to run in the next minute

Mithrandir

!!/watch bubbly\W?hub

SmokeDetector

@Mithrandir Added bubbly\W?hub to watchlist

terdon

@Undo That's running it with sh (dash on Ubuntu), not bash. Should not be relevant here, I don't see any bashishms, but might be an issue at other times.

SmokeDetector

3:37 PM

tpu- by xfix

Mithrandir

Spammed and immediately self-deleted, added based on info from other mods

Undo

cat /tmp/railslogbackup.log
gzip: /var/railsapps/redunda/shared/log/production.log: file size changed while zipping
gzip: /var/railsapps/sentinel/shared/log/production.log: file size changed while zipping

well then

terdon

Ah! Nice.

Yay for informative error messages!

SmokeDetector

CI on 84c9cfd succeeded. Message contains 'autopull', pulling...

[ SmokeDetector ] SmokeDetector started at rev 84c9cfd (SmokeDetector: Auto watch of bubbly\W?hub by Mithrandir --autopull) (running on teward/Lunar Eclipse)

Undo

Do I just have to copy them before gzipping, or is there some other way to avoid that error?

doppelgreener

3:38 PM

@Mithrandir that one hit RPG.se too. Was just coming here to watchlist it. Thanks.

SmokeDetector

Restart: API quota is 14074.

Undo

Those logs should be append-only.

Ferrybig

I always add

export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ga‌mes:/usr/games:/bin/did-you-really-scan-for-every-path-in-this-chat/bin"

on top of my cron scripts, just to avoid PATH issues with my cron jobs

terdon

@Undo Obviously, the real fix would be to make sure nothing's writing to /var/railsapps/redunda/shared/log/production.log, but if that's not an option, try cat "$filename" | gzip -c | ...

Henders

Bah, I hate debugging cron issues. Fix for 20 seconds then just sit there waiting for ~40 seconds until the next one runs.

Undo

3:39 PM

@Henders or type like mad to get the right numbers in the next five seconds

Mithrandir

> did-you-really-scan-for-every-path-in-this-chat

Ferrybig

@Undo Usually, logrotate handles these issues by first moving the log files to a new place, then reloading the program, and on the next rotation, apply gzip, as no problem should write to a log file for 24 hour non-stop

Undo

#!/bin/bash

HOME=/home/ubuntu

bucket="s3://erwaysoftware.mysqlbackups"

# Timestamp (sortable AND readable)
stamp=`date +"%s - %A %d %B %Y @ %H%M"`

for filename in /var/railsapps/*/shared/log/production.log; do
	APPNAME=`echo $filename | cut -d \/ -f 4`
	echo Backing up $APPNAME

	object="$bucket/$stamp/$APPNAME.log.gz"

	cat "$filename" | /bin/gzip -c | /usr/local/bin/s3cmd put - "$object" --multipart-chunk-size-mb=100
	/usr/local/bin/s3cmd --dump-config
	echo Done
	# rm $filename
done

There we go

terdon

Works?

Undo

Looks like it

Pointing to the right bucket, we'll see if it still works

terdon

3:43 PM

Logrotate and dumping the latest log as @Ferrybig suggested would be more elegant, but hey.

Undo

There we go. Awesome

Thanks @terdon @Ferrybig