The Comms Room - 2014-02-14 (page 5 of 5)

« first day (1252 days earlier) ← previous day next day → last day (3744 days later) »

10:00 PM

Their stats aren't bad

283

Network Engineering – networkengineering.stackexchange.com

Beta Q&A site for network engineers

Currently in public beta.

@DennisKaarsemaker likewise. Of course if it gets love from canonical staff like @iain says then it's still a good thing maybe. I do wonder what the response would be if they launched a Microsoft-sponsored "ask Microsoft"

AU had to be different, since all Android did was sponsor an SO tag

Dennis Kaarsemaker

ok, I'm out of here for the night, need to get some sleep: moving lots of furniture 6 floors down tomorrow

@MDMoore313 Yeah, I had a similar Meta question about that...

5

Q: Will/Should Serverfault be dropping the "networking support" in the upcoming future?

There is a currently private beta site for Network Engineering here: http://area51.stackexchange.com/proposals/52519/network-engineering The beta is new, but seems to be garnering a decent following so far, and will probably ramp up participation when it becomes a public beta site. I think thi...

@MDMoore313 So you're saying it was all about Mark Shuttleworth's ego?

I sure do....

10:03 PM

on that note, night all.

'noight.

@Magellan ...who became the first citizen of an independent African country to travel to space... I think that sums it up.

l8r folks

@RobM cya mate

@Magellan Then immediately stashed them for another 8 months? :)

@JoelESalas Normally, that would be about right. It's been unusually dry here too though. We only have half our normal snowpack. Not as dry as down there, of course, but it never is here.

10:09 PM

I want to live in Seattle for a few months

nordstrams is there

@JoelESalas Was out at Fashion City while I was there. I really vastly preferred OC to here.

Wandered around on Balboa Island with Lady and my MIL whilst they reminisced.

This just doesn't seem right...

user image

I almost want to say eff-it and put SSDs on every customer's server.

@Magellan Orange County is not my cup of tea

@JoelESalas It's 'hood

@JoelESalas I may have been biased because Lady grew up there and it was the first sunlight I'd seen in a month.

10:14 PM

Orange County is what poor people from the Midwest think is "nice"

Sad but true.

You got your Cheesecake Factory, you got your PF Changs, what more do you want

big 10 lane streets, traffic for days, not a decent cup of coffee for 40 miles

@JoelESalas Red Lobster Cheese Biscuits.

@JoelESalas Got those in every other city too.

@ewwhite Never going to outgrow those

10:15 PM

@JoelESalas we bought some at Sam's Club.

@ewwhite Oh god I didn't know they sold them

@ewwhite Whats all the writes on?

user image

@JoelESalas Well, if you do come visit, do it in the summer. Most folks from SoCal completely lose their shit after 2 straight weeks of rain and a cloud deck 200 feet over their head.

@MatthewIfe Flat-file database solution that runs the ERP application I support for my Produce industry clients. 100% disk-busy right now on a array of 6 x 300GB 10k SAS drives in RAID 1+0 with flash-backed cache, etc.

10:19 PM

How big is the database?

that still seems like quite a bit of write ops for a flat file... Someone importing or indexing or something?

@MatthewIfe ~300GB of real data...

Whats the open flags that it uses to write?

You can get this by doing fuser /path/to/flat/file, get the pid, do lsof -p <pid>, get the fd, then `cat /proc/<pid>/fdinfo/<fd>

the language is called DATABUS, so it's an interpreter written in C... checking file flags.

@Magellan I was in Bellevue for a week and I loved it, didn't want to leave

10:22 PM

Theres some work to do to translate the flags btw, just paste the flags value.

If Maria wasn't in LA I might not have

@MatthewIfe lots of files in use...

open("/ppro/wrk/00311101.ISI", O_RDWR|O_CREAT|O_TRUNC, 0666) = 62
open("/ppro/wrk/00311101.TXT", O_RDWR)  = 63
open("/ppro/wrk/00311101.TXT", O_RDWR|O_CREAT|O_TRUNC, 0666) = 63
open("/ppro/wrk/00311101.ISI", O_RDWR)  = 62
open("/ppro/wrk/00311101.TXT", O_RDWR)  = 63
open("/ppro/wrk/00311101.ISI", O_RDWR)  = 62
open("/ppro/wrk/00311101.ISI", O_RDWR|O_CREAT|O_TRUNC, 0666) = 62
open("/ppro/wrk/00311101.TXT", O_RDWR)  = 63
open("/ppro/wrk/00311101.TXT", O_RDWR|O_CREAT|O_TRUNC, 0666) = 63

but right now, it's building files in a work directory...

@ewwhite thats messed up

open("/ppro/wrk/00311101.TXT", O_RDWR) = 63
open("/ppro/wrk/00311101.TXT", O_RDWR|O_CREAT|O_TRUNC, 0666) = 63

opens file, then truncates it

I assume that this is strace -e open or something? IT wouldnt reallocate that fd on the subsequent open otherwise.

Yeah strace -p 4832 -e trace=file.

That isnt lioekly

oops

that isnt likely a problem tbh.

10:27 PM

@MatthewIfe it's running... it's just slow and inefficient.

Those files are being opened, then truncated way before dirty writeback gets a chance to write them in.

can you run pidstat -d 1 10 and see where its io bound?

obviousloy dont paste all the output. Look for anomalies ;)

(I never use pidstat)

use iotop if you want ;)

does /ppro/wrk contain a lot of files?

8,561

meh.

10:31 PM

XFS

double meh ;)

Be glad it's not /ppro/prt -

[root@General /ppro/prt]# ls  | wc -l
48733

pah! I've got some dirs > 2000,000. We always bitch to people who do this.

Thats all nfs/wafl though.

[root@LAS /ppro/prt]# ls -1 | wc -l
472931

Most people push it to 400k-900k

iotop is just showing the wrapper process.

Do the wmb/s tally with your graphs?

10:33 PM

user image

it's very low throughput, but still keeps the logical drive incredibly busy during this process.

Can you strace -e fdatasync -e fsync and see if your syncing to disk

@ewwhite Based off of the strace so far, I dont understand how these hit the disk.

They are all within writeback thresholds.

No output

@JoelESalas Yeah, Bellevue is kind of the antithesis of Seattle though.

Me: I’m so hungry, I’m going to eat the *shit* out of this sandwich. Sandwich Maker Lady: … Me: That, um, did not come out right.

Tweeted by wilw on February 14, 2014 at 10:28 PM

KICK

@ewwhite as a matter of fact

swapin looks pretty bad

10:37 PM

@Magellan How so?

could be just a utlization blip though

Scrambled Eggs with Parmesan, Garlic, and Cumin topped with Tapatio and Sour Cream and two lightly fried tortillas on the side. mmmm

@ewwhite whats your Writeback value in /proc/meminfo say?

@JoelESalas Very Republican town focused on flash and status. Seattle's far more blue-collar with a significant cultural cross-section focused on creativity and sustainability.

@Magellan I don't know what Tapatio is but the rest sounds good

10:39 PM

It's 8 miles across the lake but opposite ends of the political spectrum.

Tapatío hot sauce

__NOTOC__ Tapatío is a hot sauce, produced in Vernon, California, that can be found at many grocery stores in the United States. "Tapatío" is the name given to people from Guadalajara, Jalisco: the company's founders come from Guadalajara. It is exported to Mexico, Canada, Central America and a few countries in Europe. The ingredients, as listed on the product label, are: water, red peppers, salt, spices, garlic, acetic acid, xanthan gum and sodium benzoate as a preservative. Tapatío comes in five sizes: 5, 10, and 32 fl oz., and 1 gallon (148, 296, 946 ml and 3.785 liters), as well as...

@Magellan arriba

@JoelESalas I buy that stuff by the box. Soooo good.

@Magellan So where's the coolest hood?

@MatthewIfe I'll check

Writeback: 0 kB

@JoelESalas Alki along the beach if awesome. Ballard/Fremont are great for the music scene. The night life is awesome up on Capitol Hill above downtown.

Queen Anne and Magnolia are quite nice too and not as far from the downtown core as Fremont/Ballard and much closer to the nice bars and restaurants in Belltown.

10:43 PM

@ewwhite this shouldn't be the case but sysctl -a | grep vm.dirty returns sane (non-zero) values.

background_bytes can be 0 if background_ratio is a value

the values are 'toggled'. IF you set one then the other bcomes inactive and vice versa. Normally ratio is active.

@JoelESalas There's no danger of me moving to LA though. Lady forbids it as she lived down there for 3-4 years after college in the 80s.

Also mount flags for that location dont have 'sync' on or something

# sysctl -a | grep vm.dirty
vm.dirty_background_bytes = 0
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500
vm.dirty_ratio = 40
vm.dirty_background_ratio = 10

@MatthewIfe Mount is... (rw,noatime,logbufs=8,logbsize=256k,nobarrier)

@Magellan I'll give that whole mess a whirl soon

@ewwhite just for sanity, can you do iostat -mtx 1 2

and see if you see utilization high there too/

10:47 PM

@Iain it's amazing. My cooking skills took a huge upturn when I started dating someone who was perfectly okay with trying something new and having it not be perfect.

Time: 17:47:50
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.12    0.75    0.25    0.12    0.00   98.75

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0        0.00    50.00  5.00 697.00     0.03     9.54    27.91     1.84    2.62   1.34  94.20

One of my favorites of Lady's repertoire is her Simon & Garfunkel Omellete: Parsley, Sage, Rosemary, and Thyme.

@Magellan I really enjoy cooking - you have to be prepared to eat your mistakes so you know how to improve them

@Iain I was married for way too many years to someone who hated cooking, wanted someone else to do it, but expected perfection every time. Needless to say, I quit cooking for most of that time.

@ewwhite This looks very strange to me. I'd probably need a more complete strace to work out whats happening there. May need to run -f in case it does clones or something too.

10:51 PM

@MatthewIfe And it just finished for the day.

Based off of what i've seen so far. Idont see how open/write/truncate repetitions against the same file would cause so much i/o.

@Magellan She would have starved here or cooked for herself

It may be journalling but that should all be very healthily sequential.

so iostat shows low utilization now?

@MatthewIfe inefficient i/o routines at the software level.

@Iain Would be better for the entire US if that were the case here too. Way too easy access to boxed food.

10:52 PM

user image

it's a daily thing

but 6 x 10k SAS disks can't keep up, so I may throw hardware at it

@Magellan tomorrow I'm going to bake (bread with a hint of marmite) and cook a couple if Indian recipes

@ewwhite sure, but they should never land to disk with (what straces i've seen).

@Iain nice... how did you learn to cook?

@MatthewIfe Oh, I should add that nothing in this software gets cached.

Well, nothing gets synced either from what I see.

seemingly... I have a horrible hack in order to provide read caching of the working data sets on my newer systems.

10:54 PM

@ewwhite it's something I have done since I was a kid. My parents both cooked/baked and my grandma was a really good baker too

@MatthewIfe it's one of those things I don't fully understand, but I stumbled upon it by accident.

For example. Say I have a file. I write 128M of 'a' to it. Close it Reopen I then truncate it and then rewrite immediately 'a's to it. I do that in a loop.

I should get no disk i/o.

Hmm... the reading is from DATABUS data files and writing to a series of temp work files...

but that's all I could see...

it would be interesting to profile

@MatthewIfe but thanks for your help.

I'll test my assertion properly :)

maybe i'm wrong, never know ;)

@MatthewIfe But you have to take the context of this ancient software into consideration.

11:01 PM

Ancient software that uses a moden kernel.

Kernel controls i/o. Not the software.

I s'pose that's true

@ewwhite mega interesting! If you pass O_TRUNCATE it appears to write to disk!

Opening without this flag makes it never write to disk until writeback time.

@MatthewIfe so what's happening?

@ewwhite I think i see the issue here.

This is ext4 btw, possibly similar in xfs.

If you truncate the file, when you write out the file, the o/s is forced to allocate that region of space from disk - even if it doesnt actually write to it (it probably does tbh though)

once the disk allocation is made, rewriting to the same portion with different data is cheap.

truncation forces the allocation to be dropped and require re-requesting it.

Unix programming... I wish I knew more about... um... stuff.

11:09 PM

tell me, does those files you origihnally showed in the strace still exist?

All gone

Okay, so it's Valen*TIME*!! Wife wants to leave now.

enjoy. I might have an answer for you later, if your still there one final question, what distro and version?

EL5.9.

I have dozens of systems running this software. Finding a solution for servers under stressful conditions or understanding what this software is actually doing would be nice.

11:32 PM

I feel like I'll never find a job I like again

My list of red flags is like 200 items long

all the way from "no backup plan" to "uses github"

ha

what's wrong with github

it's better than bitbucket, or some of the other shit i've seen

@mossy status.github.com/graphs/past_month

three nines, it's total shit and unacceptable

ahh

also they always seem to have an outage at 2PM PST when we're doing a deploy.

That's true. I guess I've never had to rely on it that much.

11:35 PM

also also they throttle too much traffic coming from one IP

so you'll do a git clone and be rockin along at 80kbps

one engineer clocked in at 3 hours cloning one of our repos

not okay

I get it, VCs won't give you the time of day if you're not deploying to heroku and using git and docker and vagrant and all of this nonsense

holy shit.

but we're a big company, post-acquisition, it's time for a grown-up solution

yeah. i've only ever used it for this big pile of crap

github.com/q-voke

@mossy That's you?

@JoelESalas it's a project that I'm going to take over.

fantastic idea for php noobs like me.

11:38 PM

@mossy Oh I see, what's it do?

Oh its PHP.

it's basically a search engine for code.

Fast references.

I tie it in with alfredapp.com

clean

maybe i'll set it up

been lazy lately.

@ewwhite so whats happening is on the filesystem close its being written out to disk, this is because the journal wants to preserve proper ordering of metadata. On ext4 you can disable this behaviour (open/write/close <--write here) using data=writeback, so that the kernel doesnt care about keeping data/metadata in sync (so it can also be dangerous). There is no sufficient option in xfs to do this though.

@MatthewIfe If it's backed by a battery it should be fine though

11:54 PM

Still cant change the write mode in this detail in xfs

Weirdly, this only occurs on truncate. It seems also truncate on open in xfs is actaully expensive.

Isnt on ext4, although you get delays on close without data=writeback

@ewwhite @MatthewIfe this has been fascinating ... if/when you solve this you should write a q&a about it

I think on xfs open is slow but not iobound, close is iobound, so is ext4 on close too, but its got a workaround.

« first day (1252 days earlier) ← previous day next day → last day (3744 days later) »

Transcript for

Feb '1414

The Comms Room

ServerFault's lobby

join this room
about this room

00:00

06:00

12:00

18:00

all times are UTC

site design / logo © 2024 Stack Exchange Inc; legal

mobile