Root Access

bwDraco

00:01

!!/tumbleweed

ChatBot John Cavil

allquixotic

Rubber duck, help me fix my code

Bob

@allquixotic quack

4

allquixotic

I'm using Java 8. So for throughput and avoidance of lock contention across a large number of threads, I have an Action object that contains the time in milliseconds as a long, a String (one of maybe 50 different possible log messages, all of them interned into the string pool), and a context to an outer object (it's a non-static inner class).

Each instance of the outer class has its own private ArrayList<Action> that it queues up log messages into, keeping track of the time and the message for each. The messages are long, but stored extremely efficiently as a pointer since there are only a small number of unique messages.

bwDraco

quack

allquixotic

00:07

I want to periodically have a way to prevent all the threads from updating their internal ArrayList while I take a controller thread, empty all the ArrayLists, do a sort on the Actions by their long millis value, and then toString() all the parameters together with a StringBuilder and write them out to a log file.

bwDraco

Sorry, don't know Java.

allquixotic

During this process each worker thread needs to be blocked somehow so it can't mutate its ArrayList while the controller thread is emptying

Bob

Hm. And obviously you don't want to lock on them.

allquixotic

But I don't want a central lock because of the obvious problem of lock contention and scalability

Bob

@allquixotic Is the timing important? Can this happen at any time?

allquixotic

00:08

this really matters a LOT because I'm going to be using this on an AWS c4.8xlarge with tons of CPUs and lots of threads

the timing is extremely unimportant, as long as it happens before the Action objects that are strongly referenced (can't be GC'ed) take up so much space that the process goes OOM (or even exhibits serious system memory pressure)

I was thinking of tuning it to happen every couple of minutes depending on how intensely the messages are stacking up

Bob

Hmmm.

allquixotic

and a brief process-wide thread pause is perfectly fine for the workload

brief as in up to a few seconds

what I need is consistent throughput during the time when it's not doing that log writing and sorting stuff

and scalability across say 40 vCPUs

Bob

Why can't you just read from each worker one at a time? If you lock on each ArrayList there should only ever be two threads that care about the lock.

allquixotic

hmm... so you're saying have a Lock for each thread, and the only users of each Lock will be the thread itself and the controller thread?

Bob

@allquixotic Yes.

allquixotic

00:11

that might even benefit throughput, since only one thread will be blocked due to log-emptying at any given time

the remaining ones can keep chunking

Bob

I don't know what the performance impact of taking a lock that's usually available will be.

Each thread will waste a bit more time locking on every log action... don't know how severe that would be.

allquixotic

yeah, that's my worry, is frequently taking and releasing the lock...

basically, 99 out of 100 times (or so), when the thread that the Lock "belongs to" calls lock(), it's going to get the lock right away

it's that 1 time when the controller, rarely, has the lock, that it needs to block

I don't know if Java or the underlying mutexes (this is on Linux 4.7) are smart enough to optimize the locking behavior enough to not matter

Bob

tbh I'm more familiar with multithreading in .NET... I can only assume Java offers similar functionality

allquixotic

well Java on Linux uses pthread's mutexes, which boil down to kernel futexes

I think there's even a clever optimization they do where you can assume you get the lock and start working, and only go back if it turns out it's blocked

which will be, frankly, so rare (because the threads do things other than constantly logging in a busy loop) as to be noise

Bob

@allquixotic One way some of the newer .NET 4.0 synchronisation mechanisms work is by busy spinning first.

In the case where there is little contention, this is far more efficient than calling into the kernel.

allquixotic

00:15

I think low-contention locks overall are extremely well optimized on modern Linux and/or Java... I'll have to try it and see, but I don't think it'll really be a deal-breaker

if it turns out that my threads are spending a lot of time in a wait state because of it, I'll just have to throw more threads at it -- which the apparently weird and somewhat broken CFS should be able to at least handle some of the time

i3s.unice.fr/~jplozi/wastedcores/files/extended_talk.pdf

I could be crazy and apply the wastedcores kernel patch, as this is "my" EC2 instance (I'm in charge of it, but company rents it) and there's no limitation on what software I can use to get this done

but probably just going with Amazon Linux 2016.03

Bob

ugh... javadocs annoy me

@allquixotic Could always try a basic lock see how that goes.

Not sure how you'd go about profiling that though.

allquixotic

well, it sounds good right now, and it's definitely an angle to explore; we have some pretty good application performance engineering tools (proprietary) for JVM instrumentation that we could load up if we really find ourselves struggling with this problem

or, worst case, we throw more hardware at the problem, though that would kind of suck

hmm, although one possible nuclear option that might be possible at a low level (not sure if possible in Java without native code) would be to basically tell the kernel scheduler to stop giving a given thread any time slices, then notify us when its current time slice has expired

basically have the OS kernel just stop servicing a given thread with any time whatsoever, read its private data, then start scheduling it in again

that would prevent the thread from having to care about a lock

Bob

@allquixotic That only works if you can guarantee that the thread is not in the middle of modifying said data when you stop it.

allquixotic

true, good point

that's probably why we do locking instead of yanking threads off the scheduler :P

Bob

Yep :P

That Brazilian Guy

00:28

That moment when you realize you left home to a business meeting with a client and came back drunk.

I hope it's a good sign

Bob

@allquixotic Can you just use a ConcurrentLinkedQueue?

It's supposed to be non-blocking...

allquixotic

well, all of the elements of the collection are immutable, so I could do a shallow copy

I could have a centralized ConcurrentLinkedQueue shared by all threads

Bob

@allquixotic How much write contention would you have across worker threads?

If it's relatively little, the ConcurrentLinkedQueue looks very efficient.

See page 4 here: cs.rochester.edu/~scott/papers/1996_PODC_queues.pdf

It also saves you from having to DIY anything :P

allquixotic

well all of the worker threads would be pretty much unpredictably writing at all different times to the queue

I'm loading the box down with several times more threads than there are CPU hardware threads because I know that the threads will be spending significant time waiting for the network

Bob

hm

Well, I'd vote for the simple lock approach unless it proves too slow :P

allquixotic

00:37

indeed

the problem with locks isn't that they're slow; the problem with locks comes when you need every thread in your program (or every thread in a large pool) to share the same lock

in this case I'm just tapping each thread on the shoulder once in a while

not getting everyone to stop and wait for the controller

Bob

@allquixotic The question is how much overhead is involved with obtaining a lock with almost zero contention.

Depends if the JVM is smart enough to check first (or spin a bit) before calling into the kernel.

Sounds like a SO question to me? :P

allquixotic

8

A: Synchronized vs ReentrantLock on performance

1. ReentrantLock might be more apt to use if you need to implement a thread that traverses a linked list, locking the next node and then unlocking the current node. 2. Synchronized keyword is apt in situation such as lock coarsening, provides adaptive spinning,biased locking and the potential fo...

> Synchronized keyword is apt in situation such as lock coarsening, provides adaptive spinning,biased locking and the potential for lock elision via escape analysis. Those optimizations aren't currently implemented for ReentrantLock.

bwDraco

HPC application, I suppose.

Bob

9

Q: What is adaptive spinning w.r.t lock acquisition?

One of the four major changes brought to java 6 with regards to improving the performance of intrinsic locks is Adapative Spinning technique. What exactly is adaptive spinning? Is it a combination of spinlocks and mutex lock? Can someone explain in an easy to understand way the usefulness of this...

allquixotic

so, for(Context c : contexts) { synchronized(c.logs) { logList.addAll(c.logs); c.logs.clear(); } }

Bob

00:41

Looks fine then! :)

@allquixotic Yea, that seems to be the way to go.

allquixotic

private void log(String s) { synchronized(this.logs) ... stuff... }

Bob

Overhead with 'adaptive spinning' should be negligible.

So your worker will block when the controller reads from that that thread only.

Other workers should keep going.

allquixotic

@bwDraco of sorts - designing my own load testing solution for a web app based on Selenium

Bob

> It is “adaptive" because the duration of the spin is determined by policy decisions based on factors such as the rate of success and/or failure of recent spin attempts on the same monitor and the state of the current lock owner.

allquixotic

strategy:
1. take existing functional tests written in Selenium
2. provision an enormous EC2 node in the same VPC as the application server
3. write some infrastructure around the Selenium tests to have them run in an infinite loop on many threads concurrently, logging their activity periodically

the server-side is also instrumented with a JVM hook that does function level profiling

and client-side page response time via JS/AJAX

company selling the server side profiling stuff claims an overhead of 1 - 5% so if we can meet our expected concurrent users metrics with the instance size we have, we know we'll be in good shape

if we can't meet it with the given instance size we'll have to determine what instance would get it done, or how to optimize

Bob

00:45

@allquixotic If you can't have blocks when logging mid-loop (e.g. timing) then maybe keep a local list of messages to be written once per loop.

allquixotic

I'm just being very careful about my performance consciousness as I'm doing this because I don't want to have to provision 10 c4.8xlarge nodes to do a job that one node should be able to handle

these are multi-processor E7 Ivy Bridge boxes IIRC

Haswell actually

I wonder if ReentrantLock has been optimized -- it's so weird that synchronized is faster

mr enigma

02:55

help

help!!

Bob

holy mother of misinformation: tomshardware.com/answers/id-1781107/…

"There is no such thing as port multplier for SATA." => "Of course there is, its even officially in the SATA RFCs"

"eSATA specifically supports Hot Swapping, whereas plain SATA does not;" => uhm, no?

mr enigma

toms hardware sucks

good raster image editor for windows?

Michael Frank

This could take a while... :/

mr enigma

03:11

woow

Ive had worse though

Michael Frank

Plex Media Server... Creates a hell of a lot of metadata.

mr enigma

just format it, outside of the OS

Bob

ooh. $244 Samsung 850 Pro (512 GB)

mr enigma

paint says it doesnt have enough memory to close itself

what a load of bull shit

is there any application that is like it

a simple app unlike photoshop etc

it has to be raster, not vector

and yes ive googled it

Michael Frank

03:33

Paint.net?

mr enigma

its too complicated

I tried it before

actually wait

Michael Frank

03:44

Journeyman Geek

paint.net's too complicated?

whaaaat?

mr enigma

i may take that back

looks good so far

can I move the color window so its part of a toolbar

Journeyman Geek

naw, its very much multi window interface

mr enigma

ugh but its in the way of the image

Journeyman Geek

oh

you can move/dock it

outside of the image too

mr enigma

03:57

how to dock it

Journeyman Geek

just move it to a side and it'll stick there

bwDraco

04:14

yesterday, by Journeyman Geek

http://www.pcworld.com/article/3060273/components-processors/a-new-amd-licensing‌-deal-could-create-more-x86-rivals-for-intel.html

...and AMD stock saw explosive growth, jumping a whopping 52%!

AMD licensing its Zen architecture to a technology consortium in China is a potential gamechanger, one that could enable them to beat the odds and come back with a BANG.

Journeyman Geek

@bwDraco and/or sell a lot to the chinese government

especially on the HPC market.

... 52%?

What confuses me is that they could

though I guess they licence the whole core, rather than parts of it/the architecture

bwDraco

It's their uarch and IP core they're licensing.

Journeyman Geek

Which is wierd, since I didn't know AMD could do that

cause of the tangled web of IP cross licencing in x86

HackToHell

04:39

Looks cheap Mi headphones buy.mi.com/in/item/3153700006

Journeyman Geek

04:50

@HackToHell @bob has those I think

HackToHell

05:15

Oh cool @bob how are they?

They are pretty cheap

mr enigma

05:32

whoa I just made a bitmap that was just all black and was 99 inches x 99 inches and it loaded super fast on a web page

well, I typed File:///C:/users/hh/desktop/image.bmp in the url bar actually

Michael Frank

@mrenigma compression!

bwDraco

Probably several hundred megabytes but next to no CPU power needed because it's totally uncompressed.

If it's PNG, the file will probably be very tiny but it'll be slightly slower to load.

mr enigma

hmm what if the image is more complex

how can I fill the whitespace with just random noise and save it as .bmp

it still has to be the same dimentions

bwDraco

I believe GIMP can do that. Just a sec...

mr enigma

ah I dont have it installed

Michael Frank

05:44

@bwDraco Render Clouds should do it.

mr enigma

I have paint and paint.net

bwDraco

Filters > Noise > RGB Noise. Just turn the sliders all the way up.

Michael Frank

mr enigma

thanks

how to resize the canvas

Michael Frank

getpaint.net/doc/latest/index.html

mr enigma

05:47

ah I missed that, eyes are terrible

wow the image is 27 GB

lol

bwDraco

Not surprised.

Michael Frank

That might be the working size. If it saves, it should be smaller.

mr enigma

the real image, not some test will be even bigger

so my real question is, should I do this digitally or print it

you can only have 2 dimentions with printing

and you cant interact with it, i,e click parts of the image to open new images over the top of the main image

its a huge pain in the ass trying to do it in two dimentions

infact its probably impossible.

I have 16gb on one machine

and 4 on the other two

bwDraco

What are the image pixel dimensions?

mr enigma

16 gb

bwDraco

05:51

Sorry.

mr enigma

I dont know the dimentions of the final image, its not finished at all

bwDraco

What are you trying to do here?

For reference, Adobe Photoshop is capped to 300000x300000 px.

mr enigma

woow

bwDraco

GIMP is capped to 262144x262144.

mr enigma

I might just give up

!! google three utilities problem

ChatBot John Cavil

05:56

@mrenigma * Three utilities problem - Wikipedia, the free encyclopedia ; Math Forum: Ask Dr. Math FAQ: Three Houses, Three Utilities - The ... ; Most Wanted Solutions: Water, Gas and Light puzzle - Archimedes Lab

bwDraco

This is something one would use Mathematica for.

mr enigma

the problem is it has to be 2 dimentional

even if there is a 3d model of it as a copy

cause paper is 2 dimentional

bwDraco

I'm not sure if I can buy you a copy of the software, given how expensive it is...

mr enigma

it looks like a programming language

bwDraco

Yup, and it would make much more sense to use that instead.

I'm not all too familiar with what you're trying to do here.

mr enigma

06:01

you would only say its a bad idea if I said

Im literally the worst person ever at math

dislexia and crap

bwDraco

You're not alone. Journeyman Geek is dyslexic.

In fact, dyslexia can give you an edge in some situations.

Apr 6 at 0:50, by Michael Frank

The upside of dyslexia is the ability to perceive the world from many perspectives, allowing visual-spatial thinking and special talents and skills to flourish in fields such as the arts, design, leadership, entrepreneurship, engineering, sciences, business and technology.

mr enigma

the graph has to be 2d but its whats called nonplar because there is more than K2,2 items

I dont know how to explain it

Planar graph

In graph theory, a planar graph is a graph that can be embedded in the plane, i.e., it can be drawn on the plane in such a way that its edges intersect only at their endpoints. In other words, it can be drawn in such a way that no edges cross each other. Such a drawing is called a plane graph or planar embedding of the graph. A plane graph can be defined as a planar graph with a mapping from every node to a point on a plane, and from every edge to a plane curve on that plane, such that the extreme points of each curve are the points mapped from its end nodes, and all curves are disjoint except...

HackToHell

NASA | A Year in the Life of Earth's CO2

►

Whoa much

mr enigma

I cant have any lines crossing over, or things replicated

HackToHell

via reddit.com/r/videos/comments/4fya5n/…

mr enigma

06:09

theres too many points in my graph and you cant link them with lines without needing to cross lines over, or replicate the points

so I might as well just give up

for my own sanity!

ive wasted months on it but at least I dont have that much done and therefore not much to throw away

Bob

@HackToHell Basic ones? I don't have those... they look like basic v3s. I had hte paperwhite ('basic') v2.

so-so

better than some other cheapo/crap headphones

but if you compare side-by-side with a good pair then music sounds a bit muffled

mr enigma

ive tried joining lines up instead of crossing them over, thats technically legal, but messy

it looks like a flipping MS excel sheet then

I have done it using a text file though, which is easier

but thats not really a map/graph, just a list

anybody around

Boris_yo

07:40

Annoying when From spam or phishing emails have my email address. What will happen if I mark them as SPAM or phishing? Would that block my emails to myself? When I share something I often email myself.

Bob

@Boris_yo Depends how dumb the spam filter is.

Fairly common knowledge that the mail From field cannot be trusted

because you can send pretty much anything you want in there

mr enigma

07:59

shame drawing digital graphics is slower than hand drawing

but hand drawing means planning where to start on the page so it all fits on

to hand draw a text box and line takes like 5 seconds

digital graphics looks odd if one textbox is un unnessecarilly biggger than another next to it

quite an obsession

Caterina 'Catty' McCatface

09:12

@Bob

Bob

@Caterina'Catty'McCatface

Caterina 'Catty' McCatface

I Just Came to Say HELLO - Martin Solveig et Dragonette Lyrics

►

Journeyman Geek

I just posted a slightly cynical, well researched answer cause there was a 350 rep bounty ;p

Bob

@Caterina'Catty'McCatface How's the holiday?

Caterina 'Catty' McCatface

@Bob good thanks

Bob

09:16

@Caterina'Catty'McCatface Huh, haven't heard that one in a while.

Caterina 'Catty' McCatface

Heh

Vlad

09:28

Hello.

Will ubuntu amd 64 bit work on my laptop

But I don't see EMT version

7

A: What`s the difference between Intel 64 and AMD64?

At Wikipedia's x86 page you can read In 1999-2003, AMD extended this 32-bit architecture to 64 bits and referred to it as x86-64 in early documents and later as AMD64. Intel soon adopted AMD's architectural extensions under the name IA-32e which was later renamed EM64T and finally Intel...

Bob

@Vlad same-same.

Vlad

ok, downloading :)))

Journeyman Geek

@Vlad EMT64 is intel's name - and its based off AMD64

intel can't call it IA64 since itanium is IA64

Bob

There's something ironic about the trouble I'm having planning where to put bootmgrs and how to chainload them.

Journeyman Geek

Lol

Bob

09:39

I'm usually the one answering questions about bootloaders here.

Suppose I'll just have to stick with legacy BIOS boot :\

Vlad

too bad i didn't install 64 bit OS in the first place

But din't know dual core porcessors have 64 bit instruction set

Some programs novadays work only on 64 bit OS

Mongo DB new versions will discontinue 32 bit support

Bob

@Vlad There's no direct correlation between number of cores and the architecture.

Vlad

@Bob yes, but I didn't know my processor can run 64 bit OS :)

DavidPostill

@JourneymanGeek Did you reject my spam flag on this deleted (by you) answer? :/

superuser.com/a/1068331/337631

Bob

@Vlad Yea, almost all of them within the last decade can.

Except first-gen Atom IIRC.

Vlad

09:44

From my experience I can run both 32 and 64 bit Java on 64 bit OS. Should I install just 64 bit or both?

Bob

@JourneymanGeek Ended up with an MX Master :)

Feels nice in the hand. Actually better palm grip than the G602 (slightly wider)

Won't be able to use it much till Tuesday though.

@Vlad Generally, it's up to you. Each has pros and cons.

I'd just go for the 64-bit version because I have memory to spare.

64-bit will be (CPU) faster because it has more general-purpose registers available.

It also has more RAM available.

But pointers will use more RAM for the same code.

If you had 2 or 4 GB total RAM, that might be a concern.

Vlad

C++ pointers?

Bob

I have 24 GB, so... yea, I don't really care :P

Vlad

I have 3GiBs memory to spare

Bob

@Vlad Language-agnostic. Pointers are a general concept; I'm referring to storing references to memory locations which will necessarily double in size because of the address space.

Vlad

09:47

1Gib for graphic card

Bob

@Vlad Ah... in that case you might benefit from 32-bit. On the other hand, it'll also need to load a copy of the 32-bit shared libs into memory, so... yea.

Personally, I default to 64-bit version of things like the JVM.

Vlad

So with 3Gibs onboard should I install 64 bit OS?

Bob

@Vlad Eh... so-so. Hard to give a definitive answer.

Depends on what you need.

Vlad

Well its a laptop for learning basically and for gaming

Bob

:\

Vlad

09:50

I plan dual boot Ubuntu /Windows

But games I can play

Journeyman Geek

I tend to go with 64 bit installs in general

but I consider 4gb a 'minimum' for most systems

3gb is an odd amount, unless its an older box

Vlad

I see software tends to go 64 bit, so I though it would be better when I format to install 64 bit OS

Well technically is 4Gibs but 1 is dedicated to graphic card

Journeyman Geek

@DavidPostill yeah. The user didn't seem to have any intention to spam, she just repeated an error message

Vlad

I don't think I can add more RAM to laptop, can I?

Journeyman Geek

Often you can

Bob

09:52

@JourneymanGeek He's subtracting shared graphics mem

DavidPostill

@JourneymanGeek "This reads a lot like ad copy, rather than an actual answer."

Journeyman Geek

If its a single stick of ram and has two slots, you can

@DavidPostill true, but it felt like laziness over spam

Vlad

let me check motherboard

Bob

Funnily enough RAM tends to be one of the few easily accessible things on most laptops.

Journeyman Geek

Also, I was looking at his questions elsewhere to be sure

Bob

09:54

@Vlad What's the model number?

DavidPostill

@JourneymanGeek <shrug> OK.

Journeyman Geek

@DavidPostill that wasn't a bad call on your part, but I need to often figure out "Does this guy need to be fed into the ~~wood chipper~~ spam detection system?"

And I hate spammers as much as anyone else. The point of passing these through mods (sometimes?) is to reduce false positives after all.

DavidPostill

@JourneymanGeek np

Vlad

moment my PC is busy, Windows :)

From system devices which one do I chose to see for Motherboard?

Bob

@Vlad better off reading the laptop model off the bottom (maybe in battery compartment)

Vlad

10:03

Either this one or this one:

emag.ro/…

winwin.rs/…

X54HR-SX052D Asus

X54HR-SX070

no sure about to second character group

DavidPostill

:29204679 Try wmic baseboard get product, manufacturer, model in a cmd shell

Vlad

ASUSTeK Computer Inc. K54HR

Board: K54HR

Bob

> DDR3 1333 MHz SDRAM, 1 x SO-DIMM socket for expansion up to 4 GB SDRAM

asus.com/Notebooks/X54HR/specifications

Vlad

So I guess I reached the limit already :))

4Gibs already installed

Ok, nvm, at least I can install 64 bi software :))

Will it get cloged up if I run 32 bit game, or I should run 64 version of it

?

I play SC2 on this one no prob

Ok, nvm, have to go. Will research later. Thanks for da info

Bob

@Vlad You can try either to find out. Either will work; one might be slightly faster.

Hard to say which is faster :\

If you're running close to system RAM limits, 32-bit will be faster.

If not, 64-bit will probably be slightly faster.

Bob

10:29

@MichaelFrank reddit.com/r/AskReddit/comments/4fzph7/…

Michael Frank

I dunno man... I call bullshit on that one. There's no way they would have been hiring staff at this point.

Bob

@MichaelFrank True, but I've seen weirder :P

Ramhound

11:34

Can anyone tell me if superuser.com/questions/1068890/… is the repost of the same question they submitted earlier?

DavidPostill

@Ramhound I'm sure it is - but I can't find it - I reviewed today with a vtc.

Ramhound

Well the original question was deleted, I assume, by the author because it was closed

Wondering if its worth my trouble to post (again) the information I provided.

or if the user is one of those types where they won't accept the answer or even be appreciative of my input like this genius

@Ramhound Ouch! This is an infinite loop you genius. Don't you see "i=i-1" ? Btw. Thanks for the downvote – jhmnb 3 hours ago

DavidPostill

@Ramhound No, that's my VTC on this question.

Ramhound

Oh, right

DavidPostill

@Ramhound Here is the deleted question - google found it ;)

superuser.com/questions/1068747/…

Ramhound

11:44

Was it the same user

don't have enough rep :-)

DavidPostill

Same user. Screenshot coming up.

Ramhound

alright

DavidPostill

" I presume your talking about illustrator CC 2015, if not, do some research. All your questions are answered here – Ramhound 12 hours ago "

helpx.adobe.com/illustrator/…

@Ramhound

Ramhound

imagine this, I did a google search, and found the information again.

SHOCK!

yuck just sneeze all over myself :$

Transcript for