The h Bar

3:24 AM

hi all. looking for volunteers for a challenging, cutting-edge physics open science project but with aspects for all experience levels.

toy models of QM

Danu

4:04 PM

^lolwut

Gowtham

@Danu hasn't it been around for nearly a week now ? that chatroom

ACuriousMind

5:05 PM

@vzn That room could use a room description, all I can determine from reading it is that it is about something related to determinism and QM...

vzn

hi ACM. ok. thats basically it right there. its based on a question by TK cited at the very beginning of the chat. & other brief intro there etc.

D. lol^2

vzn

5:22 PM

ACM see you quote popper in your profile & specialize in quantum-mechanics, cool. several refs to popper already in the other room :)

Danu

5:33 PM

@vzn Sounds like a contradiction

ACuriousMind

@Danu You mean challenging and physics? ;)

Kyle Kanos

Or cutting edge and all experience levels?

ACuriousMind

^I'm quite certain he meant that

Kyle Kanos

...Maybe

If I use 4 times as many processors for my simulations, it be 4 times faster, right?

ACuriousMind

@KyleKanos You're the coder here :P

But I think only if the code is really awesomely parallelized

Danu

5:44 PM

@KyleKanos Somethingsomething parallel somethingsomething series

right

Kyle Kanos

the answer is no

Scaling laws, unfortunately, aren't really linear

For my code, the slope of run time vs number of processors is around -0.75

Also, I've come to realize that I made a pretty sizable mistake in one of my test cases :(

alarge

@KyleKanos What should the algorithm, in principle, achieve? I've found that many slow-downs are not even caused by the maths, but inefficient handling of memory.

Kyle Kanos

Ideal case on ideal HPC should be a -1.0 slope

On another cluster, the code gets closer to -0.85

Just the cluster I'm currently using has it around -0.75

Most of the slow down, at least as far as I've observed & been told, is due to communication between processors

That's why people want the MPI + OpenMP parallelization. MPI to communicate between HPC nodes and OpenMP to distribute the work on a shared node

alarge

6:02 PM

Depends. For HPC applications will well developed code it is quite often true.

But you might often be waiting for memory, too. That's often more difficult to measure.

But yes, certainly adding cores will always bring the scalability down a bit.

Kyle Kanos

Most HPCs I deal with have at least 2 GB/proc on each node.

alarge

If you're waiting for node-to-node communications you'll quickly notice whether they are infiniband or whatever connected.

Kyle Kanos

And, for my code, the amount of temporary allocated memory per node is between 5 and 200 MB

alarge

Right, but you're dealing with kilobytes when we're talking about fast memory access, the CPU caches. It's easy to screw them over by noncontiguous memory access patterns, say, or false sharing.

But these scalability issues (if present) you should already notice on a local core

Kyle Kanos

It's Fortran man, memory management is hella awesome

alarge

6:18 PM

I doubt Fortran transcends the problems inherent in parallel memory management. I think that even with Fortran you can easily accidentally write programs that are susceptible to false sharing, say. That said, I'm not a Fortran expert; The last time I even gave a serious look at a Fortran code must be over 5 years ago.

And with memory layouts, and I'm no expert but I've attended conferences, I often see Hilbert curves and such to maximize locality. I don't think that these come for free in Fortran (I suppose that in principle a supersmart compiler might see these, but I doubt compilers that intelligent do or will exist).

Kyle Kanos

What do you mean by Hilbert curves "not coming free in Fortran"?

alarge

That you still have to lay out the memory by hand, that's all.

Kyle Kanos

AFAIK, the Hilbert ordering is for setting up the domain of each processor and isn't for laying out any memory

At least in the context of computational fluid dynamics

Kyle Kanos

6:37 PM

Also, with Fortran, when you send non-contiguous arrays (e.g., call foo(f(:,a,:), ret)), it will make a temporary copy that is contiguous. Obviously the size of the array can become an issue with timing.

alarge

Right, but isn't the point of it to reduce the communication, i.e. transfer of memory, between processors? Anyway, my point with memory in general was more related to how things work out at the low level, inside CPU caches: That for example things should be contiguously read and that cachelines should not be invalidated between threads (in false sharing, for example).

But how is the contiguous array generated, though? You'll be hopping all over the memory. With slices, though, it should be said that the compiler (and the hardware) should probably be able to see the access pattern.

Kyle Kanos

You'll have to ask Intel or GCC people how they implement the temporary copying, I'm not familiar with the details

alarge

Like in C++, linked lists are theoretically more efficient for a lot of things than contiguous read/writes. And they are, when the size goes to infinity, of course, but in practice because the reads from RAM are so slow, you should basically always only use contiguous memory rather than chase pointers to heap.

Kyle Kanos

I think false sharing occurs primarily with OpenMP and common blocks in Fortran. In any MPI program I've written or messed with, I've never seen it.

I could be wrong, though

alarge

Right, you would need multiple threads, so with OpenMP for example you can get it.

Kyle Kanos

6:47 PM

My code doesn't use OpenMP (yet, it's being sorta developed, they're kinda taking their time with it)

It's pure MPI at this point. Only information that's shared between processors is the flux between processor boundaries

alarge

Do you use GPUs in your field (CUDA an such)?

Kyle Kanos

Nope

There are some people who are doing that though

alarge

In case you're interested, Coursera.org is running a course on CUDA right now.

I suppose other MOOC sites have courses as well.

Kyle Kanos

An interesting, yet fairly open-ended question over on Academia:

21

Q: Are there people who have done so much work before PhD that they get a PhD automatically?

This question addresses why it's a bad idea to try to do a lot of independent work before PhD in the hope of finishing PhD quickly. Nevertheless, are there famous examples of people who have done so much work before PhD that they get a PhD pretty much automatically (say, 1-2 years or less)?

phd

user54412

7:17 PM

@KyleKanos In my experience that number is not really important. What matters is the size of L1 data cache, which is usually 32-64 KB per core. If you're operating on data that's not that close together in memory, you'll suffer cache misses that will kill performance.

user54412

This is why I predict most HPC codes will grind to a halt on the next generation of clusters, since Intel's new model is "screw cache, just add more flops"

Kyle Kanos

Yeah, I'm not too familiar with that context of memory management. I am really superficially aware of the stack vs heap memory (stack being faster, but smaller). Are there correlations to stack & L# caches?

user54412

@KyleKanos Jeff Atwood wrote up a nice analogy: blog.codinghorror.com/the-infinite-space-between-words

user54412

my limited understanding is that stack and heap are abstractions that don't really apply on the hardware level anymore

user54412

memory is memory, what varies is what copies of the relevant parts are close to the CPU at any time

alarge

7:24 PM

L# is hardware managed. Basically. You can help the hardware by using access patterns (and I think you can do prefetches with intrinsics, but that's beyond my expertise)

user54412

btw I haven't checked Jeff's numbers -- they seem reasonable but could be off by a factor of a few for some machines

alarge

The bottom line is that you fetch memory from upper caches or main memory in cache lines: You'll get what ever # of KB. If you only read one of the values out of this, that's wasteful.

Also, if you read in cachelines in some order, the hardware will be able to start prefetching them

@ChrisWhite These numbers are close to what I've seen too. What is more important, though, is maybe that the gap is increasing. 30 years ago it didn't matter, now it does: In the future even more so.

Lots of gamedevs have moved to "data oriented design", which is basically just to say that in some cases object oriented programming is not good, especially if you read in the whole object, read one boolean to check if it has been computed, and move to the next object, for example.

I think Ogre is a game-engine that got several times faster due to DOD principles.

That said, HPC applications have always been quite careful not to waste resources. It helps that OOP was never really big in those circles because few took formal programming classes (more than the intro ones, at least).

user54412

@KyleKanos for a real-world demonstration of the power of cache, the Athena->Athena++ rewrite focused on nothing else in terms of performance, and the new code is uniformly ~4-6 times faster at everything.

Kyle Kanos

Interesting

AstroBEAR 3.0 got some speed up compared to 2.0 version due to reordering the sweep algorithm

They must have had an idiot write the original version of the sweep because it kept the first index, i, constant and the looped over j and k (the variable is q(i,j,k,:))

user54412

ouch

Kyle Kanos

7:37 PM

I imagine that the author came from a C background and thought he was doing it right

It also doesn't make sense because what happens when you are doing 2D only? It doesn't make sense to loop over j,k while keeping i constant. You should keep k constant and loop over i,j

user54412

Yeah, index ordering can reverse between languages, but one can usually count on i being the inner loop

alarge

@ChrisWhite I would've said the exact opposite as the languages that I most use, C++ and Python, are row major.

user54412

@alarge Maybe that's just our in-house style. We only use row-major languages too, but we nest loops as for k: for j: for i, accessing array elements array(k,j,i)

user54412

Indices can either appear alphabetically in the code, or they can be associated with the first/second/third dimensions alphabetically, but alas one can't really have both.

Kyle Kanos

^similarly, I would do for k: for j: for i: array(i,j,k) in Fortran

Because Fortran does it the right way and orders it via columns ;)

alarge

7:53 PM

@ChrisWhite That makes sense, I suppose.

Kyle Kanos

WTF...Clemson's in-house cluster (Palmetto) says it'll take 20+ hours to compute my simulation (1500^2 grid + 1 AMR, 8 primitive variables + 2 tracers) while the same exact simulation on TACC's Stampede will finish in 64 minutes

user54412

MHD?

Kyle Kanos

Yessir

On both

Oh, and I forgot the most important bit

10 nodes with 8 processors on Palmetto and 8 nodes with 16 processors on Stampede

So 320 at 20 hrs versus 128 at 1 hour

Unless I'm reading the TACC user guide wrong and I'm actually requesting 128 nodes...

They're example batch script says #SBATCH -n 32 # total number of mpi tasks requested, which seems to mean that my using -n 128 is 128 total cores

And a little later The job acquires enough nodes to execute total_tasks tasks (launching 16 tasks/node).

user54412

are you setting --cpus-per-task?

Kyle Kanos

I don't think you need to

user54412

8:05 PM

"The default is one task per node, but note that the --cpus-per-task option will change this default."

Kyle Kanos

Stampede has only 16 processors per node

And my output file says Running MHD simulation on 128 cores

user54412

does it say 320 cores for Palmetto?

Kyle Kanos

Yes

user54412

It could be stampede configured slurm to default --cpus-per-task to 16, but Palmetto doesn't?

user54412

oh well, so much for my theory

Kyle Kanos

8:08 PM

Palmetto only has 8 or 12 cores per node

And you can't mix the different counts without messing around with the qsub script

user54412

I was going to look at how our cluster is configured, but it seems my workstation has been deactivated in preparation for moving buildings, so I can't log in to campus.

Kyle Kanos

I know that Palmetto is a giant hunk of trash these days (and it's sadly only 6 years old)

But I didn't think it was that bad

user54412

@KyleKanos Around that age I'm told clusters cost more in electricity (per year?) than it costs to replace them with something equivalent.

user54412

Not that anyone ever replaces it with something equivalent.

Kyle Kanos

They started migrating towards GPGPU to artificially inflate their ranking for whatever reason.

From what I'm told, no one is using the nVidia cards they've invested in

user54412

8:12 PM

lol

user54412

everyone talks about grade inflation and authorship inflation, but more attention needs to be given to HPC performance inflation

Kyle Kanos

Only because millions of dollars are being poured into these clusters

alarge

8:26 PM

I think GPU based solutions usually use less electricity

I think there is a Green 500 ranking or something like that where they measure FLOPS/Watt or something (of supercomputers)

alarge

8:44 PM

@KyleKanos So if you run just on 1 node per cluster (if they have queues for that), will you still get the drastic difference? Just to make sure it has nothing to do with node interconnects. Also, I am confused about how you came up with the numbers 320 vs 128 (rather than, say 320 vs 1024).

Kyle Kanos

@alarge I could do one node runs. I originally was doing both at 128 processors, but change the Palmetto one to 10 nodes (instead of 16)

But then I changed it this morning to have it run faster than 56 hours by jumping to 40 nodes instead of the 10

40*8 = 320

T Abraham

9:17 PM

@MarkMitchison according to Plenio's paper, wouldn't $\hbar \omega$ be equivalent to $\epsilon$ ?

Mark Mitchison

10:01 PM

@TAbraham yes

T Abraham

10:34 PM

@MarkMitchison and $\sigma^+_i\sigma^-_i$ = number operator?

vzn

10:46 PM

@Danu yes. but so did QM and other new theories to early founders :)

Physics Meta

0

Q: Why were my recent postings removed?

There was nothing in them that could have reasonably been considered offensive; furthermore, there was no notification about their removal.

support

vzn

re GPUs, looking for scientific refs on them & their utilization (eg in supercomputers) eg wrt this question etc

5

Q: What is the state of the art research in analysing algorithms on GPU architectures?

I have found many papers on sequential algorithms that have been implemented and tested on GPU architectures. Each of these papers usually as a result contains the amount of speedup that was achieved when using a specific GPU. However I could not find a lot of papers on theoretically analysing a...

machine-models analysis-of-algorithms

T Abraham

11:06 PM

@MarkMitchison u there?

Mark Mitchison

@TAbraham How does $\sigma^+_i\sigma^-_i$ act on the states for site $i$?

T Abraham

@MarkMitchison in the plenio paper, the sigma operators were defined as raising and lowering operators, and a raising operator * lowering operator = number operator.. Correct?

Mark Mitchison

11:30 PM

@TAbraham Yes, that's right.

T Abraham

@MarkMitchison how would I use fourier to find energy eigenstates?

Mark Mitchison

@TAbraham Why do you think you need Fourier theory?

In general finding the eigenstates is hard. It will depend on the geometry of the problem.

I suggest you start with just two sites.

T Abraham

@MarkMitchison to translate to momentum basis.. makes it easie..doesn't it?

Mark Mitchison

Momentum doesn't even usually make sense for such a system, unless you have periodic boundary conditions.

In the case of a linear chain with periodic boundary conditions, you can use the (discrete) momentum basis

But first you have to transform into a fermion representation using the Jordan-Wigner transformation.

So solve it for two sites first.

T Abraham

@MarkMitchison what about translation operator?

@MarkMitchison what that?

Mark Mitchison

11:43 PM

The translation operator in a finite system is gross. Or maybe it doesn't exist. Either way it is not useful.

Think about it. The entire universe consists of a bunch of sites in a network. What does "translation" mean in this context?

OK, hang on

It's probably not that bad

It will be an operator that takes functions $f_i \to f_{i+1}$

but when you reach the end of the network, you must have $f_N\to 0$

To be honest I have never thought about it that much. It isn't that useful for solving the problem unless you have periodic boundary conditions.

That is because there is no translation symmetry in a finite network

So the eigenstates of the Hamiltonian are not eigenstates of the translation operator anyway.

But yes, you are right that one can always define a "momentum", it just isn't useful for diagonalising the Hamiltonian

The form of the momentum operator will look something like $\sigma^+_{i+1}\sigma^-_i$ in a 1D spin chain.

toy models of QM

Transcript for