« first day (160 days earlier)      last day (1074 days later) » 

Nike Dattani
Nike Dattani
5:55 PM
A 128 core workstation would probably be done with dual AMD EPYC CPUs, not Arm. But I fully agree that a more standard, single-CPU (probably Intel) configuration is most likely easier to manage and to get to be performant for these applications. There's also a value in the redundancy you'd get in having 2-3 of these workstations. — Anyon 17 hours ago
+1 Your answer might benefit from a direct suggestion on CPU. This seems to fit your general feeling (I looked for 40 core CPUs but couldn't find one easily). amd.com/en/products/cpu/amd-epyc-7643 — Tristan Maxson 12 hours ago
I wasn't sure how this still-forming community would feel about specific Hardware Recommendations, but the 40-core Intel processor is listed here and the link you've given in your comment is for AMD which my answer says has a 64-core processor. — Nike Dattani 12 hours ago
As a curious passerby, do CPU matter models use AVX512 extensively? Or what is the reason for preferring Intel over AMD? — Jan Dorniak 8 hours ago
@NikeDattani We were going to get the 128 core AMD EPYC cpu setup, but then I got cold feet...and was thinking of going smaller until we really needed 128 cores. The max job at the moment would be 12 cores, but most would be 4. There is not much physical space for several workstations. Ideally getting the 128 setup, but with 64 cores, and then in 6 months adding 64 to the same tower. — Wesley 6 hours ago
A question I have, is if I go several workstations (find the space) and I want to do 200 jobs, each 4 cores, do I need to queue 50 per workstation? this would be annoying, I would want the main workstation to take care of that, I submit 200 4 core jobs, it figures out where to run them... Would this be hard to setup? — Wesley 6 hours ago
@JanDorniak There's an answer to that question, but not one that can easily go into a comment box, especially when this comment thread will need to be used to address the questions more specific to the actual original question such as Tristan's and Wesley's. A bonus benefit of asking as a separate question is that more people would be able to chime in and answers can have their own comment threads (we don't really allow comment chains to get too long in general, and with so many questions on this thread I worry it will happen!). Would also gain you rep which would allow you to participate more — Nike Dattani 45 mins ago
@Wesley We have an HPQC chatroom if you want to discuss more. Comment chains are generally prohibited from getting too long. Let me know if you want us to implement that feature for this question. A disadvantage of going for 128 cores, or even 64 cores, is that those machines start to get extremely niche (maybe 0.001% of the population buying them vs 0.1% for the 32 core machines). It's hard to imagine the space being too limited these days: "desktop computers" no longer have to be in "towers" [cot'd] — Nike Dattani 35 mins ago
they can be a lot smaller than in the 90s and can be stacked on top of each other. I do appreciate that space can always be an issue though, and heat/energy costs can matter too. Remember what Anyon's comment says about redundancy. With these "niche" machines that are built for a tiny fraction of the population, things can go wrong and it's better to have more than one computer around. Also: you do not need to submit 50 jobs! If you have 3 workstations you're submitting 3 jobs, which is far more convenient than setting up a connected multi-node computer and scheduler. — Nike Dattani 31 mins ago
@NikeDattani sure we could set up a chatroom. I don't understand the difference between 3 workstations linked together and 3 nodes... don't the workstations each act as a node? It sounds like I would need to log into each workstation to submit jobs on it? This sounds tedious for doing 1000's of highthroughput simulations on a weekly basis :S That is why I am drawn to 128 cores on one workstation, it seems like less book-keeping. But, if it can lead to problems, well, that is an issue — Wesley 16 mins ago
 
Wesley
Wesley
Hello @NikeDattani
 
Nike Dattani
Nike Dattani
@Wesley Hi!
 
Wesley
Wesley
I am quite the novice at hardware.
and administration of it
 
Nike Dattani
Nike Dattani
Okay, so typically when people say "3 nodes" they mean a supercomputer containing 3 nodes (so they are connected by OmniPath or Infiniband, which are extremely expensive, or by ethernet, which is extremely slow for this type of purpose ... in fact the Compute Canada national clusters that are connected via ethernet, are set up by the admins in such a way that they don't even allow you to run MPI jobs on more than one node ... so it's "effectively" 3 separate workstations).
 
Wesley
Wesley
Gotcha, I would never need a single job to be split across a node or workstation
I would not want to have to log in manually to 3+ workstations though, also, fellow colleagues need to be able to use the setup from their own computers, at a later stage
I will be creating a workflow, that once proven, is meant to be used by others
 
Nike Dattani
Nike Dattani
6:11 PM
Since you say you're a novice to the administration of it, I will warn you that setting up a scheduler and maintaining it (software updates, etc.), is not typically a maneuver for a novice at sys-admin. However, doing it yourself will be valuable in terms of you learning a lot more about how schedulers work and stuff, and once you do get it working, it does have some benefits (and drawbacks.... for example a schedule setup by an amateur once, for a lab at which I used to work,
was set up in such a way that the entire cluster of nodes crashed suddenly one day because someone sent too many jobs to the scheduler at once. A professor had several month-long calculations crash, which had to be re-started from scratch, which infuriated him.
 
Wesley
Wesley
That is my worry. I am not sure how good a default linux destro would handle scheduling
I have lots of experience using SLURM, but using and administering are totally different
Quantum Mobile is appealing to me for this because it comes with SLURM ready to go
 
Nike Dattani
Nike Dattani
The user that submitted the large number of jobs, had asked the amateur who set up the scheduler, to put a limit on his job submissions so that he couldn't send more than 80 jobs at a time (because his script would generate 30,000 jobs at once, and that's too much for the scheduler to handle let alone the hardware). The amateur sys-admin said he ran the appropriate commands to limit the jobs, and the user ran the script, but the job submission did not stop at 80 and the 30,000 jobs
entered the scheduler, causing the head node (running the scheduler) to crash, along with all jobs running!
May I ask which country you're in? It sounds like you'll have an entire group submitting jobs, which gets to the stage where sys-admin can start to chew up a lot of your time that you'd ideally be spending on research and other things. Depending on your country, this thread might help you:
39
Q: Supercomputers around the world

ThomasFor a matter modelling person, the most valuable resource is computing power. For many of us, computing power at hand limits the scale of problems we can solve. There are many national supercomputing facilities for academics. What are the resources available in each country?

high-performance-computing one-topic-per-answer
Also this one:
9
Q: Supercomputers around the world, which allow access for researchers outside the country

Anibal BezerraIn a previous question it has been shown the computation capabilities of the countries all over the world. However, if I didn't miss something, the use of such supercomputers is limited to researchers within the country. Is there any supercomputer that can be used for foreign researchers?

high-performance-computing
Now I fully appreciate that these national systems also have their own inconveniences, but the balance between those inconveniences and the ones associated with running your own system entirely, is not a trivial one.
 
Wesley
Wesley
I am in canada, it is a small company, I will do most of the jobs, but at some point, I will have scripts for everything and I will be able to let others run their own jobs. I will then move on to developing other workflows
 
Nike Dattani
Nike Dattani
I know a professor who spent a lot of his money during the early stages of his pre-tenure professorial career on his own system, but there wasn't enough people in his group to deal with the sys-admin, and there were hardware issues that you can't really solve yourself without a lot of expertise, and he described this as a very bad experience. His next grant was spent on a "contributed" cluster to SHARCNET, meaning that SHARCNET took care of the systems with
 
Wesley
Wesley
It has to be done in house, I should add, these workflows are proprietary
 
Nike Dattani
Nike Dattani
6:21 PM
their plenty of full-time staff dedicated to sys-admin and trained specifically for it, but the money for the hardware came from the individual professor. All SHARCNET users had access to the cluster, since it was "contributed", but the group of the contributing professor got "priority time" so they could run longer jobs and have their jobs sit for less time in the queue.
 
Wesley
Wesley
This is administration problem is a concern of mine which is why I would like to keep things as simple as possible, even if it means hardware that is not as fast
In this case I am industry though, it must be run on site :)
this is why I lean towards a single, large workstation, or, if it is easy to take several small ones and link them, and run them from some central location, that would work too
 
Nike Dattani
Nike Dattani
Okay, sorry for my misunderstanding!
 
Wesley
Wesley
no problems :)
 
Nike Dattani
Nike Dattani
The vast majority of the users here are not doing "proprietary" work.
 
Wesley
Wesley
yep, it is new to me as well
 
Nike Dattani
Nike Dattani
6:24 PM
Okay so I suppose the national centers are out of the question.
 
Wesley
Wesley
yeah, it literally has to be a workstation, or, potentially a couple.
 
Nike Dattani
Nike Dattani
It's a tough balance. I told you there was an amateur sys-admin (this was at University of Waterloo, which you might know since you're in Canada) who set up the scheduler for a group there, and he was exceptionally good at that type of stuff, and had experience with it from his co-op jobs, but he thought he'd limited a user's # of jobs, but it didn't work, and people lost months of calculations.
 
Wesley
Wesley
yep, I would like to avoid that, but no jobs will be months long, I would stress test it before doing anything important
 
Nike Dattani
Nike Dattani
For reasons like that, I'd personally recommend to just bite the bullet and have 3 workstations and submit the jobs more manually. But you're talking about more people joining and running jobs later, so it sounds like it's a tough call.
@Wesley Well you say you'd stress test it before doing something important, but everyone intends that, it just doesn't always work, as in that case!
You might consider asking the folks at info@hpqc.org if there's someone that could be hired part-time to help with sys-admin.
 
Wesley
Wesley
haha, true. That is why I am curious about SLURM
Well, I do agree, I am not the person to be setting this thing up! but so it goes :O
There must be big names that set this sort of thing up for Industry? I can't find them though
 
Nike Dattani
Nike Dattani
6:30 PM
They (we) are in Waterloo, Ontario as well, so hopefully close to you if you're in Canada. I'm a researcher so it wouldn't be me, but there's IT staff that may be able to help you: they're a not-for-profit organization.
 
Wesley
Wesley
Is Pavel one of them?
 
Nike Dattani
Nike Dattani
You mean Pawel Pomorski?
 
Wesley
Wesley
I am not nearby, but have met Pawel
Yes
I always mistakenly say Pavel haha
 
Nike Dattani
Nike Dattani
haha
 
Wesley
Wesley
The sharcnet fellows are a good crew
 
Nike Dattani
Nike Dattani
6:32 PM
I don't know who they'd send and how it would work... you could send an inquiry there (info@hpqc.org). If you want them to know that you've talked to me you could CC nike@hpqc.org
 
Wesley
Wesley
sounds good
 

« first day (160 days earlier)      last day (1074 days later) »