/dev/chat

Seth

02:51

@terdon Thank you :)

slm

07:21

Is this account owned by a spammer?

Habi, Nepal

118 5

Please look at the series of Q's, they've either been closed or show a very odd behavior. Specifically we fielded this one last night: unix.stackexchange.com/questions/101586/…? (need 10k to see this deleted Q)

setevoy

08:28

Good monday, people :-)

terdon

14:13

@slm he seems fine if a little misguided and very ignorant. Ignorance, however, is not something we should hold against him.

zavg

Hello, guys! Can anybody spend some minutes to explain me a little bit about processes in Linux. I stucked on one problem for several days...

terdon

@zavg I can try, what's the problem?

zavg

Hello, @terdon! Thank you for reply :) I am experimenting with massive web crawling . I know that there are different ways to solve this task: parallel OS processes, OS threads, node.js, Erlang or Go approaches, etc. I want to try different solutions and compare the performance. I have started with simple bash `script for i in {1..100000}
do
curl -s "http://somesite.com/?id="$i > /dev/null&
done`
It works wast spawning up to 15000 processes and then number of processes starts to decline. What can be the possible reason?

fast

terdon

??

Processes are finishing, so the number declines.

What were you expecting?

zavg

No, the bash script is still running

terdon

14:22

Yes, but it will launch N processes very fast, then some of those N processes will exit before the other so the number of running processes will decline.

zavg

The decreasing starts before the termination of shell script

terdon

Try saving the output instead of sending to /dev/null

Does it create 100000 files?

zavg

But the spawning of process is mmuch more faster then curl-script execution

Yes the same dynamic is when I save the results to hdd

terdon

OK, so you issue is that you never have more than 15000 simultaneous processes?

zavg

Yes, when I add simple profilng in loop using the date command the maximum number of processes decrease up to 2-4 thousnads.

I have changed the file descriptor limits to 1000000 (each hard and soft)

terdon

14:27

Well, your system probably can't run more than X sim. processes so it breaks it down.

You should be able to see the limit with

cat /proc/sys/kernel/threads-max

zavg

771885

I though about it and check the value

terdon

OK, that is strange.

Hang on, let me try it on my system

zavg

I asked question on ServerFault, people also mentioned about fd-s and pid-s serverfault.com/questions/551417/…

Thank you so much

terdon

Well, on my laptop (running Debian and 8G ram), it ate up all available memory and started swapping very soon. Perhaps your problem is that processes are being killed by the OOM killer?

I saw on your SF question that you also have memory problems so it looks like the OOM killer is the reason.

zavg

I have 49Gb of RAM and top command shows just 30Gb in use while running the script

And while monitoring through top it never goes up to this limit

I met the memory problems using GNU parallel which spawns N perl scripts which eats all memory

terdon

14:40

Then I'm out of ideas, sorry.

You could try asking your question here.

zavg

At Unix&Linux?

terdon

yes

Don't repost it, just flag your question for moderator attention on SF and ask them to migrate it here.

We have a lot of people here who are very knowledgeable about Linux internals, they might know.

zavg

Done

terdon

Well, good luck with it, sorry I couldn't be more help :)

zavg

Thank you so much. I really want to know which exact limit I faced trying to fully load machine resources

Thank you for involvement)

setevoy

15:33

terdon... did you have any idea - how to trap SIGTERM to JVM? :-) i mean - tomcat sometimes fell down... I want to run action before it's really die... strange question, may be...

catalins.sh running child proccess with JDK, where Tomcat running... but catalina.sh's proccess die after it... and left jvm's proccess "alone"...

terdon

@setevoy no, sorry, have no experience with trap.

sounds like a good candidate for a question though.

setevoy

yep, may be, if I'll don't found solution myself.... it's really unusual idea...

strugee

16:21

@setevoy the monday thing's an oxymoron

slm

@terdon @zavg - are you running out of other resources besides process ids? Check into ulimits. See the man page for ulimit (part of Bash).

terdon

In my case, it was memory, just brought the system to its knees and I had to kill. Don't have 49G of RAM though.

slm

@terdon @zavg - file descriptors is another limit, and even though a system can have X number of process IDs users can be limited to a subset of those

me neither 8-)

8 on lap, 24 on a server

i can't hand hold on this today (busy w/ work) but that is something to look into.

ulimits

terdon

I think the OP did (comment from the SF thread). He flagged it for migration to us so you should be able to get your teeth into it when it's here and you have the time.

slm

Yeah, not today, maybe later tonight

strugee

16:25

@slm: what do you mean "part of bash"?

terdon

@strugee that ulimit is a bash builtin

$ help ulimit
ulimit: ulimit [-SHacdefilmnpqrstuvx] [limit]
    Modify shell resource limits.

From man ulimit:

   Warning: This routine is obsolete.  Use getrlimit(2), setrlimit(2), and
   sysconf(3) instead.  For the shell command ulimit(), see bash(1).

slm

@terdon - thanks

man ulimit takes you to the bash man page

ulimit -a

terdon

Not on my system, it takes me to the man page of the library

ULIMIT(3)                  Linux Programmer's Manual                 ULIMIT(3)

NAME
       ulimit - get and set user limits

SYNOPSIS
       #include <ulimit.h>

       long ulimit(int cmd, long newlimit);

DESCRIPTION
       Warning: This routine is obsolete.  Use getrlimit(2), setrlimit(2), and
       sysconf(3) instead.  For the shell command ulimit(), see bash(1).

       The ulimit() call will get or set some limit for the  calling  process.
       The cmd argument can have one of the following values.

@terdon /me shrugs. I use .profile to define shell-agnostic things, and then source it. see unix.stackexchange.com/questions/88201/… — strugee 3 mins ago

@strugee yes, but environment vars and aliases are two very different things.

.profile is not read by interactive shells so having aliases there makes little sense.

strugee

16:45

@terdon apparently it's also a separate command: manpages.ubuntu.com/manpages/trusty/en/man1/ulimit.1posix.html

/me wonders how his comment ended up in chat

terdon

@strugee might be an Ubuntu thing, I don't know.

@strugee :) Just so as not to clutter up the comments over there. I wanted to answer you without spamming the OP.

strugee

@terdon oooooooh. and yeah, looks like it's not in Debian

terdon

In Debian, man ulimit brings up the man page of the library (see above) and

$ type -a ulimit
ulimit is a shell builtin

strugee

ah. I'm on a Windows machine (eugh) so I did a package search on packages.debian.org (name and contents)

terdon

And found one? Huh. OK

derobert

16:55

Maybe man 1 ulimit vs. man 3 ulimit

terdon

I don't see it

derobert

Though I don't have a ulimit(1) manpage on my Debian box

terdon

$ man 1 ulimit
No manual entry for ulimit in section 1

@derobert exactly

and: $ apt-cache search ulimit
cpulimit - tool for limiting the CPU usage of a process

strugee

@terdon no, I didn't find one. that's why I said "looks like it's not in Debian"

derobert

Honestly, my thought of reading that conversation is basically "@zavg is a madman..."

He/she tried to run how many processes‽

FAR fewer than that would, I'd expect, fully saturate a gigabit link

terdon

16:59

Yes, well I think he's just curious to know what limit he's bumping up against.

derobert

Hopefully! I won't deny that I too at times have played the madman, just to see how far I can push it before it explodes...

terdon

:)

@derobert can you help me out with this?

you mean something like this repository ? cdimage.debian.org/cdimage/weekly-builds/amd64/iso-cd the first CD alone is enough for a desktop installation ? — user2485710 11 mins ago

How does one get an ISO to install testing? Do you need 5 CDs?

zavg

@slm I have already raised soft and hard limits for fd-s up 1000000...

derobert

@terdon The first disc should be enough to complete the install. It probably even has a reasonable set of desktop utilities on it

For all the common packages, DVD-1 would probably suffice

terdon

OK, cool.

derobert

17:03

And of course BluRay 1 has even more. Amazing that its now three bluerays...

Note the different variants of disc 1, for different desktop environments

terdon

I know, that's what surprised me. LMDE has a perfectly working version of testing on a single iso.

setevoy

@strugee yep... :D

derobert

Well, keep in mind that if you burnt all three bluerays, that's everything. The entire distro.

terdon

Ah OK, so they cram the entire repos into them.

derobert

Yes. You'd only grab those if you needed to operate offline.

Also, they're sorted by popularity. So DVD1/CD1/BR1 has the most popular, then DVD2/CD2/BR2 has the next most popular, etc.

Sorting of course becomes less reliable the further down the list you go, as packages start becoming equally unpopular.

[And, of course, it is testing. Any given image may be broken.]

terdon

17:08

OK, but if the first is enough to get a working system with WiFi the OP can do the rest.

zavg

@derobert I have gigabit port, and I actually really want to saturate gigabit channel. I am using iftop tool but it shows RX just up to 100Mbit/s

slm

@zavg what about ulimit -u?

derobert

@zavg with curls? You have a huge overhead of spawning one curl per URL

And if you're actually getting 100mbit fairly precisely, I dispute that you have a gigabit connection.

zavg

@derobert Also even if the network becomes saturated why the number of spawned curl processes decreases to zero before the termination of generator script

derobert

@zavg to zero? That's surprising.

Someone have a link to the Serverfault question?

zavg

17:15

To be honest I did not wait till zero but it declines monotonously till several decades

terdon

0

Q: Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do echo "club $i..." curl -s "http://some_url_here/"$i > $i.txt& END=$(date +%s) DIFF=$(( $END - $START )) echo "It took $DIFF seconds" done ...

linux performance bash curl wget

zavg

0

Q: Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do echo "club $i..." curl -s "http://some_url_here/"$i > $i.txt& END=$(date +%s) DIFF=$(( $END - $START )) echo "It took $DIFF seconds" done ...

linux performance bash curl wget

derobert

Ok!

First, where did that 15,000 processes come from before? I see 2,000–4,000 in the question...

That should be doable...

@zavg Which webserver are you running this against?

I hope it's one you control!

zavg

@slm I am not at workplace but both fds and processes limits are currently far away from 15K

derobert

Your example shows them all going against one server.

My first guess would be that your DoS attack on that server has succeeded.

zavg

17:20

@derobert Of course, I understand total overhead, I want to experiment with different approachs

derobert

So, your test script has to get through spawning at least one date command between each curl

And if those curls are exiting extremely fast—say, because you've killed the web server, and its just sending a RST to each SYN—you may not be spawning them fast enough.

Your shell script is fighting with several thousand curls for CPU time.

zavg

In question script I have profiling overhead, if drop timing and left just with loop and curl the processes amount raises up to 15K

derobert

Yeah. So that's probably it. You're hitting the limit of how fast you can spawn them.

Run two instances of your spawner script, I bet you'll get more.

zavg

The server is not mine, but it is highload project ready for such loads.

derobert

I hope you have their permission!

Because you're throwing 15000 simultaneous connections at them.

Note that you won't get much higher than say 30k, because you will have run out of TCP ports.

(Each TCP connection must be unique across four values: source port, source IP, destination port, destination IP... and three of those four are the same for your connections. So you're left with just source port)

Also, if there is a NAPT firewall between you and the server, it may well collapse under this load—or at least, start discarding the connections.

zavg

17:28

I can use more thab one IP to increase TCP connections numbrr.

How to check your last hypothesis?

derobert

Yep. That'll give you another 30K connections or so, per IP

The firewall? The best way would be to ask the network people.

Hopefully you know about any firewalls on your side.

The server folks should know about any on their side.

BTW: If you want to get a really high number of processes running, you need processes that don't exit.

Each process should be given a bunch of URLs to work on

The more work each process has to do, the less overhead you're paying for setup & teardown.

zavg

Yes, I thought about writing node.js or Erlang crawler, but if I meet network bounds with simple processes approach why should I make things more complicated..

derobert

@zavg well, you can stick to simple—just give each curl 50 URLs

I think curl can do that...

wget surely can

yeah, curl can do that, according to the manpage

I haven't done this before, but I'd guess your best performance will come from a small number of processes (~2 × number of CPU cores), each handling many sockets, using epoll or similar.

And each getting work, in batches of several thousand URLs, from a control process

When you do this for real, keep in mind a lot of things will collapse before you fill your gigabit pipe. For example, your nameservers.

@zavg I assume you see those messages? Just saw your icon pop back in...

zavg

17:43

Yes, I see it)

nameservers?

derobert

@zavg I assume when you're not testing, its going to be a bunch of different servers—so you'll be creating a lot of DNS traffic.

zavg

Yes

derobert

When you're throwing tens of thousands of queries per second at your nameservers, they may well fall over.

zavg

@derobert I ran two bash scripts in parallel as you suggested, but the upper bound for processes nmber is the same ~15000

derobert

Odd. I wonder what other limit you're hitting then. Most of them should produce an error message...

zavg

17:51

I thought about it! How too catch these messages?

derobert

They should be spewing to your console.

... the terminal you're running it from, that is

zavg

No, it is clean

derobert

also check your 1.txt … 100000.txt files, is the output as expected?

zavg

Yes, I did not check manually all, but random checking shows that they are correct

derobert

you can't use some shell script (with, say, grep) to check them all?

@zavg BTW: stackoverflow.com/questions/6294133/maximum-pid-in-linux unix.stackexchange.com/questions/16883/…

zavg

18:11

Yes, I have tried to change the max PID number, but it did not helped... Yes, it seems that on this speed there are lot of empty files.

derobert

@zavg So you're getting errors of some sort. I'd guess you killed the server.

Grabbing a tcpdump of it (preferably from another machine, with a switch port in monitor mode) would let you confirm

zavg

I cannot kill it with such a small workload. I am running queries to the service decades of millions of daily visitors...

But the biggest question for me is why the number of curl processes starts to decline and does not stays at certain value?

derobert

@zavg You do realize that even 10 million requests/day, even if its only over 8 hours of "prime time", is only 350 connections/second? i.e., 40× less than you're throwing at them?

Have they actually told you that you can do this?

If not, please stop, you are conducting a DoS attack.

zavg

18:29

They have lots of front-ends and distributed load balncing along them...

derobert

And have they actually said its OK to test this kind of load on them?

If not, you're probably hitting some DoS mitigation they have. Eventually, it'll get you permanently firewalled, reported to your ISP's abuse desk, and possibly law enforcement.

You really ought to set up your own server to test against

zavg

Ok, thank you for your warning...

But actually I am not testing, I am receiving the information which is in public access (it is even accessible for search engines indexing). Like, for example, google do with its crawlers.

I have no aim to test the security or load sustainability.

terdon

Have you tried simply adding a sleep 0.2 or so between calls? If this is indeed a DoS protection, that might get you past it.

derobert

18:44

@zavg I'm confident Google doesn't crawl 15k URLs simultaneously from the same site

zavg

@terdon it will be unfortunately too slow

derobert

How many things are you trying to download from this site?

terdon

@zavg not for testing, you can at least see whether the number of processes behaves as you would expect. Then you would at least have narrowed the problem down.

zavg

@derobert every URL from these 15K is the separate page with different content visited by lots of people per day...

terdon

Adding a 0.2 second delay will not make it that slow. You can easily wait a few seconds to get to a few thousand procs.

derobert

18:47

@zavg If you just had 15K URLs to crawl, you could do that at one per ten seconds, creating hardly any load on the site, and you'd get your results in <2 days

zavg

@derobert I just start the crawling I described and then meet 4K limit with script with profiling date calls and 15K limit with pure spawning script...

@derobert No, the total amount is much more then 15K

derobert

48

Q: How to be a good citizen when crawling web sites?

I'm going to be developing some functionality that will crawl various public web sites and process/aggregate the data on them. Nothing sinister like looking for e-mail addresses - in fact it's something that might actually drive additional traffic to their sites. But I digress. Other than hono...

web web-scraping

zavg

@terdon If I didn't miss something the 0.2s delay between calls means 5 calls per second, so in N seconds it will be less then 5*N processes because lots of processes will be already terminated...

@derobert Thank you very much for the link)

derobert

You are trying to crawl at a rate that is utterly insane to do against a single site. At 5000/sec, you'd be pulling around 13 billion documents/month

terdon

@zavg yes but how many processes need to be launched for you to start observing the problem? At least you can give it a few minutes and make sure that this is the source.

@derobert I think he is hitting different sites

14 mins ago, by zavg

@derobert every URL from these 15K is the separate page with different content visited by lots of people per day...

derobert

19:01

@terdon Not in the test script he's posted.

zavg

@terdon In my first version with some delay existed due to profiling almost all pages are got correctly

But the processes numbers starts to decrease approximately at number of 4K

terdon

Ok, so you should be able to run a test with a delay of 0.2 secs in about ten-twelve minutes right?

zavg

Of course

terdon

19:17

I just figure that since you've already spent a few hours on this, launching a test that will run for ~13 minutes and will help you narrow down the error seems worthwhile.

Gilles

20:29

Winter Bash is coming back this year

I demand a winter zsh!

3

derobert

21:11

We could go lightweight, and have Winter dash or Winter ash!

Gilles

@derobert I think they had plans for that but they went up in smoke

derobert

Well, if the plans are going to go up in smoke, we should go for winter hickory instead. Then at least we'll get some good eats out of it...

Amazingly, apt-cache search hickory gives nothing

slm

As opposed to Winter Flash, which crashed 8-)

6

strugee

this just made my day

@Gilles when does it run? are we doing it? (is there a meta question on it?)

Gilles

21:29

3

Q: Winter Bash, édition 2013

Comme l'an dernier, Winter Bash approche, et les site du réseau Stack Exchange désireux d'accroître leur charisme du 13 décembre 2013 au 3 janvier 2014 au moyen de chapeaux fort seyants sont priés de se faire connaître. Y participerons-nous cette année encore ? Les réponses des modérateurs sont ...

discussion design

(there's an English version as well)

Michael Mrozek

21:51

@strugee We're probably doing it, we did last year, but I messaged the other mods to get their input

strugee

@MichaelMrozek \o/

Gilles

@MichaelMrozek this site has other mods?

Michael Mrozek

@Gilles Dozens! Most are undercover, working in secret

2

Gilles

should we burninate gnu?

strugee

looks at [gnu] tag oh gee I own the third question in that list

derobert

21:57

@Gilles Why?

Gilles

@derobert because most questions have it only because some GNU software was used somewhere

derobert

But its also reasonably valid to tag questions that are about the GNU project, or GNU software...

terdon

It also does not give much useful information now that I think about it.

@derobert yes, it would be useful there. I would guess most gnu Qs are referring to coreutils

Gilles

@derobert not questions about GNU software unless they are specifically about GNU software as opposed to other versions

Transcript for