Ask Ubuntu General Room

Roland Taylor

8:00 PM

@GeorgeEdison tis true. They roll out things incrementally.

George Edison

So that might be the reason you're only seeing it now.

rlemon

grumbles.. They should tell you shits gonna change before they change it

I feel violated when sites do that.

George Edison

I love Wolfram-Alpha ads:

Roland Taylor

BTW, I'm looking to offer bounties soon from now :). So point me to good answers or good questions in need of a good answer.

George Edison

Okay.

I think Wolfram-Alpha messed up something...

Octavian Damiean

8:10 PM

heh

George Edison

That's not a kingdom IIRC.

rlemon

@GeorgeEdison you might know... in the chat system here, how can I programatically tie into a new message being displayed?

George Edison

Well, I do it the hard way...

...override the send() method of XMLHttpRequest.

dragosrsupercool

Anybody knws curl here?

rlemon

$('#chat').live('change', function() { }); ??

Octavian Damiean

8:15 PM

@dragosrsupercool Hello. What is your question?

George Edison

@rlemon I'm sure there's an easier way, but I don't know what it is :P

dragosrsupercool

@OctavianDamiean: I am trying to write an architecture in which one thread shall send url to the remote server to request data while the other thread would fetch data when avaibale.. idk how to do this using curl

George Edison

@dragosrsupercool What programming language are you using?

jrg

@JorgeCastro after an hour wrestling with it, I give up on that importer.

dragosrsupercool

@GeorgeEdison: libcurl C/Python

Octavian Damiean

8:17 PM

Why do you need to use curl for that?

jrg

If you really want, I'll try and do something else, but right now, i'm out of ideas. :\

dragosrsupercool

@OctavianDamiean: curl is stable i guess.. what do you suggest/

Octavian Damiean

For Python there is urllib2.

dragosrsupercool

@OctavianDamiean: urllib2 is far behind curl in lot many ways

George Edison

@dragosrsupercool If you want to use libcurl in Python, unless there is a wrapper somewhere, you'll need to use ctypes.

dragosrsupercool

8:20 PM

@GeorgeEdison: I just want to use curl

jrg

github.com/Lispython/human_curl

George Edison

@dragosrsupercool Ohh.... you mean invoke the curl command?

dragosrsupercool

@GeorgeEdison: libcurl (curl.h in C)

or pycurl in Python

George Edison

PycURL... so there is a Python wrapper for cURL.

dragosrsupercool

@GeorgeEdison: yes

George Edison

8:22 PM

What is it that you need help with?

dragosrsupercool

@GeorgeEdison: what exactly happens during curl/multicurl perform ?

George Edison

Installing it?

Octavian Damiean

Huh?

Jorge Castro

0

Q: How do I enable Unity when using Ubuntu 12.04 Alpha 2?

At the moment, I am using Ubuntu 10.04 with visual effects on, i.e. with the shadows behind the windows, the windows wobble when moving etc. I downloaded Ubuntu 12.04 Alpha 2 and booted up using the livecd and typed echo $DESKTOP_SESSION in a terminal window. It returned Ubuntu-2d. How do I en...

one more folks!

dragosrsupercool

@OctavianDamiean:??

Octavian Damiean

8:23 PM

@dragosrsupercool That question is a bit vague. What exactly do you want to know?

Jorge Castro

hey @GeorgeEdison

did you ever find an answer to this?

4

Q: How to get my Flash drive to cache disk reads to my primary partition?

Basically I'm looking for a way to mimic Windows' ReadyBoost. I asked this question on Unix.SE to get an idea of whether this was even possible and it seems like it is. Appparently there is a cachefs filesystem I can use to do this. My question is then: how can I configure Ubuntu to do this? F...

performance usb-drive cache

George Edison

Lemme check...

dragosrsupercool

@OctavianDamiean: I wish to create two threads, one which sends request for data to remote server and other which receives data from remote server. How to do this using Pycurl? The problem is we use multicurl perform thing which does parallel things internally while if we use easy handle the problem you cannot do anything before perform function return

*returs

*returns

George Edison

@dragosrsupercool I don't think you can have two threads working on the same socket.

@JorgeCastro Not really... I got an answer that was far less than helpful.

Jorge Castro

yeah, that answer is totally incorrect

George Edison

8:27 PM

Just telling me to install a package and figure out the rest... which I couldn't.

Jorge Castro

that's for network filesystems

dragosrsupercool

@GeorgeEdison: So does it mean once a request is send, we have to keep the thread in wait mode than making it useful for something else?

@OctavianDamiean: what do you say?

George Edison

Once your thread sends a request, it needs to wait for the reply.

For Python, I'd actually recommend looking into Twisted.

...it's an excellent framework for network communications - used by lots of Python apps, including Bazaar.

dragosrsupercool

@GeorgeEdison: What about C?

George Edison

Is C++ an option?

dragosrsupercool

8:30 PM

@GeorgeEdison: Does C++ have any such option?

George Edison

For C++, I would recommend the super-awesome Qt framework.

It's cross-platform and it makes network applications really easy to write.

Questions Feed

Bounty offered: How is disk space shared from nodes when using Ubuntu Cloud? http://askubuntu.com/questions/102574/how-is-disk-space-shared-from-nodes-when-using-ubuntu-cloud #cloud

Tweeted by StackUbuntu on February 10, 2012 at 8:27 PM

Thomas Ward

@GeorgeEdison OHAI

i see you checked the bug :P

dragosrsupercool

@GeorgeEdison: why are we talking about frameworks than curl?

George Edison

@ThomasWard Yup.

Thomas Ward

8:32 PM

@GeorgeEdison if the 2010 libraries are loaded into a system, it works. MSVCRx64 installs both the 32bit and 64bit libraries for cross-compatibility

George Edison

@dragosrsupercool Do you really need to use cURL or do you just need similar functionality?

Thomas Ward

and the only reason i know that is because I actually code in Visual Studio :P

George Edison

@ThomasWard Ah, excellent.

Thomas Ward

@GeorgeEdison since XP only has 32bit support for the MSVCRuntime, the x86 version is what I linked

Vista and later can support the 64bit though

George Edison

@ThomasWard I used to use it and ended up quitting because of the issues with the VC++ runtime.

Thomas Ward

8:33 PM

mmm

i have to use it :/

mainly because of Sharepoint development

i just installed every language because i could, though xD

MSDN has its uses :P

points to his MSDN license

also, i'm still subscribed to full bug updates for StackApplet xD

George Edison

I used VC++ to build 64-bit editions of some other applications on my 32-bit machine.

dragosrsupercool

@GeorgeEdison: Anything would go if its better than cURL!! But then you must elaborate why is it better?

George Edison

I still remember all of the configuration mess I had to go through to get that to work.

Thomas Ward

@GeorgeEdison and this is why i use 64bit OSes

George Edison

@dragosrsupercool Qt offers an event-based setup for network communications.

...so you don't really even need multiple threads at all.

A single thread can be used for the interface / GUI and network requests.

...without one or the other getting blocked.

dragosrsupercool

8:37 PM

@GeorgeEdison: I am writing a crawler.. Are you suggesting Qt over Curl? I

George Edison

If you are looking to stay in Python, the Twisted framework I pointed out earlier offers similar functionality.

@dragosrsupercool Why do you need multiple threads for a crawler?

dragosrsupercool

@GeorgeEdison: I need to make sure I am utilizing CPU and Network fully!! No over load no under load

George Edison

@dragosrsupercool A crawler probably won't need to worry about CPU usage... unless you are doing a lot of processing on the data you retrieve?

dragosrsupercool

@GeorgeEdison: Using CPU fully is how I plan to scale up the crawler performance..

George Edison

What is the crawler doing?

dragosrsupercool

8:40 PM

crawling.. .

George Edison

All it's doing is fetching data?

...it's not interacting with the data at all?

dragosrsupercool

it fetch data it saves data thats all

now when its waiting for remote server reponse cpu is free

which is like waste of cpu cycles

i want to make sure another requests can be entertained by the cpu at that time

now for this there can be two approaches

one can be you make every url request a thread

other can be you just use two threads

one keep sending request

other keep receiving reponse

so cpu get utlilized fully

so does network

George Edison

I'm sorry, but I'm pretty sure that two threads cannot use the same socket.

If one thread writes data to the socket, then that thread is the only one that can read from it.

dragosrsupercool

may be may be not.. if a thread writes data he must be using some memory reference to read

if that memory reference is used by some other thread

then it might be able to read

George Edison

No, no - I mean writing to the socket.

Thomas Ward

8:46 PM

@GeorgeEdison also, regarding StackApplet, Windows 64-bit can still run 32-bit apps :P

dragosrsupercool

you mean no two threads can write to the same socket at one time?

I agree...

but my concern is one thread reading from socket while other writing to that socket

and they are not doing that at the same time

George Edison

@ThomasWard I know... but it will still need the 32-bit runtime libraries, right?

dragosrsupercool

i want to implement this using curl

in someway you can say kind of pipelining over different domains

George Edison

@dragosrsupercool I don't think two threads can even use the same socket.

Thomas Ward

@GeorgeEdison which are installed usually by default. On a 64bit system, the x64 runtime libraries should autoinstall both 64 and 32bit support

George Edison

8:47 PM

Hmm... then I'm puzzled over the error message.

Thomas Ward

@GeorgeEdison one moment

George Edison

Maybe the problem is with the PyGTK libraries...

Thomas Ward

@GeorgeEdison

0

Q: Unhandled Exception in Microsoft Visual C Runtime

This is the error I'm getting: The procedure entry point '_except_handler4_common' could not be located in the dynamic link library msvcrt.dll This occurs on a Windows XP SP3 system with Microsoft Visual C++ Redistributable 2005 / 2008 installed, but does not occur on a system with the Visuial ...

c++ visual-c++ exception

generally speaking it might be needing MSVCR 2010 support

(which seems to have the entry point / symbol)

dragosrsupercool

@GeorgeEdison: TCP socket is a full-duplex stream, you can read from and write to it from multiple threads.

George Edison

Ah, well that's a reasonable requirement.

Thomas Ward

8:49 PM

which is why i added "Try running with the 2010 libraries"

if they come back and say "This fixed it", then all you need to do is add a "Prerequisites" section saying "Requires MSVCR 2010 Redistributable installed"

which is available for XP and later (XP as 32bit only)

George Edison

@dragosrsupercool I will concede that it is technically possible to have multiple threads accessing the same socket BUT it is not recommended as it is extremely error-prone and dangerous.

...and completely unnecessary in your case.

Each request (as far as HTTP goes) can only access a single resource.

dragosrsupercool

@GeorgeEdison: Writing using thread 1 and reading thread 2 that too at different cpu cycle IS COMPLETELY SAFE!!

anyways thanks.. i appreciate you help :)

George Edison

@dragosrsupercool Not really if you're making HTTP requests... you would have to write an awful lot of synchronization code.

N.N.

@dragosrsupercool You can actually edit your messages. For a minute or so, then it will be too late forever.

dragosrsupercool

@NN: lolz thats great.. i dint realize it.. anyways thanks.. i would keep that in mind from now onward :)

George Edison

8:54 PM

I would definitely recommend limiting a socket to a single thread.

There is definitely no need to use more than one.

Thomas Ward

what @George is saying is logical :P

N.N.

@dragosrsupercool It is a nice feature. Takes time to get used to for IRC diehards

Octavian Damiean

Even if it would be possible that two threads can operate on a socket it could have catastrophic results.

George Edison

I should correct something I said earlier... HTTP can be used for more than one request using KeepAlive.

...but I would treat that as an exception.

dragosrsupercool

@GeorgeEdison: As far as I know sockets are thread safe. You should only be careful when you call close() on socket from one thread. Second one could hang on some blocking function or select infinitely.

@GeorgeEdison: KeepAlive is not an exception but a feature which other protocols are planning to release in their coming versions... :)

George Edison

8:58 PM

@dragosrsupercool Well, you can definitely take advantage of it if you like but realize that it means a lot of special-case code to handle the extra complexity.

The bottom line is you don't need more than one thread per socket (regardless of whether it is a good idea to use more or not).

dragosrsupercool

@OctavianDamiean: I agree.. . But in this specific case thread 1 makes the request and dont access the socket ever again.. . While thread 2 handles all writing operation.. correct me if I am mistaken

George Edison

@dragosrsupercool Why not have the same thread read the response?

Octavian Damiean

You know, there is a very good advice in software development. Premature optimization is the path to the dark side of software development.

dragosrsupercool

@GeorgeEdison: Reponse can take time.. If we use the approach you are suggesting I guess we need to keep a thread per url.. Many threads can make system totally inefficient.. .

George Edison

@dragosrsupercool I'm only suggesting a single thread.

Octavian Damiean

9:02 PM

So basically, don't try to over-optimize it now but get a working single threaded version and find out if you need to optimize from that.

George Edison

A Qt app, for example, can use a single thread to send multiple requests and receive the responses all without blocking a GUI or command-based interface.

dragosrsupercool

@GeorgeEdison: Using single thread is going to waste your cpu cycle and make the complete process very slow.. .

Octavian Damiean

@dragosrsupercool Is that an assumption or an observation?

Because assumptions are yet another path to the dark side.

George Edison

@dragosrsupercool I doubt it - you're not doing any processing on the data so it should be light on the CPU.

dragosrsupercool

@OctavianDamiean: I appreciate but I already have written crawlers to crawl like 2M urls per day using single thread :)

its an observation

George Edison

9:04 PM

It's just network-heavy and disk IO-heavy.

dragosrsupercool

my current project is to enhance the crawlers

George Edison

@dragosrsupercool Are you sure the CPU is the bottleneck?

...and not something else?

dragosrsupercool

@GeorgeEdison: explain IO heavy?

Octavian Damiean

Alright, I just wanted to know why you want to add that much complexity. :)

George Edison

@dragosrsupercool Heavy on disk reads / writes.

(Since you said the data retrieved was being saved to disk.)

dragosrsupercool

9:05 PM

@OctavianDamiean: Its just because I need to optimize codes to an extreme level.. .

@GeorgeEdison: Crawling time is greater than saving time.. .

in this particular case

George Edison

@dragosrsupercool Meaning the network is the bottleneck?

Compiling code, for example, is heavy on CPU usage... but network crawling shouldn't be.

dragosrsupercool

@GeorgeEdison: network is not the bottle neck, the algorithm is.. we are waiting for server response and wasting cpu and network bandwidth while waiting

all i am trying to do is using the wait time ;)

George Edison

@dragosrsupercool Why are you wasting CPU cycles while you are waiting?

It's completely unnecessary if you are using an event based framework.

dragosrsupercool

@GeorgeEdison: Do you think CPU cycles are getting utilized? How and What exactly for?

George Edison

What is your application doing when it's waiting?

Octavian Damiean

9:09 PM

So, what would you want to do with the now wasted CPU cycles?

N.N.

As long as the programmer cannot come up with perfect assembly code the bottleneck is the programmer.

Octavian Damiean

Assembly? :o

Sheesh!

dragosrsupercool

@NN: I agree :)

George Edison

@NN No, a compiler is far better than a programmer at optimization.

After all, it's the compiler that's generating the assembly code.

dragosrsupercool

@OctavianDamiean: Simple, I want them to work on other urls till then ;)

George Edison

9:11 PM

...unless you are writing the application in assembly - in that case...

dragosrsupercool

@GeorgeEdison lolz :p

N.N.

@GeorgeEdison I was not serious. Also, programmers are probably more expensive than cpu cycles.

dragosrsupercool

@NN: I agree again lolz

George Edison

@dragosrsupercool Is the network connection saturated at that point?

...or is there spare bandwidth?

dragosrsupercool

@GeorgeEdison: No No No ,, And that is why I am saying lets use it!!

Octavian Damiean

9:13 PM

Wait, so you actually just want multiple parallel requests and save the responses to disk?

George Edison

@dragosrsupercool So why not have a single thread issue multiple requests?

dragosrsupercool

@OctavianDamiean: we dont use disk.. we use Amazon SQS and S3 .. it wont be a bottle neck atleast till 20x scaling

Octavian Damiean

Well to that then. :)

And where exactly was the problem? I have lost track. :P

N.N.

Would it make sense to try to get these three meta.askubuntu.com/unanswered/tagged/status-completed out of the unanswered questions list or would people just complain if I bring the to the front of meta for some days?

dragosrsupercool

@GeorgeEdison: And how is that done? Can you achieve parallelism using a single thread?

George Edison

9:15 PM

@dragosrsupercool In Qt, for example, this is easily done using events.

N.N.

I think they should be answered so that they might have a chance of getting a vote and go away from the unanswered. Seems easier to do this than push the devs to accept meta.stackoverflow.com/questions/65429/…

dragosrsupercool

@OctavianDamiean: lolz the question was i have two threads i want thread 1 to keep requesting data while thread 2 to keep receiving the data.. I can do this using socket programming easily,,, I wish to implement C curl..

George Edison

@dragosrsupercool Why do you need cURL?

I'm really confused.

dragosrsupercool

@GeorgeEdison: Again, Qt uses threads.. . You are just suggesting to use internal threading over external threading .. .

Octavian Damiean

Why don't you just launch new threads that do everything?

George Edison

9:18 PM

@dragosrsupercool No it doesn't.

dragosrsupercool

@GeorgeEdison: Do you understand cURL? What is it exactly for?

George Edison

Not unless you specifically write the program that way.

Octavian Damiean

I mean thread one is busy handling URL 1, thread two is busy handling URL 2 and so on.

George Edison

@dragosrsupercool cURL is a command line tool for fetching data from a remote server using a number of different protocols.

cURL (the command line tool) uses libcurl, a C interface to the actual code.

@OctavianDamiean That's a possibility, but I'm not convinced that a single thread wouldn't be better if possible.

is getting frustrated trying to compile subversion with MSYS.

dragosrsupercool

@OctavianDamiean: maintaining too many threads is very costly for the kernel.. moreover again threads waits which is like wasting resources for sleeping

George Edison

9:21 PM

@dragosrsupercool No it isn't.

A thread that doesn't do anything shouldn't be wasting CPU cycles.

dragosrsupercool

@GeorgeEdison: a thread doing something or else its called dead

George Edison

@dragosrsupercool It's really the creation and destruction of threads that take their toll on the processor.

@dragosrsupercool Not if it yields to the kernel.

Octavian Damiean

Right and for that there is thread pooling.

dragosrsupercool

@GeorgeEdison: my dear friend George, at the end of the story you are suggesting to access same socket with multiple threads... From last one hour I was trying to say the same

George Edison

Exactly - have a pool of threads and just assign a new request to one that is idle.

Octavian Damiean

9:22 PM

nods

George Edison

@dragosrsupercool Which language do you want to use?

dragosrsupercool

C

George Edison

@dragosrsupercool Uh... no. I still maintain a single thread would be better.

It's not me that wants more than one thread.

dragosrsupercool

@GeorgeEdison: So you think a single thread is better haan? What way?

apart from your crashing threat which it wont in this case

George Edison

@dragosrsupercool It's less complex and still equally efficient.

dragosrsupercool

9:25 PM

how is it even 1% equally efficient? Explain.. .

George Edison

A single thread making multiple requests is more efficient than multiple threads making requests since the communication between threads and context switching is eliminated.

Octavian Damiean

Well if blocking operations are involved I can understand the desire for multiple threads but just not the way @dragosrsupercool wants to approach it.

dragosrsupercool

ok George leave it... You try it.. check your stats that would help you clarify the concepts ;)

George Edison

@dragosrsupercool Check my stats?

@OctavianDamiean True... but if done right, nothing needs to block.

dragosrsupercool

@GeorgeEdison not your check single threaded, n multi threaded n then too many multi threaded performanace

Gilles

9:27 PM

So I'm looking for a place to put up a PDF file for public viewing, and I thought, hey, what about Ubuntu One

George Edison

I'm not sure I quite understand what you're saying...

Gilles

First step: the Ubuntu One into page

Octavian Damiean

@Gilles Good choice. :)

Gilles

UX fail: where's the link to the Ubuntu One website?

I finally found it… by searching in the page source. That's not really user-friendly, and this is a die-hard unix command line user speaking.

George Edison

one.ubuntu.com - found by guessing

dragosrsupercool

9:29 PM

@OctavianDamiean: I am trying to propose a non blocking approach using just two threads.. this way we get relief of the blocking and the cost to run too many threads,, , why you think it wont be a good idea?

Gilles

@GeorgeEdison yup, I guessed too, then searched the source, then found where it was on the page

Octavian Damiean

Well in theory the idea is not bad but I just don't think that it is practicable.

dragosrsupercool

@GeorgeEdison: its ok :)

Gilles

That orange bar looks like an ad. I hadn't even considered it.

Jorge Castro

I think the use case is that you'd right click on the file and get the URL that way

dragosrsupercool

9:30 PM

@OctavianDamiean: With direct socket programming it did work nice.. Lets see if this can work for CURL... Still no answer on how I can implement this using curl lolz :p

George Edison

cURL can issue multiple requests in a single invocation, IIRC.

Octavian Damiean

Right, I'm not a C programmer so I can't really help there.

George Edison

I'm not either... C++ all the way :)

Octavian Damiean

is just an Android applications developer

dragosrsupercool

@GeorgeEdison: I know.. But that is blocking approach again

Jorge Castro

9:32 PM

I am going to see Episode I in 3d today

George Edison

@dragosrsupercool Not if you supply it with enough URLs.

Jorge Castro

heresy, I know

George Edison

It will fetch them in parallel.

Octavian Damiean

@JorgeCastro heh, Enjoy!

dragosrsupercool

@OctavianDamiean: language is not the barrier cURL is.. if you can help me in any language over curl network that would be great!!

George Edison

9:33 PM

Why do you need to use cURL instead of an alternative?

dragosrsupercool

@GeorgeEdison: I have worked on curl projects with million urls and did all types of experiments thus I am saying that blocking statement as an observation than assumption ;)

Octavian Damiean

Well I didn't work with libcurl so I'm not really able to help. All I can do is try to help with cURL (the application) itself which is not really what you need in this case.

dragosrsupercool

@OctavianDamiean: Anyways still thanks for your concern and interest :)

George Edison

@dragosrsupercool I don't know how you were invoking cURL, so I can't comment to that.

dragosrsupercool

@GeorgeEdison: cURL is the finest stable stuff to crawl.. Qt is not.. thats why.. .

George Edison

9:36 PM

@dragosrsupercool That's an opinion, not fact.

dragosrsupercool

@GeorgeEdison: Its fact!! bench-marking is how you can see this.. .

George Edison

@dragosrsupercool What benchmark?

N.N.

I went ahead and wrote answers to the three status-complete questions that are treated as unanswered in trying to get them treated as answered.

George Edison

@NN Great, upvoted.

dragosrsupercool

@GeorgeEdison: Make 1 of yours and see the difference.. As Octavian said Observations is the better way to comment than assuming things.. theories are fascinating but this is not how frameworks are chosen ;)

N.N.

9:39 PM

@GeorgeEdison cheers. It feels good to clean!

George Edison

@dragosrsupercool I'm sorry? I don't understand what you're trying to say?

dragosrsupercool

@GeorgeEdison: I am trying to say is that from last three hours you are presenting theories, your guesses and assumptions than experience or observations.. Anyways I really appreciate your time and energy

I must go now.. I was nice talkig to you all... . :)

George Edison

@dragosrsupercool I didn't present any theories - I just recommended frameworks and tools from experience.

Gilles

9:58 PM

What are the official Ubuntu colors? e.g. the exact shade of orange? Or is there none? I seem to find lots of variations

Jorge Castro

design.canonical.com/brand

look under "web guidelines"

the PDF there has the exact shade of orange

Transcript for