« first day (1732 days earlier)      last day (3584 days later) » 
00:00 - 20:0020:00 - 00:00

Bob
Bob
20:01
@CanadianLuke Easy enough to check - get him to provide the source :P
Or, have him run the same command
Bob
Bob
@CanadianLuke I still want to see what exactly that program is doing.
Hm
Thing is when I ran it twice on my own hardware (CIFS share, Windows 8.1 client), I got 15 & 14 files/sec, which matched his own Windows performance on his own hardware, Windows 8.1 client, Debian server.
And running the same code on his Mac client yields MUCH faster results
Bob
Bob
Would it do any harm to release the source? It should be a fairly simple program.
Probably wouldn't be. I can ask
Bob
Bob
20:06
Also, if it's several orders of magnitude faster - I would suspect a cache.
Though I just realized that he is writing out a file list to a local file at the same time as he's reading from the share, maybe his program is performance constrained on local disk access since he's doing that.
Curious if Java will disable output if I pass it nul as an output file
Trying that out
Like I said, I ran a dir on Windows, and got 16 seconds
Seems nul is a specially handled filename in Windows... it looks like it won't let you create a file named that?
Yeah. I'm running with -O nul on his program. I'll try that as well
0
A: Unable to rename a folder or a file as 'con'

DragonLordAs mentioned, you can create and manipulate files and folders with reserved names on the command line by using a device or filename namespace such as \\.\C:\NUL, but look at what happens when you try to access such a file or folder through Windows Explorer: Any access to an object with a rese...

Curious if it makes a difference. Dunno how long it would take to do just a dir /s on my remote share.
@DragonLord Yeah, makes sense. Is probably a check in the API so if you bypass the API somehow, you'll get it.
Like when you have a file named & in Linux or something.
You have to be careful how you access it?
20:21
And a /a and /d
I'll just do a dir. For all I know, using nul as an output file triggers some bug in his code. I don't have the source :P
Bob
Bob
@BenRichards The Win32 API doesn't allow it. The kernel doesn't care. If you were going via the POSIX layer (e.g. SUA) or otherwise bypassing Win32, then it's allowed.
@BenRichards I'm writing a test program of my own gimme a sec
@CanadianLuke does /d force it to just get the name? I see it outputs as a column.
Ok
@Bob Right.
I would like a Windows equivalent for GNU time
The /d does... uh... I think something with directories? Crap, now I gotta look
I can just use Cygwin but I don't think it's installed
Bob
Bob
20:23
@BenRichards PowerShell Measure-Command
Nah, /d is for formatting it as a column
Bob
Bob
106
A: Timing a command's execution in PowerShell

Keith HillYup. Measure-Command { .\do_something.ps1 } Note that one minor downside of Measure-Command is that you see no stdout output. If you want to see the output, then you can use the .NET Stopwatch object e.g.: $sw = [Diagnostics.Stopwatch]::StartNew() .\do_something.ps1 $sw.Stop() $sw.Elapsed

@Bob Cool, figured Powershell had something like that, but I didn't know what it would be called
Bob
Bob
Time: 00:00:01.4512453
Dirs: 193; Files: 1388
Files per second: 956.419979447996
Hm, might need a bigger dir.
Ok, this one should be gigantic.
Note that this is from Win8 to Win7 over Wi-Fi.
Hardly the most optimal. And also SMB2.
Ok. We're doing CIFS, though. Just so you know :)
Bob
Bob
20:26
Time: 00:01:02.8827280
Dirs: 2768; Files: 170706
Files per second: 2714.67230238485
That didn't take very long.
I'm running Measure-Command { dir -Recurse \\dlink-034417\Ben }
Lots of files. LOTS of files. So we'll see what I get
Bob
Bob
@BenRichards Console output?
Make sure there's no console output.
Console output is slow.
Measure-Command apparently suppresses console output. I know.
Bob
Bob
Also, FYI, this is my test program:
Stopwatch sw = new Stopwatch();
VistaFolderBrowserDialog dialog = new VistaFolderBrowserDialog();
dialog.ShowDialog();
string root = dialog.SelectedPath;
string output = Path.GetTempFileName() + ".txt";
uint dirs = 0;
uint files = 0;
using (StreamWriter writer = new StreamWriter(output))
{
	sw.Start();
	foreach (FileSystemInfo fsi in new DirectoryInfo(root).EnumerateFileSystemInfos("*", SearchOption.AllDirectories))
	{
		if ((fsi.Attributes & FileAttributes.Directory) == FileAttributes.Directory)
nice and simple
That's C# code.
Bob
Bob
20:29
Yes. Yes it is.
OK, I'm running @BenRichards's command
I'm trying to see how I can get powershell to count number of files in the list. I'm sure it's easy XD
Bob
Bob
1. I don't have a Java environment on this laptop, 2. I don't intend to install a Java environment if I don't need to, and 3. I wouldn't use Java even if it was an option.
Bob
Bob
Anyway, this is supposed to benchmark SMB file/dir enumeration.
20:31
40.735 seconds
Bob
Bob
It doesn't matter what language it's done in, assuming it goes though every file correctly.
And this one does.
Or not.
1
Q: Powershell get number of files and folders in directory

BeninjaI am going to write a powershell script that gets various statistics on the dir it is in. If I do get-childitem I get this d---- 06/23/2014 2:49 PM asdf; -a--- 06/23/2014 2:49 PM 23 New Text Document - Copy (2).txt -a--- 06/23/2014 2:49 PM ...

Bob
Bob
This is also why I'd like to see the Java code, see if there's any glaring mistakes.
@BenRichards Hm? There's .Files in PS4 IIRC
@Bob I'm still learning
Bob
Bob
@BenRichards gci -File -Recurse
20:35
That's basically dir -File -Recurse, right?
Bob
Bob
measure-command { $files = gci -File -Recurse; write-host $files.Count }
Get-ChildItem -File -Recurse \\dlink-034417\Ben | group @{Expression={if($_.PSISContainer){'Folder'} else {'File'}}} -NoElement
Bob
Bob
I don't like using dir in PS
It maps to the same command
Bob
Bob
it's not the same as the cmd dir. Better not to confuse the two.
20:36
yeah
Bob
Bob
... wtf is that group o.O
just run the one I provided above
probably faster too, since you won't be filtering inside a script
unless you want to count folders too?
I can simplify it
Oh yeah, your thing
I just pulled it from a SU answer (linked it above)
I still think it's pretty awesome how Powershell passes data around encapsulated in objects rather than as plaintext.
Bob
Bob
Alternatively, run the C# one further above :P
at the end of the day, it probably does the same thing PS does
More work than I want to do :P
Bob
Bob
shrug it's pretty much paste in LINQPad and go
20:39
Don't have that
I do have Powershell though :P
I have Visual Studio but that's more than I need for this task :P
So what script or command are we going to use to measure the time?
Measure-Command { $files = gci -File -Recurse \\remoteshare\folder }
Then $files.Count gets you the number of files, and just divide that by the minutes.
I get about 424 files/sec
I bet his program is local disk I/O constrained
And the Mac has better performance or a larger output buffer
28166/38 seconds
741.2 files per second
Bob
Bob
@BenRichards PS can run C# :P
@Bob Figures. :P
Bob
Bob
20:46
$source = @"
using System.Diagnostics;
using System.IO;
using System;

public static class P {
public static void test() {
Stopwatch sw = new Stopwatch();
string root = "C:\\Intel\\";
string output = Path.GetTempFileName() + ".txt";
uint dirs = 0;
uint files = 0;
using (StreamWriter writer = new StreamWriter(output))
{
	sw.Start();
	foreach (FileSystemInfo fsi in new DirectoryInfo(root).EnumerateFileSystemInfos("*", SearchOption.AllDirectories))
	{
		if ((fsi.Attributes & FileAttributes.Directory) == FileAttributes.Directory)
^ you can paste that directly into PS
don't mind the shitty formatting
@Bob Not worth it. I just ran your command and then did the math in-line in powershell :P
Bob
Bob
shrug
21 mins ago, by Bob
Time: 00:01:02.8827280
Dirs: 2768; Files: 170706
Files per second: 2714.67230238485
Probably depends some on the folder structure on the other end, but that's certainly not slow.
Yeah
I asked him to throw his code up on pastebin or something so we can have a look. My best guess is it's I/O constrained and he's just measuring how aggressively the OS flushes an output file buffer to disk.
@DragonLord CON!!!!!!!!!!!!!!
I keep getting 'Path too long' errors
20:54
I dunno, maybe you need the semicolon at the end? I had the write-host call after as @Bob had in his version but it didn't make it to the actual console output because Measure-Command eats any output from its command block.
Bob
Bob
@BenRichards write-host writes directly to the host, as the name implies
it's not stdout.
I didn't see anything IIRC
Bob
Bob
it's at the very top
Oh wait, yeah, there it is
My bad :P
Bob
Bob
immediately under the prompt, above the measure output
20:56
measure-command { $files = gci -File -Recurse; write-host $files.Count }
@BenRichards The default output for that command is horrid. :P
Bob
Bob
it's just a timespan
do a | select TotalSeconds if it makes you happy
or (blah).TotalSeconds
measure-command { $files = gci -File -Recurse | out-default }
Anyways, I just used it to get the total seconds and then divided that into the files count.
21:02
Measure-Command { dir -Recurse . | Out-Default }
Navigate to the folder and use that :P
Will that list everything? Because I don't want that. Slow :P
I got my numbers already anyway :P
It'll list everything in the current directory.
Bob
Bob
@MichaelFrank the point is measuring the enumeration speed, not the console output speed
37 mins ago, by Bob
Console output is slow.
never ever benchmark anything with console output enabled, unless you want to benchmark the console itself
it's slower than just about anything else you'd do in a microbenchmark
this goes for most *nix terminal emulators too
Ahh, I thought someone was complaining that measure-command supressed output.
Oh nope
:P
@Bob @CanadianLuke here's his code:
http://pastebin.ca/2995871
http://pastebin.ca/2995872
http://pastebin.ca/2995874
Hmm. He's flushing every time he writes to the output.txt file, during the enumeration of the directories.
21:27
Dammit, NetBeans is hung scanning my SSD
Bob
Bob
@BenRichards that's not a simple enumeration
delete ops, listing files multiple times...
and, yes, the constant flushing
depending on how hard the flush is (how hard does it try to force the file to be written to disk) and the OS implementation, that could easily introduce milliseconds per op delay on a HDD
Yeah
It's doing a lot more than I'd expect. I wonder if he adapted code that was used for other things.
Bob
Bob
shrug looks overcomplicated
compare the snippet above :P
and I feel even that was overly complex
in the worst case, I can easily see 10ms or more delay per flush
that would effectively limit it to a max of 100 files per second
it really depends on just how much the OS honours the flush, how much disk contention there is, and how fast the disk is
some OSes won't actually flush to disk
@Bob You wrote a tool? Is it what you pasted here?
Bob
Bob
some disks won't actually flush to actual storage (on-disk cache)
@BenRichards what?
1 hour ago, by Bob
Stopwatch sw = new Stopwatch();
VistaFolderBrowserDialog dialog = new VistaFolderBrowserDialog();
dialog.ShowDialog();
string root = dialog.SelectedPath;
string output = Path.GetTempFileName() + ".txt";
uint dirs = 0;
uint files = 0;
using (StreamWriter writer = new StreamWriter(output))
{
	sw.Start();
	foreach (FileSystemInfo fsi in new DirectoryInfo(root).EnumerateFileSystemInfos("*", SearchOption.AllDirectories))
	{
		if ((fsi.Attributes & FileAttributes.Directory) == FileAttributes.Directory)
21:37
Yeah
Bob
Bob
took all of 10 mins
though, it is Windows-only at the moment
fairly trivial change to make it generic - remove the Ookii VistaFolderBrowserDialog
was just easier to use it
in fact, I'd expect the C# impl to be slightly slower because it theoretically performs filtering (wildcard *)... but possibly faster if it hands off to a native Win32 impl?
It could be a simple python or perl script, even. I've made scripts that recursively walk through a directory before. Just keep low overhead or even only count time when doing the directory list itself.
Bob
Bob
nope, looks like it recurses in managed code => referencesource.microsoft.com/#mscorlib/system/io/…
Bob
Bob
and mono does the same
so you could probably chuck the C# impl into a *nix system and get similar results
21:43
I'm disappointed your test() function doesn't take parameters for directory and stuff :P
Bob
Bob
@BenRichards the original was just a sequence of statements in LINQPad
had to wrap it in a class + method to run in PS
Bob
Bob
and, of course, you could do something similar in Java
don't look at me though :P
This is how I implement a recursive directory search in Perl: github.com/benrichards86/Verify/blob/master/TestIndex.pm#L46
Obviously it'd be modified slightly to just count, rather than calling a handler on individual files
And not filter on a file extension :P
If I wanted to do that in a benchmark program, I'd probably modify a few things.
Bob
Bob
@BenRichards Python impl:
def rdir(parent):
	root, subdirs, files = os.walk(parent);
	return len(files) + sum([rdir(d) for d in subdirs])
21:50
nods
22:00
So he says he cares about extracting metadata, not just file list.
I think problem is he's dumping it to file, and doing other stuff with it, rather than ignoring it.
Extract it, sure, but doing other stuff with it is unnecessary and adds more variables.
He's gonna try and retool his java program.
Bob
Bob
@BenRichards Ok, so he wants modified time and length?
I guess so.
Bob
Bob
Ok.
Stopwatch sw = new Stopwatch();
VistaFolderBrowserDialog dialog = new VistaFolderBrowserDialog();
dialog.ShowDialog();
string root = dialog.SelectedPath;
string output = Path.GetTempFileName() + ".txt";
uint files = 0;
using (StreamWriter writer = new StreamWriter(output))
{
	sw.Start();
	foreach (FileInfo fi in new DirectoryInfo(root).EnumerateFiles("*", SearchOption.AllDirectories))
	{
		files++;

		writer.WriteLine("{0}|{1}|{2}", fi.FullName, fi.LastWriteTime, fi.Length);
	}
	sw.Stop();
	writer.WriteLine("Time: {0}", sw.Elapsed);
that should do it
I'm no longer going through directories because he doesn't do anything with them anyway
(I'm relying on EnumerateFiles to do the recursion for me)
also note that this is running in debug mode, not release/optimised mode
there, done
compare, previous results:
2 hours ago, by Bob
Time: 00:01:02.8827280
Dirs: 2768; Files: 170706
Files per second: 2714.67230238485
this time, same dir:
Time: 00:01:09.2213603
Files: 170706
Files per second: 2466.08849147392
Bob
Bob
slower, but not that significant
it's in the same ballpark
22:15
Cool
Bob
Bob
@BenRichards also, depending on how Java resolves his property access, he could easily be performing many independent requests per iteration
@BenRichards A rather minor change equals three accesses per it rather than one:
writer.WriteLine("{0}|{1}|{2}", fi.FullName, File.GetLastAccessTime(fi.FullName), new FileInfo(fi.FullName).Length);
so far it's been running for 3x longer and counting
4x and counting
basically, either the Java API he's using is shit (esp. if it doesn't provide other options) or the way he's using it is greatly suboptimal
Submit it to codereview.SE
22:31
@BenRichards woof
Bob
Bob
8x and counting
@BenRichards Found your problem.
...
17 mins and counting
this is ridiculous
You two should have a competition to write the slowest script!
Rules: As slow as possible while still actually working towards completion. No wait commands, the script must always be doing something.
Bob
Bob
@MichaelFrank pretty sure there's one of those on Programming Puzzles & Code Golf already
hey, it finally finished
Time: 00:18:24.2081300
Files: 170706
Files per second: 154.59585504048
@BenRichards @CanadianLuke ^ that's about right for the slow tests, yea?
and I haven't even added the slow enumeration of dirs yet
22:47
@Bob When my friend and I was running his original test program, it was clocking ~14 files/sec.
Bob
Bob
basically, the fast version makes few requests. the slow version shoves in individual requests three times for every file
Bob
Bob
@BenRichards I'm perfectly willing to blame that on other faults in his benchmark
e.g. this crap
So every call to fi queries the SMB server again?
Bob
Bob
                if (listOfFiles[i].getAbsoluteFile().list() != null) {
                    if (listOfFiles[i].getAbsoluteFile().list().length == 0) {/*Directory empty*/
                        listOfFiles[i].getAbsoluteFile().delete();/*delete the directory*/
                    }/*Empty list*/
                }/*delete directory*/
that smells horribly inefficient
22:49
Yeah.
Bob
Bob
@BenRichards in my original code, I just retrieve the properties already obtained in the initial enumerate
Benchmark coding is its own skillset. Coding for efficiency isn't something that's necessary for most coding projects anymore due to the hardware being so powerful these days.
@Bob I'd do that, yeah.
Bob
Bob
in the slower version, I explicitly take the file path and grab the modified time and size independently
the file obj => string => file property transformation requires an extra independent request every time
and that's what the Java code was doing
Makes sense.
Bob
Bob
@BenRichards No, minimising round trips is still a thing. And very important for any webdev.
22:51
I don't know the Java API very well, so I wouldn't know that, but usually if I care about speed, I read everything at once to RAM and then query that.
Bob
Bob
An extra round trip can easily add an extra half second or more.
@Bob I suppose, I don't do web dev :P
And I know that disk queries are also very slow, comparatively
Especially for HDDs.
Bob
Bob
@BenRichards In this case, it's not really the API at fault, I think.
            if (listOfFiles[i].getAbsoluteFile().length() != 0) {
                writerSafe(out,(fileIO.makeNameParsed(listOfFiles[i].getAbsolutePath().replaceAll("\\p{C}", ""), fo) + "|" + fileIO.getTimeSeconds(listOfFiles[i].lastModified()) + "|" + listOfFiles[i].getAbsoluteFile().length() + "\n"));
            }
But yeah, when I code, minimizing round trips is one of those things that I do by habit because doing more round trips than is necessary just feels wrong to me.
Bob
Bob
if he just cached listOfFiles[i].getAbsoluteFile()'s result, that'd probably help
22:53
nods
Bob
Bob
Then again, I checked the JRE lib code and it looks like those method calls are actually on-demand
e.g. File.getFileTime() doesn't return a cached result
so it still could be the Java API's fault
either way, I think we've quite conclusively proven that SMB itself isn't the main issue here
as for why it's faster on other OSes?
I wonder if the JVM on Mac or Linux caches the entire set of metadata locally on the first request.
Bob
Bob
shrug if I were to guess, the Java API may be implemented differently there
I could only find the Windows implementation, and there getFileTime passes straight into native code
I wonder if he's using open source java there and official java on Windows?
Bob
Bob
@BenRichards Official Java is based on openjdk now IIRC, but the implementations are still different on different platforms
22:55
Ah didn't know that.
I invited him here
Bob
Bob
just to be certain, I'll reimplement this with a managed loop manually
gimme a sec
Bob
Bob
void Main()
{
	Stopwatch sw = new Stopwatch();
	VistaFolderBrowserDialog dialog = new VistaFolderBrowserDialog();
	dialog.ShowDialog();
	string root = dialog.SelectedPath;
	string output = Path.GetTempFileName() + ".txt";
	uint files = 0;
	using (StreamWriter writer = new StreamWriter(output))
	{
		sw.Start();
		files = GetNumFiles(new DirectoryInfo(root), writer);
		sw.Stop();
		writer.WriteLine("Time: {0}", sw.Elapsed);
		writer.WriteLine("Files: {0}", files);
		writer.WriteLine("Files per second: {0}", files / sw.Elapsed.TotalSeconds);
I'm expecting this to be somewhat slower, maybe 2x
mostly cause I'm separating the dir and file retrieval
Bob
Bob
that's a limitation of the .NET API - if you get both at the same time in a FileSystemInfo, you can't get the size out of that
23:03
@Bob I haven't seen the slow speeds yet
Bob
Bob
...huh
Time: 00:01:10.8189033
Files: 170706
Files per second: 2410.45811281294
surprisingly, it's not any slower
well, one second. less than 1% :P
so, yea, this proves that a recursive EnumerateFiles doesn't really do anything special that I can't do in managed code
Oh dang, he doesn't have the minimum 20 rep to chat I guess :(
Had to create a new account
Bob
Bob
chuck a quick'n'easy answer somewhere :P
Told him to :P
Bob
Bob
or get @CanadianLuke to add him to approved writers in this room
23:09
Hmm? What? Who?
Bob
Bob
hm, probably better not to
Bob
Bob
it's theoretically possible for a mod to give <20rep write access to a room
but we don't really do that for RA, judging by the short list of people on it shrug
That's circumventing the system and probably not a good idea though :P
Sets a bad precedent.
If it's to get help with improving a question/answer on the main site, that's what it's for
Bob
Bob
23:10
@BenRichards the rep requirement is more of an anti-spam measure than anything else I believe
Yeah I figured.
Bob
Bob
just get him to answer something and link the answer here :P
we'll upvote it if it isn't completely crap
Who is it?
I don't know his username yet.
Question link?
23:14
And searching the user list doesn't show anything I recognize. Does it only update periodically?
He didn't post a question or anything on here. I was chatting with him via twitter and offline. I brought the question here since I didn't know enough.
Brought it up here anyway.
Ahhhh. Yea, he'd need a user setup... And shouldn't be used unless it's for clarification on an existing question (or the like)
Bob
Bob
@BenRichards @CanadianLuke just jump in here and invite him => gitter.im/RootAccessOrg/PublicChat
Oh good idea
Bob
Bob
(was tessting gitter as a potential backup site, but might as well use it for this for now)
that room should be public
Link doesn't work
Bob
Bob
23:18
...I think
Bob
Bob
try now
TThere we go
23:55
@Bob: I know @BenRichards and the guy on gitter IM, and falcon on the comms room from the same place
(Actually, I got ben and falcon on SE in the first place)
Bob
Bob
o.O
00:00 - 20:0020:00 - 00:00

« first day (1732 days earlier)      last day (3584 days later) »