« first day (1183 days earlier)   

4:20 PM
Hi Richard, I really appreciate you joining the channel!
 
I am looking over the code, but it looks like your routes already exist and are not being created in this code. So to begin I need to know if you have verified whether or not the route measures actually start at 0. Have you done the Python calculation into a double field in your route feature class with an expression of !Shape.FirstPoint.M! ?
 
yes I did it manually on the road layer
and it seems that they start at zero
but do you suggest I do it right from the script?
 
Are the lines that are being overlaid derivatives of these road lines? In other words, are you certain there is little or no variance in the shared geometry between the routes and the line events you are creating?
 
both layers are the same geographical outputs from our oracle EXOR database but wit hdifferent attributes
the PDT one is done "manually"
using a third party software where engineers pick the sections that they want to do work on
 
No you do not have to do the calculation I suggested in your script, it was just to verify the assumptions about the inputs to the script that are not part of the code.
 
4:33 PM
just for "continuity" sake, what would I have done if the values didn't add up to zero
 
The general cause I have seen where the MDomain prevents the beginning measure from being 0 while running the Create Route tool occurs when the measure range allows negative numbers and the spatial reference of the route is not metric (like mine).
I have had to expressly include the MDomain environment setting in every script that uses the Create Route tool to ensure that the domain values that avoid that behavior is replicated.
 
well with one layer whenever I do create route, it drops about 50% of the roads. Any idea though why my script is generating duplicate records?
 
It took some experimentation to find the range that worked. Something like -1000 was OK, but -100 was not. Anyway, it was just trial and error to figure out what I needed.
First of all, are these complete duplicates in every field including the measures?
 
most of the fields but yes to the measures
 
4:51 PM
Well, any difference means that the tool did not duplicate the records, it retained distinct records, which it is supposed to do. If there is overlap of the input events within an event table that have slightly different values then it is supposed to retain each of those separate records and divide not only against the other table records, but within the table records that overlap.
 
so perhaps, I should do the join up front
as it'll guarantee less chance of confusion
 
Most likely the Locate Features Along Routes is applying your M Tolerance and M Resolution settings in the background and making the end to end measures slightly different between adjoining segments. That creates overlaps with distinct records that cut up the events internally in that input table.
 
p.s. any comments on the code itself? is it what you'd consider "good qualitY" or "average"?
and thank you very much for all of your feedback sof ar. Trully appreciated
 
I think the code is clear enough and should being doing what you need. You could gain a lot of speed by using a single pass of an update cursor to do all of the field calculations on a record at once, rather than doing the update to each record one field at a time.
 
I didn't know I could do that
thanks for suggesting it
 
5:01 PM
If al of the fields are in a single table you only need a single update cursor. If your are doing a transfer between tables, rather than joining them and using the field calculator, you can read the source table into a dictionary and then use an update cursor to do the transfer out of the dictionary. With the da cursors that should give you about a 50 fold gain in speed.
 
I'm reading the documentation on that right now. IT doesnt' seem very clear but I'm sure I'll get it eventually
 
I am looking for the base code of the data transfer I wrote in Geonet.
Oh well, Geonet is being a bugger about the searches I am trying. I will get it from my work desktop later. So back to the subject of your code.
 
my email address by the way is my username here at the glorious gmail if yo uwant to share anythign that way
 
I'll send you the basic code that way and repost it on Geonet and here as my own post with better tags so I can get back to it later.
 
I really do appreciate this. Thank you very much
 
5:17 PM
So I just want to summarize what the code is doing to see if I understand it and can isolate the point where the problem has to be fixed.
Basically it looks like you are connecting to a database and transferring a subset of the data from it into a new feature class. Then you add and calculate fields. Then you convert that data to route line events. And then you overlay it with another preexisting event layer and make an event layer from that output.
You also do a few calculations on that output, presumably because now you have the data merged together.
 
we have a software that takes in a shape file as input to do some linear programming and markov chains modelling. The input layer comes from the road data provider as a geodatabase. Another Event table comes in that indicates which roads will be worked on over the next few years. I combine the shape file with event table to create one shape file to be used in teh software
 
So, on the event table not being created by this code and only being overlaid, are you sure that no measure overlaps exist between any segments, even by fractional amounts?
 
could very well be!
 
Please isolate one of the problem event segments so that we can do small scale tests and look at the inputs in detail.
Actually isolate a problem set of related events for one route rather than just a single event.
 
I'll be back in a bit headed to a meeting. I 'm not suer what you mean by isolate
 
5:33 PM
I just mean pick a single route with problems out of all of the data you are working with and select the set of input events and features that create the final events.
 
5:51 PM
Sounds good
 
6:50 PM
I figured out the update cursor. You're right it's much faster! thank for hte tip
 
7:00 PM
Just sent you an e-mail showing examples of a single table update and more importantly how to do data transfers between tables where there is a common join field (one-to-one or many-to-one). The transfer between tables is where the real speed gain is incredible.
 
thank you so much for all yoru help so far .. just by doing the non join stuff in cursors shaved about 100 secodns off of a 400 second process
 
More complex transfers are possible, including setting up multi-field join values, but they are harder to follow.
 
awesome. One thing I find is that it takes 37 seconds for arcpy, to load itself, open the geodatabase and duplicate a featureclass
is that ok?
 
That is typical in my experience.
 
Awesome. Well this is very helpful!
 
7:05 PM
Arcpy loading and geoprocessing has a lot going on behind the scenes to manage errors and memory and read environments and set up, so there is overhead using them.
 
any other recommendations you'd recommend?
 
I think that is about it, other than figuring out what it will take to get an output that meets your expectations or desires.
 
so when you say data tarnsfer between tables, you mean regular data join/
 
The data transfer code I sent replaces a geoprocessing operation where you would do a standard join and then use the field calculator to move data from the table added by the join to the editable table.
 
here's what I did for the calcs pastie.org/private/gnme9pnm1q7hwsloglzsa
 
7:12 PM
If you have ArcGIS 10.1 or above you should use data access cursors, not the original cursor code. That will speed it by by 10 times.
 
ok
 
I will send you an e-mail that rewrites the section.
 
thanks
 
awesome ! thanks
 
7:23 PM
Substitute FTR_TO_RS_TEMP for UpdateFC in the code I sent
Oops. Also indent the updateRows.updateRow(updateRow) one more level to the right
 
yup figured that out!
this way, I can do some unit tests on my calcs
jeez that was blazing fast
 
Told ya.
 
now time to update the joins
 
It is lunch time for me. I can be back in about an hour. Is that OK.
 
I might not be here, if I have any other questions I might email you or ask again here
thanks so much for your help so far though
 
7:38 PM
Ok. I'll keep an eye out for e-mails or activity here from you. I'll check back in an hour either way, just in case you are still around.
 
 
1 hour later…
8:44 PM
I am checking back just in case.
 
8:59 PM
Signing off.
 

« first day (1183 days earlier)