The Whiteboard

12:01 AM

@durron597 variety of approaches, depends on the technology. In MSSQL you can join across DB's, and even across servers, but the performance is never great. Typically anymore I aim for doing the data aggregation in application code treating my DB as 1 data source through a repo, and the other as another data source through a repo (which should point at a web service to abstract changes in the DB I don't control ideally managed by the people owning that DB)

alternatively, you can use ETL and just have batch jobs crossing data back and forth to get the relevant pieces into both databases that each needs

or live real-time data maintenance: Somebody updates entity X which has data in both DB's, the application updates both DB's.

Across databases - depending on the DBMS - you aren't going to get DBMS enforced foreign key constraints. You simply must manage constraints at the application level then in most cases.

So my favored approach: A team that owns a DB with data I need creates a REST service for CRUD on their DB. In my application I write a repo against that REST service, and a repo against my DB, and let the composition layer above the repos manage the data synchronization, atomicity of transactions if necessary, aggregation, etc.

They should maintain their REST service because it's not my DB - I don't want my repo breaking every time they make a DB change.

(replace REST with whatever reasonable IPC approach makes sense given the constraints of the system - performance or otherwise may push towards other IPC approaches for instance)

durron597

12:16 AM

@JimmyHoffa is ETL a concept or a tool?

i thought it was a concept but people talk about it like it's a tool

Jimmy Hoffa

people mix it up because there's a ton of tools (very large ones) that do it

It is a concept though. You can do it by hand, by one tool, by many tools, at the end of the day it is a fairly simple concept that most everyone runs into pretty quickly in software, so it is what you think it is: Take data from place A, change it to fit place B, load it into place B

durron597

it's different database platforms and different owners. ETL is currently what we're thinking about

i don't think people are going to create CRUD REST APIs for us

Jimmy Hoffa

Lots of companies entire system is run wholey by ETL tools because there are many that are so large they'll touch on anything and everything a business could want to do with wizards and such

durron597

so in our situation, if we aren't going to get people to write REST apis for us

Jimmy Hoffa

@durron597 plausibly not, you could create them and hope to be notified when they're going to make breaking DB changes. You could also just ETL the data over or directly access it in real time if that's a reasonable approach

durron597

12:19 AM

and we don't really have capacity to write an application that does a lot of the joining work for us

so the right thing to do is ETL?

Jimmy Hoffa

you could just create a repo against that DB just like you have a repo against your own using JDBC or whatever

@durron597 ah yeah, given that, ETL is probably a best bet

you can likely find tools that will make it easy, depending on how broadly available tooling is for the given DB platforms

durron597

@JimmyHoffa this is Amazon, there's an internal ETL tool

Jimmy Hoffa

haha well there ya go. Also the concern with ETL is frequency: How stale can the data be, before your application degrades or doesn't serve it's primary purpose?

ETL is typically a heavy weight operation so not something you want to run frequently. You have to start doing more fine-grained synchronization activities if you need highly current data in both sides

durron597

@JimmyHoffa I think we'll start with ETL and possibly move to an application based pipeline late

we just need to have something functional ASAP and then work on things like highly concurrent reports lter

later

Aaron Hall

In CLOS you can construct objects that haven't been defined yet.

user55340

12:52 AM

user55340

@KitZ.Fox you know you want more reps...

user55340

0

Q: I'm done with my coding from Agile perspective

Scenario: I'm working in an Agile environment. The dev environment has not been configured yet, and I'm told to code a piece of an application. I code the module and write appropriate unite test's for it as part of the coding task, and test the code offline in my own box. There is also a testi...

enderland

1:12 AM

@durron597 ETL tools are more or less designed for what you are describing - assuming you want to make reports?

@MetaFight I'm watching your upvotes roll in on that agile post. lol

MetaFight

yeah, I'm surprised at the response.

enderland

especially since you say "meh, wahtever" :P

MetaFight

that's me in a nutshell.

enderland

it was funny, since in the process of reading the question/answer I saw several upvotes on your answer. hah

MetaFight

this is my most valuable contribution to the site today:

I've occasionally considered myself a code guerrilla. — MetaFight 4 mins ago

user55340

2:14 AM

@MetaFight under cover refactoring... slipping that 'remove whitespace from ends of lines' format in with another "Adding braces where needed" commit message in Python.

Robert Harvey

TIL there are websites that push a mound of Javascript out to the browser to render the page.

Needless to say this makes them somewhat resistant to scraping.

user55340

There is "Javascript renders everything", there is "the page is rendered, but has no data, Javascript fetches everything"

Robert Harvey

It's not Angular, or anything nearly that sophisticated.

In fact, it's quite possibly the ugliest web page I've ever seen.

user55340

Sometimes is also "this is a ton of code for job/contract security because the client will never be able to change anything without paying us"

Robert Harvey

Here is a sampler.

egov.sos.state.or.us/br/…

user55340

2:20 AM

or.us: Could very well be in the "contractors did this and don't want anyone to ever be able to change it"

Robert Harvey

Could be. Most of the other states are reasonably sane, though there is one that inexplicably has a Recaptcha on it.

New Jersey actually charges for this information. What a racket.

user55340

@RobertHarvey From the days when each request was a paper request, they needed to charge a fee to avoid getting "person asked for 1M printed pages of data". The laws about requiring the fees are rarely revoked.

user55340

Also, even if the department wanted to revoke the fees for the data, they aren't the ones that make the laws - they just implement them.

Robert Harvey

Perhaps. But they are the only state that has a fee.

I remember going to a trade show in New Jersey once, as a vendor. The Teamsters charged $150 to move our booth materials 100 feet from the loading dock to our booth location. We weren't allowed to touch them until they were delivered.

Heh.

user55340

blame @JimmyHoffa

user55340

2:25 AM

btw, have you seen the unofficial blog I've been dabbling in?

Robert Harvey

Lemme see, lemme see.

user55340

the-whiteboard.github.io

user55340

And the github repo... github.com/the-whiteboard

user55340

Topics to write on: github.com/the-whiteboard/the-whiteboard.github.io/issues

Robert Harvey

Nice.

user55340

2:27 AM

I'm going to point out I'm rather disappointed with SE not trying to do this for the sites.

user55340

Its really easy.

user55340

And github has a CI server running on the backend for jekyll sites.

Robert Harvey

The blogs were never a real priority for them.

user55340

This took 30 minutes to set up once I said I'm setting it up.

user55340

They've got their blog in github.

user55340

2:29 AM

I'm building it on the idea from Room 15 (Android SO chat rom): room-15.github.io/blog

Aaron Hall

:)

user55340

For this purpose, the local wiki and issue tracking are killer features.

Robert Harvey

Oh, blargh.

Keep the "m" Prefix

Even a leading underscore makes more sense.

user55340

note: not going to add Discus to the-whiteboard -- we've got a place to discuss things. Right here.

Robert Harvey

Is Discus the Github chat?

user55340

2:33 AM

Discus is a third party that makes embeddable comments.

Robert Harvey

Oh.

user55340

That part of it.

Robert Harvey

Nice example.

So we're not inviting the great unwashed masses then.

user55340

disqus.com

user55340

2:34 AM

Nope. Well, we are - I mean its not that hard to get 20 rep.

user55340

But I feel that comments on the posts distract from the post themselves.

user55340

And it requires moderation of that

Robert Harvey

@MichaelT So you say.

user55340

And I don't want to have to deal with "The evil moderators are deleting all my comments

user55340

@RobertHarvey Go to World Building. Ask how many zombies it would take to take over the world.

Robert Harvey

2:36 AM

user55340

Go to ~~English.SE~~ EnglishLanguageLearners.SE and ask how to pronounce "disqus"

user55340

Go to outdoors and ask what ~~sex~~ gender the chipmunks are.

Robert Harvey

Ah, yes. World Building. The site where everything's made up and the points don't matter.

user55340

Thing is, 20 rep isn't that hard to get somewhere if you want to participate in chat.

user55340

Shoot, could go ask how to tie your shoes when you're holding a dog in LifeHacks.SE.

user55340

2:40 AM

Anyways... still don't want to deal with it there.

user55340

3:34 AM

Ouch - I70 in Glenwood Canyon to be closed for weeks from a rock slide bellinghamherald.com/news/nation-world/national/…

user55340

Not a lot of ways to go around that: google.com/maps/@39.5666993,-107.1920623,12.44z

user55340

3:57 AM

@AaronHall ps. I hate history questions. But I'll be damned if I let people write crappy answers that are guesses to it.

Aaron Hall

Yeah, but was that the first one?

user55340

@AaronHall en.wikipedia.org/wiki/Microprocessor#History - start there and go forward.

user55340

The 6100 was very odd. Only had 3 bits for instructions - 8 possible instructions total. Talk about RISC.

user55340

It actually had more - but they were all "macros" on top of that.

Aaron Hall

:)

user55340

4:01 AM

homepage.cs.uiowa.edu/~jones/pdp8/man/mri.html

user55340

000 - AND.
001 - TAD.
010 - ISZ.
011 - DCA.
100 - JMS.
101 - JMP.

user55340

Though they might be able to make an argument for the PDP 8

user55340

homepage.cs.uiowa.edu/~jones/pdp8/man/micro.html

user55340

> On all models of PDP-8 prior to the PDP-8/E, the group three instructions were only implemented if the EAE option was present. If this option is absent on such machines, all group three instructions become no-ops.
On the PDP-8/E and following models, CLA, MQA and MQL instructions are implemented in the standard CPU, while the other operations are implemented in the optional EAE. Portable code should avoid relying on the availability of any of these instructions.

user55340

> 7501 - MQA or Multiplier Quotient with Accumulator

MQA ors the contents of the MQ register into AC. Usually, the microcoded combination CLA MQA is used; this first clears the accumulator, so the result can be viewed as a load operation.
7421 - MQL Multiplier Quotient Load

MQL loads the contents of the MQ register from AC and then clears AC.
The microcoded combination CLA MQL first clears AC, so the result is to clear both AC and MQ. Some assemblers supply the mnemonic CAM for this combined operation.

Duga

5:59 AM

This question probably belongs on programmers.stackexchange.com — Matthew Herbst 11 secs ago

PreferenceBean

10:17 AM

keming lols: someecards.com/news/so-that-happened/…

Duga

10:36 AM

I think this question belongs more to codereview or programmers. — WorldSEnder 57 secs ago

PreferenceBean

10:53 AM

Easy solution, get a VPN so you can spoof your location as North Wales! — yshavit 12 hours ago

heh

14

A: Numbers within strings?

(and please don't post code as pictures)

LOL

PreferenceBean

12:24 PM

We have a piece of software that, while being packaged, requires specific versions of third-party firmware to be pulled in from somewhere to be bundled into the package. The "specific versions" will change over time.

In order to allow for full build automation, I'm going to set up a webserver with these third party resources hosted on them, e.g. http://<host>/resources/<name>/<version|"latest">/<filename>; our software's build script will then ask a lookup API (Python script probably — keeps builds of historical versions working even if the resource storage location/mechanism changes) to get the URL of <name> (version <version>).

Now, I can set up a basic public wwwroot (with a little magic to auto-generate the "latest" symlink directory) ... but what's the common way to do this? Is there a commonly used software page that manages resources like this? As I type this I wonder why we don't put them in SVN and tag, despite not being source. Hmmz.

Ixrec

are you talking about "vendoring" your dependencies?

PreferenceBean

not sure - depends what that is

Ixrec

vendoring dependencies basically means checking them into your source tree, with the goal of making builds repeatable even in the face of dependency updates

PreferenceBean

ok well sort of but sort of not

Ixrec

you can find many blog posts explaining why this practice is either the spawn of the devil or absolutely mandatory for anyone who's serious about product stability

PreferenceBean

12:34 PM

I guess one option is to do that (and we've done it on a smaller scale in the past without many problems)

but I'm trying to come up with a better practice, such as hosting all the third-party deps on an internal webserver

and having the version-controlled build scripts obtain the resulting URL for a specific version of a specific dependency through a version-controlled API layer

I guess the real question is whether storing the deps on a webserver in the first place is considered okay, or whether I should just stick with SVN for everything. ^/dep-name/tags/v1-3-02/ and external it surely

Ixrec

personally, I'm failing to see what the benefit is of storing those deps on an internal server as opposed to straight-up vendoring them

unless the goal is for many different projects from separate repos to all have reliable access to them

PreferenceBean

one benefit is availability to non-software folks (say, hardware testing, who love to be able to flash these f/ws independently of anything else) - they won't be using SVN but a webserver's probably okay

(atm they take the binaries off a horribly organised network share directory)

it would also be cool

the blog posts I've read so far on the subject seem to assume an open-source ecosystem where everyone's version control is accessible through Git

how myopic

another downside of SVNing it is that it would be weird. copy newly received build to trunk, tag. copy newly received build to trunk, tag. copy newly received build to trunk, tag.

Ixrec

1:08 PM

in my ideal fantasy world, if I chose to vendor dependencies there would be a daily Jenkins job that checks if a new version is available and if so generates the PR for the update, and all PRs have a webhook to run tests so I can just look for the checkmark and hit merge

but the hardware testing thing is a good point

RESTful CRUD microservices it is

PreferenceBean

1:33 PM

@Ixrec these are binaries delivered by email by vendors though

there won't be anything automated about the dependencies ;)

@Ixrec wheee

Ixrec

@PreferenceBean wat

PreferenceBean

@Ixrec GNSS chip vendors don't publish their code on github

Ixrec