Root Access

Bob

10:00

So you'd do a text => JSON/XML transformation.

Once you have the data in a nicer format, worry about the SQL part.

It mgiht also make it easier to see what data you actually have to work with if you ignore all the relational aspects at first.

marcusdoesstuff

Or perhaps I should just be paying someone to sort it out for me... Might be more efficient. Hell, even pulling all the data out manually might be quicker than learning a language..

Bob

@marcusdoesstuff Unless you plan to deal with a J2EE webserver, Java has little to do with the web :P

Maybe JavaScript, but that's not really suitable here.

Python might end up being your best bet.

marcusdoesstuff

hmm messages timing out.

Bob

And would be pretty easy, too, as an added bonus.

marcusdoesstuff

Or perhaps I should just be paying someone to sort it out for me... Might be more efficient. Hell, even pulling all the data out manually might be quicker than learning a language.

my thoughts exactly... i need it all in a nice neat set of table first.

Bob

10:02

In fact, @marcusdoesstuff, mind sending me a full csv file, I'll see if I can drum up something quickly in Python.

(yes, I'm bored tonight)

Journeyman Geek

@marcusdoesstuff: as someone who regularly does unspeakable things with grep cut and the like... you're trying to turn unformatted data into formatted data. That's not trivial

marcusdoesstuff

That makes me look awfully lazy then! But maybe it's something very simple for you.

Something I am coming to realise @Journeyman

Bob

@JourneymanGeek It's actually not that hard in this case.

Assuming all this data follows the same format.

I could easily see going through it line-by-line and chuck the appropriate regex at each line.

At the very least, transforming these sections/headings into nested JSON would be simple.

marcusdoesstuff

i need to get hold of more than 1 csv file to confirm how much changes.

Bob

Then you have a nice machine-readable data format you can read and dump into the SQL database.

@marcusdoesstuff Well, it wouldn't be the final result, but it'd be a decent starting place.

Psycogeek

10:08

that boored :-) from reading a lot of MSDS seems that a lot of the data is repeated for many chemicals (or chemicals in something). A precisions compilation of every MDS out there could have single referance pointers .. Is Cancerous , Flamable A flamable b and Explosive A

Journeyman Geek

hm

marcusdoesstuff

@Psycogeek I think that may have been what I was thinking but just with different words.

Journeyman Geek

IIRC MSDS formats are VERY standard

Psycogeek

I was just trying to make stuff harder for bob :-)

Bob

@JourneymanGeek Not necessarily when in a human-readable form :P

Journeyman Geek

10:11

@Bob: I mean in a human readable form ;p

marcusdoesstuff

they are indeed @JourneymanGeek.

Bob

@JourneymanGeek It looks standard to the eye, but a stray character here or there could mess up parsing.

Really depends on where you got it from.

marcusdoesstuff

it's only the varying amount of data in some sections that makes it more difficult.

Anyway, I've badgered QC for more files.

Bob

oh hey look they call it SDS now

when did that happen

... 2012, ok

marcusdoesstuff

The CSV files are far easier to work with than the PDFs at least.

haha

Bob

10:13

@marcusdoesstuff Varying amounts are fine, as long as the text itself is consistent

Journeyman Geek

@marcusdoesstuff hm. It may not be something that works well with a traditional database.

Bob

@JourneymanGeek already went through this above :P

marcusdoesstuff

the titles are ALWAYS exactly the same.

Journeyman Geek

@marcusdoesstuff: except when they arn't

marcusdoesstuff

i've seen the original database... albeit with missing encrypted sections. the company we work with would never open that up for us as we'd not need to continue paying them :(

but these SDS are not made manually.

so I doubt there would be many inconsistencies.

not unless i manually scrape the data hah

Psycogeek

10:19

Well he is trying to prioritize safety? So "vaporises your skin" "High explosive" "causes cancer in rocks" ya figure that is probably the worst of it :-)

marcusdoesstuff

@Psycogeek Yeah, I'm didn't think that there were major problems in the database diagram I put up above. There are some custom phrases that would be used in some parts but I don't think any titles change.

Thank you all for all the help so far, and your patience, by the way!

Picking up the correct vocabulary here has made it easier for me to search around the topic too. :D

Bob

10:35

@marcusdoesstuff See also, for relational DBs: en.wikipedia.org/wiki/Column_%28database%29

(the terminology I use isn't 100% correct with DB theory, but... eh.)

marcusdoesstuff

@Bob Hah well it still seems to work.

@Bob I'd looked up database types before and relational seemed like it might work... It's also fairly simple for me to get my head around. What other database types should I be considering?

Bob

@marcusdoesstuff Well, the basic two types are relational and key-value (flat/indexed).

Relational is what you've seen: a DB field (foreign key) might refer to some ID in another table

Key-value is just a map/dictionary where a specific key is linked to a specific value (or list of values)

The key might be indexed. Or might not.

Oh, there's also hierarchical.

That's something like a filesystem, where you have nested 'directories' and 'files'. Or like in the Windows registry, with 'hives' (top-level), 'keys' (directories) and 'values' (files).

Journeyman Geek

which would work very well here ;p

Bob

For your particular case, the intermediate format of JSON/XML would be hierarchical.

@JourneymanGeek Yep.

That's what I would have recommended.

Key-value is very simple, perhaps too simple.

Hierarchical is fairly simple, and flexible enough to represent all your data in a nice way.

Relational only makes sense if you want to index more fields (combination of hierarchical/key-value could also work) or if you have actual relationships between tables that you might want to express/query at a later time.

marcusdoesstuff

I was planning to use the hazard codes and then have them refer to other tables with the full phrases in each language, for example. half of the time i'll only need the codes anyway. it seemed work when i drew it out. i think i need to read up more on these other types to understand them fully.

Bob

10:43

About a decade ago, relational was all the rage. Everyone used it, for everything.

marcusdoesstuff

yeah i read about that

Bob

It's swung a bit in the other direction now, and key-value is quite popular nowadays.

At the end of the day, any format works - but some are better than others for some tasks.

@marcusdoesstuff Yes, that would work, though it is more complicated.

That's one use of a relational DB.

marcusdoesstuff

flexibility will be useful in future; we're already seeing other potential uses of the data.

Bob

But you could also use a hierarchical DB to represent the actual data and a key-value store to link codes to descriptions.

It's not an all-in-one solution, and probably not as efficient, but it's more lightweight and easier to wrap your head around.

@marcusdoesstuff Relational DBs actually end up not very flexible.

When you first design the DB, you can do a lot with it.

But if you ever want to change/update something, it can be painful.

Sometimes that's a good thing. It forces you to set out what you want, and have well-defined specs for where everything goes and how it all links together.

marcusdoesstuff

that's what i'm worried about :P

Bob

10:46

On the other hand, it makes it harder to develop a quick lightweight system.

Journeyman Geek

Its also fairly efficient if you do it right

Bob

For large systems, well-defined specs are great.

For simple short-term stuff, the startup costs are too high.

marcusdoesstuff

isn't relational quite flexible?

Bob

@marcusdoesstuff At design time.

Modifying an established relational DB's structure is painful, at best.

Journeyman Geek

;p
We do it at work. Its 'fun'

marcusdoesstuff

10:47

ha

Bob

Then you have formats like XML. In their raw form, they're simply hierarchical. Very flexible with no fixed structure.

marcusdoesstuff

as this is SDS stuff... it will need to be updated in areas from time to time... new regulations may change the SDS a bit too :(

Bob

Once you introduce an XSD, you can enforce a rigid structure. But then it also becomes harder to change.

Oh, that's the word I was looking for. Schema.

Most relational DBs don't make it particularly easy to change an existing schema.

@marcusdoesstuff Adding tends to be easy.

Removing tends to be difficult, especially if a primary key, composite key, or foreign key is involved.

Changing... well, normally you wouldn't have to change much. Even if SDS format changes, your data format might not have to.

marcusdoesstuff

i noticed that about taking away...

Bob

Your parsing code might have to change, though.

marcusdoesstuff

10:52

phrases might change, etc... new ones introduced etc... not too much problem.

once these are on the DB, frankly they could be updated manually.

Journeyman Geek

At worst, throw out all your old mistakes and do it again ;p

marcusdoesstuff

if there's a huge change then i might need to re-do it all anyway, yeah.

Bob

@marcusdoesstuff Anyway, if you could provide a full csv, I can try to get it into a hierarchical format (JSON, or maybe XML). Then you can figure out if you still need a relational DB.

Since you only have ~500, you might find it easier to just iterate through all of it.

marcusdoesstuff

the only reluctance i have at the moment about other database types it that i don't fully understand anything other than relational.

Bob

Only problem is someone would have to write code for querying. Would be fairly simple code.

marcusdoesstuff

10:55

been waiting on an email with more and they just arrived.

Bob

@marcusdoesstuff For hierarchical, just think of your filesystem.

marcusdoesstuff

how do i send you csv files easily?

Bob

You have named directories, that contain attributes (description, etc, in XML; JSON has no attributes) and a list of named subobjects. You have data (files).

marcusdoesstuff

haha paperclip mascot to the rescue.

Bob

@marcusdoesstuff Just chuck the text on gist.github.com or similar

Psycogeek

10:56

Deficient question of the week award : superuser.com/questions/951640/…

Bob

Assuming you don't mind this being public.

I don't see anything particularly secret about an SDS, but your company might.

marcusdoesstuff

it's okay as it's all public in pdf format on our website anyway.

give me a couple mins

here is that one from earlier:
https://gist.github.com/marcusdoesstuff/9e3fa31efba1a5cf7194
I don't get while this csv file has multiple columns though (that's not how csv is meant to work is it?)... They don't show here.

gist.github.com/marcusdoesstuff/a107cf301f94e50dd8bf

gist.github.com/marcusdoesstuff/a7e3e555515c1370701c

i've realised that there are a few others that are classed under cosmetic regulations or as medicines. i'm looking at how different they are but if they are, they'd be best to do manually anyway as they are a small minority.

it's all quite regular up until about line 35 when they drift out of being in the same cells.

Nate Kerkhofs

Well, that was a REALLY weird bug

Psycogeek

@Ramhound Because i was asking the guy where he picked it up from superuser.com/questions/951664/… because it is rare for anyone to call it that. plus he has a problem that is showing a number that is different "Faulting application name: SearchUI.exe, version: 10.0.10240.16401" He shows it as version .16384 , And Dont freaking treat me like that, i just asked a question.

Nate Kerkhofs

I was playing a game, and all of a sudden, my keyboard went all messed up, like "my keys move me, but can't type anything" messed up

And my Escape button opened the start menu

is there a function-escape shortcut that opens the windows menu?

ah, now I understand it

somehow, my control key got bugged out

Bob

11:12

@marcusdoesstuff hmm, 404...

Nate Kerkhofs

and stayed active

bertieb

@NateKerkhofs Ctrl+Esc, your con... yeah

HANDS, TYPE FASTER

4

So I was messing about with a very old, not-touched-in-years Firefox profile; and realised I'd read @nhinkle's Win 7 network connectivity determination article on the SU blog (blog.superuser.com/2011/05/16/windows-7-network-awareness).

Did the blog die due to lack of contributors?

Bob

@bertieb More or less, yea.

Super User Blog Editor Room

Where the editors drink coffee and try to get things done!

bertieb

@Bob So it goes :-/ Pity

marcusdoesstuff

"One of our mostly harmless robots seems to think you are not a human."

it's hidden my stuff haha. give me a min.

"We promise we won’t require DNA proof of your humanity."

bertieb

11:19

There's some good stuff on there. Any idea what viewership was like when it was active?

Er, readership

Bob

...dunno

I think some questions linked from the blog did gain more views. Can't remember the numbers.

Dunno if the mods can look that up (@JourneymanGeek?)

Or if you'd have to ask a SE employee.

bertieb

Ah fair enough; it's really just idle curiosity fueling these questions :P

marcusdoesstuff

@Bob going to have to wait for a github support non-robot to get back to me i think. :/

Bob

O_O

marcusdoesstuff

i'm going to see if i can get my web server working again.

bertieb

11:28

Those QOTW posts are still interesting to read

A question answered by John Carmack! (superuser.com/questions/419070/…)

And sort of prompted by him out-of-band, but still

Bob

@bertieb I don't think we've had those for over a year now :P

bertieb

@Bob Yeah, it's a shame; the old ones are still interesting

I guess there aren't interesting enough questions

I mean, how interesting is HAY I CANT UPGRADE TO WIN10 anyway? :P

Bob

@bertieb New OS is time to step away from the main site for a bit :P

bertieb

@Bob True that :P Or go all greasemonkey on Qs with a certain title...

Actually, are there any greasemonkey (etc) scripts for SU I should know about? I searched a while ago but didn't find anything obvious

Bob

@bertieb [stackapps.se]

Link shortcuts broke.

Oh, that's nice.

DragonLord

11:36

0

Q: Should coffee grounds be refrigerated?

A package of store-brand coffee grounds we have specifies that they should be refrigerated or frozen after opening. However, we have never had an issue with not refrigerating coffee grounds—as long as the container is airtight and stored in a cool and dry place, the coffee usually comes up reason...

bertieb

@Bob Oh yes cheers, I think I had [mentally] bookmarked that for perusal when I last asked a similar question

scurries off to search box

Oh yeah, it was chat IRC-interface related

Bob

@marcusdoesstuff Figured anything out? Could always email them. Or post them on pastebin or similar.

DragonLord

Above question is deleted because it is a duplicate.

28

Q: Good ways to store coffee?

What are some key points to store coffee to preserve freshness? Different methods for whole beans vs ground?

storage-method coffee

> Storing coffee in the refrigerator or freezer for daily use can damage the coffee as warm, moist air condenses to the beans whenever the container is opened.

marcusdoesstuff

12:00

@bob could email, yeah. i'm just going to lunch, i'll be back in a bit.

Bob

@marcusdoesstuff lemme know when you're here, I'd rather not leave my email address up for long

might have to wait for tomorrow, though, it's getting late here. we'll see

allquixotic

12:15

hi

@Bob you mean you're actually going to sleep at a reasonable hour?

Bob

@allquixotic maybe :P

Journeyman Geek

12:32

@Bob I really have no idea

And erm. Its pretty hard finding people to write stuff

or things to write about

HackToHell

12:43

uurrggh windows kernel is eating up 200 mb of ram

Bob

@HackToHell ...where do you see that?

Also, chances are it's either (a) a dodgy driver or (b) a handle or GDI leak elsewhere

A leak in the core kernel itself is exceedingly rare.

HackToHell

Probably my wifi driver

Bob

Hm. I wonder if Win10 counts system memory usage differently.

marcusdoesstuff

@bob drive.google.com/open?id=0BwzV6ocSg7t6SXU1VnBKSUJyV1E

drive.google.com/open?id=0BwzV6ocSg7t6NkhrdkpmU1JVZ0E

drive.google.com/open?id=0BwzV6ocSg7t6a01aSS1KLVJfb3M

ugly as hell on google sheets though!

oh actually it's only the preview that looks weird.

If the hazard sections didn't vary in length then i could just extract cell ranges.

Bob

@marcusdoesstuff That makes my eyes bleed.

,,,H314 Causes severe skin burns and eye damage.,,,

^ CSV abuse

marcusdoesstuff

12:54

Perhaps an easy option would be to have a batch process that pushed all the titles down so all the data it always in the same cell ranges.

IKR

Bob

@marcusdoesstuff At this point you might as well do a proper parser. Not that hard.

No value in reading it as CSV, though.

Better off stripping the commas and parsing it as plaintext.

marcusdoesstuff

you should see an extract from a pdf. I have a feeling it is this way because of how it is exported into those pdfs actually.

Bob

Doesn't look like any cases where there's multiple fields on the same row.

marcusdoesstuff

yeah i was discussing that idea with a colleague earlier

Bob

I'll be back in an hour... gotta do something

marcusdoesstuff

12:56

there are some titles sharing rows with data but that's it.

alrighty!

@bob if these are not easy for you to do things with then please don't spend lots of time trying, i know you will have far more important stuff on your plate.

bertieb

@marcusdoesstuff Not to further volunteer him (!), but @Bob seems to have an amazing capacity for helping with very non-trivial problems

As do other folks here in RA of course, but @Bob helped me track down the cause of a rather obscure BSOD

I still need to write up a self-answer Q for that; real life has gotten in the way unfortunately

marcusdoesstuff

13:16

@bob =INDEX('Dairy Hypo SDS in csv'!A:A,MATCH(""&A1&"",'Dairy Hypo SDS in csv'!A:A,0)+1)

this string searches for a text string (that is specified in A1) and returns the text that is contained in the cell below!

@bertieb everyone is certainly very patient.

Psycogeek

"Windows updates installed something called "Diagnostics Tracking Service" in my Windows 8.1 Pro." "THEY CAN INSTALL SPYWARE ANY MOMENT. . ."
reply "Diagnostics Tracking Service was an Optional update from Microsoft,
It did not install unless you selected it since it was optional."
Oh :-)

answers.microsoft.com/en-us/windows/forum/windows8_1-update/… But of course it does not end there, if you want to read the rest.

makes you wonder though, if he knew about the other 5 things that had been shipping out error and diagnostic stuff before pushing the button to get this one :-)

DragonLord

14:15

In Windows 10, even the low-level error messages, such as those in the boot sector, have changed.

> An operating system wasn't found. Try disconnecting any drives that don't contain an operating system.
Press Ctrl+Alt+Del to restart

It used to be:

> BOOTMGR is missing
Press Ctrl+Alt+Del to restart

(Looks like this was introduced in Windows 8, not Windows 10.)

Psycogeek

14:30

superuser.com/questions/951795/windows-10-email-app-error A dupe ?? ok i can see calling it a possibility, but already marking it as one, would only be done as an AUDIT that anyone is paying attention :-)

Really do some people use a script that just wildly picks up on any repeated words , and flags them that a dupe exists .

I guess just like the malware virus thread, we just make a "Something Happened" thread and point the whole universe to it from now till microsft actually puts dialog in them. !8,000 possible problems, 1 solutions.

DragonLord

DavidPostill does far more review work and posts a lot more answers then I do.

Time to go back to reviewing...

Psycogeek

@DragonLord his answers many many are quite good to, yours i assume do not copy paste everything . and that is all i am going to say about it :-)

DragonLord

I'm starting to feel the pressure to stay competitive

bertieb

@DragonLord Do it not for competition, but for the love of reviewing! :P

DragonLord

I've always like to flag bad content, but I've gotten bored about reviewing.

Bob

14:46

@marcusdoesstuff ok, back now, I'll take a look at it (sorry about the wait)

Obvious patterns: SECTION # and #.#.

bertieb

@DragonLord There's plenty of folks reviewing AFAICT. The auto review comment proforma userscript is handy for the First Post (frist psot?) queue :)

DragonLord

@bertieb Have been using that userscript for a while (it's why I installed Greasemonkey in the first place).

> Unfortunately, this doesn't answer the question. We're a Q&A community, not a conventional forum, so answer posts must answer (or attempt to answer) the question—answer posts which fail to do so are subject to deletion.

bertieb

@DragonLord Ah cool :)

@DragonLord Heh

> Hey kid, here's a nickel; go buy yourself some newlines.

DragonLord

DavidPostill is a leading reviewer and has a high flag count.

marcusdoesstuff

@bob drive.google.com/file/d/0BwzV6ocSg7t6ektjU3B4S1puS1U/…

DragonLord

14:54

Time to review again...

marcusdoesstuff

@bob I've been experimenting... This shows what info I need to extract too. On the second page I began pulling stuff out with a function.

Bob

@marcusdoesstuff I'd say it's best not to use Excel for this: unless you want to dive into VBA, there's no easy way to export it into a relational DB AFAIK

marcusdoesstuff

Yeah I saw some VBA stuff on my travels through Google.

I assumed as much too.

Bob

oh dammit

it's not simple csv either

they're using bloody quoted blocks :(

marcusdoesstuff

what i was resorting to here was to find the titles and then drag out other cells in relation to the position of the title.

crude, i'm sure!

Bob

15:00

the use of quoted blocks makes this... annoying

marcusdoesstuff

could they just be erased?

Bob

@marcusdoesstuff take this example:

Supplier,,,"Evans Vanodine International
Brierley Road
Walton Summit
Preston. UK. PR5 8AH
Tel: 01772 322 200
Fax: 01772 626 000
[email protected]",,,

That's one row

marcusdoesstuff

seen my last link?

Bob

But naive parsing would ignore the quoted block and interpret it as 7 rows

marcusdoesstuff

that info is not needed//

Bob

15:01

@marcusdoesstuff I'm dealing with the raw CSV at the moment

@marcusdoesstuff no guarantee that similar quoted blocks won't appear elsewhere

marcusdoesstuff

docs.google.com/spreadsheets/d/…

see the areas that are grey... those are all not that important.

and then on the second sheet there is a kind of table with all the important bits only.

DragonLord

@bertieb This is my current list of comments, having grown extensively over time as my needs have evolved:

> ### Q: Off-topic (be sure to fill in information)
Voting to close as *off-topic*, because questions about [topic] do not belong on $SITENAME$. See the [help center](http://$SITEURL$/help/on-topic) for details. This question should be migrated to [site] soon; please do not cross-post, as this will happen automatically once enough close votes have been cast on this question. To ensure you retain control of the question, please create an account at [site] and associate it with your $SITENAME$ account.

marcusdoesstuff

that or they are titles so important, but table titles rather than data.

Journeyman Geek

My comments are fresh, not canned ;p

marcusdoesstuff

@JourneymanGeek Is this the start of a rap?

Journeyman Geek

15:08

naw ;p

marcusdoesstuff

aww

DragonLord

@JourneymanGeek Pro-forma comments a bad idea?

Journeyman Geek

I'm personally not fond of them

I tend to prefer to consider before I put down a comment and writing them out helps with that

marcusdoesstuff

@bob i keep bouncing between different colleagues and learning more...

@bob seems like sections 4 and onwards are all DEFINED by the hazards so i don't really need to extract them.

@bob i edited that sheet and highlighted the info i need to extract now... substantially less.

allquixotic

haha, I can get a GTX 980 for $349 at the employee pricing portal for my company...

Bob

15:22

@marcusdoesstuff Hm. I was thinking about something general, but this actually simplifies matters.

bertieb

@DragonLord Very handy, thanks!

Bob

@allquixotic You going for it?

marcusdoesstuff

@bob it certainly does seem to.

Bob

@marcusdoesstuff What does the 'Not Classified' look like in other cases?

@marcusdoesstuff And do you need anything from section 3?

marcusdoesstuff

"Flam. Liq. 2 - H225"

for example for a hand sanitiser alcohol gel

for our risk scores we don't need section 3, it would seem.

Bob

15:34

Wait, did you want all the physical, environmental, health hazards?

shrug

I'll get something up for section 3

allquixotic

@Bob I have a "credit" in "points" to the tune of $178 from the car I purchased through the same website in March

that, and buying a video card will net me about $50 more in points to spend on something else

Bob

o.O

allquixotic

it's definitely way cheaper than buying it on Amazon

they don't have the 980 Ti in stock for some odd reason, but I can buy the 980, the TITAN X, or the Fury X

the Fury X just seems overpriced for what it delivers; I really can't justify buying it, as much as I'd like to

if it were $100 less, I'd buy it

Bob

@allquixotic For the price of the 980, 2x980 would be cheaper than 980Ti, no?

allquixotic

@Bob well, I'd be paying $178 more for the second 980 :P so no

Bob

15:36

Oh, ok :P

@allquixotic $128 more*

heh

allquixotic

true, because of the extra points :P

the 980 Ti is what I REALLY REALLY want, since it is within my budget and meets the best price/performance and is only trivially slower than the much more expensive TITAN X

marcusdoesstuff

shiny talk

i settled for a 970, heh

allquixotic

it's almost stupid how little the difference is between the titan x and the 980 ti

like Nvidia almost doesn't want people to buy the TITAN X

who would? people with more money than sense?

Bob

<== still using a 560, still don't really feel like upgrading

probably been years since I've played a game that really pushed the GPU hard

allquixotic

that'd be like Intel putting out a 4769K that's got 99% the performance of the 4770K but at half the price

it's just silly

the only thing I can think of the TITAN X being useful for, is that 12 GB of GDDR5 will help for crazy people using dual 4K displays

but two 980 Tis is only slightly more expensive than one TITAN X and delivers way more GPU horsepower and the same amount of GDDR5

marcusdoesstuff

15:42

money is no object to a ton of people

allquixotic

except that many people with a ton of money, naturally care a lot about keeping their money and don't want to lose it, because they have a lot of money because they care about money - so they're not going to throw away money needlessly

marcusdoesstuff

i think a 980 is too much as it is! :P i'd rather get two 970s (other than perhaps the 3.5+5GB vram thing).

allquixotic

if they get a sense that the TITAN X is not meaningfully better than a 980 Ti, they won't just waste $500

marcusdoesstuff

get rich by not spending it all, yes

but some people are not like that heh

it does, indeed, defy logic.

@bob i need to go home and i'm not going to be back on for a couple hours. i'll be here tomorrow though.

@bob you've been unbelievably helpful.

Bob

@marcusdoesstuff just figuring out the basic scraper... probably going to be rather specialised to just the data you highlighted

the hazards lists are a pain

nothing really distinguishing the list from the next 'heading' :\

bertieb

15:51

@marcusdoesstuff 2 970s here, it's a good setup (except when something works slightly wrong due to SLI)

allquixotic

I'm really against SLI/CrossFireX after being burned by so many damn games that just can't do it right

little glitches here and there that are just too jarring to play

I need a single card that is powerful enough to last me the next 3 years

I used to have money to upgrade every 2 years but now I'm targeting a 3 year GPU upgrade cycle

I ordered my first HD7970 in mid-2012, so that puts me right on schedule for an upgrade now

Bob

@marcusdoesstuff Here's an example script:

import re
import json

singleLinePatterns = [
    r'(Product name)[,]+(.*?),',
    r'(Product number)[,]+(.*?),',
    r'(Internal identification)[,]+(.*?),',
    r'(Identified uses)[,]+(.*?),',
    r'(Contains)[,]+(.*?)$'
]

singleLineMatchers = [re.compile(p) for p in singleLinePatterns]

multiLinePatterns = [
    r'Physical hazards',
    r'Health hazards',
    r'Environmental hazards',
    r'Hazard statements',
    r'Precautionary statements',
    r'Supplemental label information'
]

multiLineMatchers = [re.compile(p) for p in multiLinePatterns]

Stick the path to your file in the open() statement.

Example output:

{
 "Internal identification": [
  "P.V6"
 ],
 "Contains": [
  "\"SODIUM HYPOCHLORITE SOLUTION, ... % Cl ACTIVE\",,,"
 ],
 "Product number": [
  "R064 EV"
 ],
 "Environmental hazards": [
  "Environmental hazards",
  "Aquatic Acute 1 - H400",
  " "
 ],
 "Hazard statements": [
  "Hazard statements",
  "H314 Causes severe skin burns and eye damage.",
  "H400 Very toxic to aquatic life."
 ],
 "Physical hazards": [
  "Physical hazards",
  "Not Classified"
 ],
 "Product name": [
  "DAIRY HYPOCHLORITE"

Should be relatively easy to dump it into a DB from there.

Ok, I've finished interrupting chat with massive pastes now :P

@allquixotic Speaking of which, any word on Star Citizen?

allquixotic

@Bob apparently, it can run at 1080p60 smoothly on the GTX 980. The Ti can run it at 1440p60 smoothly with very few frame drops. Both of these are on Very High / Ultra settings for most things (probably with at least something cut back; there's always That One Setting that requires 4xSLI TITAN X to stop frame drops)

4K basically requires SLI/CFX with any currently released graphics card

you can probably get decent FPS with the TITAN X at 4K, but not solid 60fps

basically, the Maxwell GTX cards are the "target" hardware for Star Citizen. To a lesser extent, GCN 1.2, but since GCN 1.2 is such a colossal disappointment, you pretty much will need the Fury X if you're on the AMD side to get acceptable performance. And that's for, at most, 1440p. Don't even try 4K on AMD.

On the AMD side, the "ideal" hardware to run Star Citizen on will be whatever comes up after GCN 1.2.

Whatever it is, it's going to be faster than GCN 1.2 by a significant margin (we hope), which should be good enough to run it perfectly at 1080p, extremely well at 1440p, and good (but not perfect) at 4K.

Bob

16:26

@allquixotic I wonder how they're doing with 14nm.

Intel had some trouble with it.

allquixotic

16:38

@Bob the next generation of TSMC process will be 20nm, I think; but I also heard rumors swirling that AMD's foundry spin-off (GlobalFoundries?) will be putting out 14nm or 16nm or something for the next AMD GPU generation

AMD was pissed that TSMC dragged their feet so long with 28nm

the 28nm node itself was late, and slow to scale up to the desired production levels, and had yield problems

and the node after 28nm is even later, and has even worse production problems

in short, TSMC sucks ass and AMD is sick of it, and they have an alternative that should be better

Bob

@allquixotic Rumors of Nvidia doing something on 14nm with TSMC or Samsung. Of course, just rumors.

marcusdoesstuff

20:23

@Bob thank you very much. i'll give it a go as soon as i'm in the office in the morning as i don't have office on my home desktop yet. i thought those hazards might be a pain... One pattern with them, however is that they are always either "H" or "P" with three numbers following (or sometimes several strung together with more of the same). Perhaps a GREP like function could pull them out that way and then it wouldn't matter how many cells there are.

@Bob There are some at the bottom that repeat though so it'd need to be limited to the first 100 rows only or something.

@Bob looking forward to trying it out :)

Actually... I'm not entirely sure on the best structure, within a database, for hazard codes...

Example:
product A has hazard H101, H207 & H209
product B has hazard H101 & H209
product C has hazard H207, H211 & H210

Is it better to have a column for every possible hazard and have each cell as 'true'/'false'? Or to just have columns saying "hazard 1", "hazard 2", "hazard 3" and just have the hazard phrase underneath (so product C has H207 under "hazard 1", H211 under "hazard 2", etc)? Or some other method?

At first I thought I would only need the hazard code itself but apparently there are several different phrases under some of the codes. This new GHS regulation is not amazingly well thought out in too many areas actually.

marcusdoesstuff

20:39

@allquixotic @bob Hmm, I really like the look of SC but it's too early for me to bother yet. I did enjoy playing a bit of Elite though. I even made my own headset attachment with IR LEDs that gets picked up by a webcam that I modded to remove the IR filter and replaced with a visible light filter. It tracks my head movement really well and was far more fun to make than buying one. :P

@allquixotic @bob @bertieb rather than go for anything like 980s, I'm considering a g-sync monitor, heh. waiting for an ips rog swift, heh. won't worry about frame drops that much then anyway. :D

@allquixotic can't you just upgrade to the 2nd/3rd best every 2 years rather than the best every 3? :P well maybe next time SLI will be naied down better.

DragonLord

20:57

@marcusdoesstuff To get this right in a relational database, you'd need to understand fourth normal form. The situation you describe would produce a multivalued dependency in the database schema.

(I know this because I've taken several database courses in college)

The multivalued dependency here is that each product may be associated with none, one, or multiple hazard codes.

Super User Blog Editor Room

Transcript for