@ypercubeᵀᴹ - I worked on my own fiddle as promised. I was thinking this'll take me 30 mins, max - ha, ha... A project manager friend of mine told me once that he took his programmers' estimates and multiplied them by π and that was normally close to the mark - looks like I'm in the (2*π) category! :-) I did flute about a lot and did a lot of testing (some of which I've left in...). I would be grateful for any comments (and maybe a little help with the regex (see text at beginning of fiddle).
I picked a sample financial instrument trading scenario - but my real question is when (and if) triggers would be better for this and/or what are the relative pros & cons of this type of solution vs. TRIGGERS (again, see text). The fiddle itself is here. Thanks for any input! p.s. really liked your approach - learned a lot!
@ypercubeᵀᴹ - no: I don't have a previous_change_date - don't see the need for it really - do you think it's essential? I do have this: CONSTRAINT m_id_d_id_m_type_uq UNIQUE (man_id, d_id, m_dt), -- no txns within same micro-second on the same deal... I've tried my hardest not to have any NULLable fields - I really don't like them (but can deal with them), but excluding them simplfies some of the logic - I don't require: CHECK ((previous_change_date IS NULL AND previous_total_amount IS NULL)
@Vérace eg: dbfiddle.uk/… (where the last 3 rows added have all kind of nonsense)
You could of course enforce consistency through stored procedures (and restricting access only though them) as it was suggested yesterday. But in that case, you don't need the "previous" values stored.
@ypercubeᵀᴹ - yes - that's to be expected. I "imagined" there would be a dealer UI where the dealer (and/or the client) would be prompted for the (man_id, d_id, m_type, m_dt, m_amount) - (m_dt is NOW()) - as you can see from my INSERT (with CTE) SQL, there is a select done on the previous record and calculations to boot - of course you can enter any old rubbish if you do it directly - how many people (apart from DBAs) actually get to do that?
@bbaird It's an imaginary scenario - this started when @ypercubeᵀᴹ took exception to this:
CHECK ( NOT EXISTS ( SELECT SUM(loan_amount) AS s FROM portfolio GROUP BY manager_name HAVING SUM(loan_amount) > 100 ) )
I wondered how you'd enforce constraints differently...
@ypercubeᵀᴹ As for the "wrong values in d_tot and d_tot_prev" - there'll be no errors if you update them correctly through the interface. I do see the point though - the data should be protected purely by "the DDL, the whole DDL and nothing but the DDL" (motto for a DBA to fight and die for! :-) ).
Yes, I can sort of see now - if I allow NULLs and do some sort of COALESCE, then no matter what the INSERT method (via interface OR via CLI), then nobody can mess up - not even root/sysadmin... ah well, back to the drawing board! I'll look at it this afternoon...
CHECK ((previous_change_date IS NULL AND previous_total_amount IS NULL) OR (previous_change_date IS NOT NULL AND previous_total_amount IS NOT NULL)), - just add an AND something = 1?
Could use MATCH PARTIAL - not sure if I've totally understood the whole MATCH FULL, PARTIAL, SIMPLE thing...
Yes - that might be good. As I've said - I'm big into using DDL as a DSL... :-) And expanding its capabilities - can define previous as to the portfolio_id - with microsecond inserts it's a good candidate.
I could use a "magic date" - 1970-01-01 00:00:00 - but that's an ugly hack!
@ypercubeᵀᴹ - I've a feeling this is developing into an O(𝛑^2) scenario and not O(𝛑*2)! :-)
Well, I mentioned earlier on about my PM friend who multiplied his programmers' estimates by 𝛑 and found that was a good metric - as I said, I've already used 2*𝛑 my original ~ 30 min. estimate - I'm now moving into 𝛑*𝛑 territory!
@billinkc Are you seeing use cases for file processing or are people literally trying to replace a database with s3 storage and distributed processing? I can't imagine the economy (pricing or otherwise) actually working out in the end, although I guess if you can risk things being wrong and don't mind waiting a bit for things to get loaded into memory so you can process things somewhat quickly without a good key structure.
My one use case for Databricks right now is parsing a 20GB JSON file and staging it in our SQL Server
@bbaird I'm subbing on a databricks implementation right now. They're just using databricks as a very expensive VM from what I can see. Original sub was an ML person but this is just ETL work that they didn't have the chops for
It's an analytics project for marketing and it's small data but been interesting to do something "new." FB, GoogleAnalytics and Twitter ingestion are similar yet very different mechanisms
Reporting layer is going to use a tool called Dremio and it runs off parquet files. We also slice the data out into CSVs for the sciencey people to play with as well as storing the source data into json.
@billinkc They're (IT + consultants) are trying to replace our entire data infrastructure with Databricks and DeltaLake and I'm just sitting here in awe if they think they can really pull that off. Who's going to build this magical lake, ensure the data isn't riddled with duplicates, etc?
Similar story to a different DBrix project I worked on, it was just overly complicated ETL that needed to move data from different databases to other databases
Granted, it's probably no worse than our current method of just having a lot of SAS code do the same thing over but it's not substitute for... I dunno, fixing the data once and placing it in a well designed database
Right tool for the right job. While I'm thinking about changing my focus from whatever I do now to databricks consultant because the money I think will be better, based on a sample size of 2, I don't know that there is any advantage to this over traditional db & etl
Pity, I was enjoying the fin/insuretech work but I've also become a big fan of not leaving my house
@bbaird - I did what you suggested re. the \- and that seems to have worked - of course hyphen is a metacharacter within square brackets... Found a beauty here - the only problem is that Irish names can be like this: Micheál Ó hUiggin - not to mention our own @Colin'tHart's name which has a space followed by an apostrophe - I'll study the regex when I get a chance - it's a really nice one to study.
@ypercube - financée is grumbling about how I spend all my time in front of the computer... have to go walking around the grounds of some neo-classical mansion built by our oppressors... I'll take a look at the whole CONSTRAINT imbroglio tomorrow...
Micheál Ó hUiggin contains a space followed by a lower case letter internally!
@PaulWhite Is it though? Lock down all write/update permissions to stored procedures, only grant read access and execute on stored procedures based on role. I'd say the biggest challenge to getting things right is Postgres' inability to maintain rows in a specified order.
And getting the accents in. It enrages me when I'm told that a computer system can't cope with my name here at home - for the French and the Germans for example, an accent is an integral part of the name. I've tried [[:upper:]] but it doesn't appear to accept accents...
I've something pretty horrible here - '^[A-ZÀ-ÝŐ][a-zA-Zà-öø-ÿő \- \'']+[a-zà-öø-ÿő]$' - I even have an o double acute separate at the end (went out with a Hungarian girl who had it in her name...) but it's not contiguous with the other [[:alpha:]]'s... see here.
(Australia, New Zealand, Britain, informal) Of poor or dubious quality, shoddy, unreliable; deviously dishonest, fraudulent.
1986, Sociological Association of Australia and New Zealand, ANZJS: The Australian & New Zealand Journal of Sociology, Volume 22, page 475,
The argument gets even shonkier when they claim that the boundary between clerical and administrative work is difficult to define anyway, and that routinisation and fragmentation have affected lower and middle management too (pp. 95–6).
2000, Australian Senate, Parliamentary Debates (Hansard), page 13548,
The committee heard much evidence that this is a farce, yet this government have continued to try to pretend that their forecast, based on the shonkiest of assumptions—their model, with the shonkiest of assumptions—is really what is going to happen.
Even if what you implied was true - that Postgres heaps perform seriously worse than SQL Server's CI tables - what does this have to do with the DDL vs SPs discussion earlier?
Even assuming there is a infinitesimal chance for race conditions, making the duration of the transaction as short as possible reduces the chances substantially.
@bbaird That is true, and in some cases that may be a risk one is willing to take. On the other hand, anything that can happen is bound to happen sooner or later. Sooner if it is inconvenient, in my experience. Constraints offer the nice guarantee of being enforced by the engine (not user code!) at all times.
Maybe there is a reason banks aren't very good at preventing people going over credit limits etc.
They're not terrible at imposing fees in such situations, it must be said.
In SQL Server or Sybase you could certainly create a CHECK constraint using a scalar function to check the balance, which would be evaluated after the insert is attempted but before commit to ensure, based on the timestamp of the transaction it was not overdrawn. Not sure if that's too different than having a procedure that locks the appropriate rows until commit.
@bbaird Well they're slow, prevent parallelism for the whole query, and might not even be checked in some situations. So, no, we don't use functions in constraints.
Triggers add no benefit either, in terms of concurrency anyway.
Well well, I just learned something.
So it turns out that with CHECK CONSTRAINTS and UPDATES, the CONSTRAINT only gets checked if one of the columns referenced in the CONSTRAINT changed.
In your case, your CONSTRAINT is checking a UDF that you pass ItemID to.
In your UPDATE, presumably you...
I don't know if that is a great example, but it's repeated all over the internet.
@bbaird They do. They also sometimes put too much faith into locking hints, make unsafe inferences about isolation levels, and think transactions themselves guarantee more than just success or failure of the whole unit of work.
@bbaird Queries in constraints would be very nice. Unfortunately, it's extremely hard to do in practice.
Firebird seem to have had a crack at it, but with limited success.
I'm exploring possibilities - and I did ask about pros and cons of using triggers as opposed to using CHECK CONSTRAINTs. Yes, I did look at it in detail and modelled my fiddle on his - I've always been banging on about how DDL should be much more powerful - and the 1992 standard has stuff about asserts and database level constraints - they should be implemented in the engine so as not to oblige programmers to write (often flawed) trigger logic!
That's why you have NOT NULL - you can choose - personally, I like the idea (consistent) that no NULL is either equal to nor not equal to any other NULL !
Talking of weird semantics, I was surprised again by that question looking to double rows in a table. The answer using generate_series is an abomination.
If you're using PostgreSQL, put a SET RETURN FUNCTION in the ORDER BY clause.
CREATE TABLE foo AS
SELECT *
FROM ( VALUES (1,'A'),(2,'B'),(3,'C') ) AS x(id,name);
And, then
SELECT id,name
FROM foo
ORDER BY 1, generate_series(1,2);
id | name
----+------
1 | A
1 | A
2 | B
2 | B
3 |...
We should all seek alternate employment immediately.
@PaulWhite That's what COALESCE is for - or not! This is a bit like the set paradox... is a set a member of itself then it can't be in the set of... &c...
In mathematical logic, Russell's paradox (also known as Russell's antinomy), is a set-theoretic paradox discovered by the British philosopher and mathematician Bertrand Russell in 1901. Russell's paradox shows that every set theory that contains an unrestricted comprehension principle leads to contradictions. The paradox had already been discovered independently in 1899 by the German mathematician Ernst Zermelo. However, Zermelo did not publish the idea, which remained known only to David Hilbert, Edmund Husserl, and other academics at the University of Göttingen. At the end of the 1890s, Cantor...
My first thought would be to use the INFORMATION_SCHEMA first, so you get to know (in one query for all tables in the MySQL instance) which tables have an active column and then use that info to construct your queries. And this is probably the most sane approach.
There is one other, tricky way t...
The following query uses the same idea as in this amazing answer by ypercube:
SELECT x.*
FROM (SELECT NULL AS SomeCol) AS dummy
CROSS APPLY
(
SELECT
ID,
SomeCol AS MyTest
FROM dbo.Customers
) AS x;
It works like this:
if dbo.Customers has a column named SomeCol, then SomeCol in S...
ARJ (Archived by Robert Jung) is a software tool designed by Robert K. Jung for creating high-efficiency compressed file archives. ARJ is currently on version 2.86 for DOS and 3.20 for Microsoft Windows and supports 16-bit, 32-bit and 64-bit Intel architectures.ARJ was one of many file compression utilities for DOS and Windows during the early and mid-1990s. Parts of ARJ were covered by U.S. Patent 5,140,321 (expired). ARJ was well documented, and was the most feature rich compression utility at the time, with over 150 documented command line switches, which contributed to its wide-spread adoption...
@Vérace right. That is also what the standard says about array comparisons.
Lists: pgsql-bugs
Manvendra <manvendra2525(at)gmail(dot)com> writes:
> Comparison should return UNKNOWN but it's returning a true.
> postgres=# select array[null]=array[null];
Yeah, that's intentional, because we have to be able to sort arrays.
Comparison of composites behaves similarly, btw.
> SQL-99 standard use to say it should come out as UNKNOWN
We're going to politely ignore the spec on this.
regards, tom lane
The <s>Evil Empire</s> err... MS never did that! :-) Nor did the other <s>Evil Empire</s> err... Amazon didn't, nor did the different <s>Evil Empire</s>... oops Facebook... oh, no, wait <s>Evil Empire</s>... Google... ahh... that's the real <s>Evil Empire</s> - aha Netflix... (insert successful company of whose founders you're insanely jealous here)...
@Colin'tHart - my understanding of Dutch names is that if the surname is given separately, it starts with a capital? Anyway, I may just store it any old way and have a search_name UPPER(TRIM... &c. - remove all accents, spaces, hyphens, apostrophes and capitalise... It's the Irish names that are particularly difficult Ó hUiginn for example... Or Ua hUiginn...
Basically, two vowels together don't work in Irish... so I could write a regex for that... still, no matter what you do, some rubbish will be let through no matter what your regex...
The problem is how to have "proper" names in my db and not freeform rubbish - of which I"ve seen far too much!
Oh, yes! The phonebook, official forms and most Dutch websites (eg ecommerce sites) store the "voorvoegsels" (prefixes) separately, with the phonebook listing surnames as Hart, 't Poel, van der and Beijeren Bergen en Henegouwen, van
> Technical Details > > The regular expression was: ^[\s\u200c]+|[\s\u200c]+$ Which is intended to trim unicode space from start and end of a line. A simplified version of the Regex that exposes the same issue would be \s+$ which to a human looks easy (“all the spaces at the end of the string”), but which means quite some work for a simple backtracking Regex engine. The malformed post contained roughly 20,000 consecutive characters of whitespace on a comment line that started with -- play happy sound for player to enjoy. For us, the sound was not happy.
@Vérace the regex in combination with a malformed post resulted in a 34 minute outage of SO