« first day (3344 days earlier)      last day (1538 days later) » 

12:38 PM
Hey all. I haven't been very active on this site but I'm fairly active at Code Review SE. I've encountered a database design issue. I wanted to go with DynamoDB as that's essentially free at AWS, but it's tricky to design a table there for my use-case. I have an online turn-based game and want to store all the games in a database, everything went well until I wanted to have some statistics, like my player profile should list which opponents I have played against and how count wins and losses.
Everything was fine until I thought of this use-case. I couldn't come up with a good solution as some players would have 60000+ games distributed over 5000+ opponents. On average about 100 games per day are played (I have a lot of existing historical data, ~1.4 million games) but I would like to be prepared for bigger more traffic.
An idea I have had is to have DynamoDB for active games, and then store a summary of the games in a traditional RDBMS solution like PostgreSQL (players playing, scores, wins and losses). Any spontaneous thoughts or ideas?
(The question feels a bit opinion-based which is why I'm asking in chat and not on the main site)
 
My first thought is, why not use the same database (at least why not the same database product) for everything. But I admit I've got no experience using AWS (or cloud DB solutions in general for that matter). And I agree that this question wouldn't be easy to phrase in such a way as to make it acceptable on the main.
 
@SimonForsberg define the problem you have with using DynamoDB for this.
 
 
4 hours later…
4:49 PM
@MaxVernon I would like to have a "Player profile page" where anyone can see some information for that player. Games they have played and players they have played against. And the possibility to click on one of their opponents and get a list of all the game PlayerA has played against PlayerB. The biggest problem here for me is to do be able to query the games that PlayerA has played and group by opponent and also count the wins and losses per opponent
(I have also considered being able to filter to a specific time-period here but feel that I could possibly live without that feature)
 
5:15 PM
@SimonForsberg have you considered this approach?
 
5:42 PM
@MaxVernon Query/Scan all data and aggregate in code I have considered and feel like it would probably be suboptimal. Have not heard about Hive before. Read a little bit about it, not completely sure how it works. I'm wondering if it might be a bit like using a nuclear bomb to crack an egg
 
@SimonForsberg I have zero experience with DynamoDB. I would probably use a well-designed RDBMS like PostgreSQL or MSSQL.
Something I understand, at least :)
 
@MaxVernon Ah okay. I have some experience with DynamoDB by now, I saw that it does not seem to be a frequent tag on DBA.SE though. I'm also considering PostgreSQL for this, but I really like that DynamoDB is basically free (cheap is good), and in the cloud (I don't have to think about upgrading version or scaling or backups).
I currently have MySQL at my web-hotel, but that service is unreliable - causing ~10 minutes of query time a few times a week, leading to a horrible user experience. I want to start moving away from that web-hotel and go more towards AWS.
I'm considering hosting PostgreSQL on my own server, in my apartment. It's an acceptable but not ideal solution.
 
 
1 hour later…
7:03 PM
@SimonForsberg 10 min queries should not happen. Get the query, explain plan and table structure for those queries and I'm sure somebody will be able to help
2
 
@TomV-trytopanswers.xyz The thing is that 99.99% of the queries go just fine and return a result within 1 second. So I think it might be a connection issue between my server and the database, or stability issue with the database. It's a very cheap web-hotel (~25 USD per year) so I don't have high expectations.
 
even 1 sec is too high for most queries. My guess is your tables are not well indexed or the queries are written sub-optimally. The db is small so you can get away, although it means that a (should-run-in) 10ms query runs in a 1 sec.
Occasionally as the db slowly grows, you are running (or you will) into queries that start to take more and more time.
What you observe - once in a day, all run very slowly - could be due to many reasons. Yeah, cheap hardware with low memory and disks can be the cause or a daily automated backup of the server that takes 5 min and during that everything else slo
At least that what I've seen in many cases of low-budget websites / databases that started as a hobby and didn't expect to grow too much.
Feel free to shame me with your incapable design and extensive indexing ;)
 
7:25 PM
@ypercubeᵀᴹ Looking at the past 24 hours: Minimum 728ms, Average 1402.51ms, Maximum (slowest) 3222ms. Past 7 days is: 728ms/2397.35ms/603425ms.
The query is SELECT player_id, display_name, user_level, rating FROM players WHERE auth_system = :authSystem AND platform_id = :platformId
^^ And there's the indexes.
I should add that the database is hosted in U.S. while my server interacting with it is in Sweden
 
@SimonForsberg Nice, you have a 2-column index, fits the query.
 
@ypercubeᵀᴹ It's not that "once in a day, all run very slowly", it's that "once or twice per week, one query runs very slowly"
 
@SimonForsberg How are the stats collected? Do you log all queries or above a threshold?
 
@ypercubeᵀᴹ Yeah, it feels like I have done everything "right", so I'm pretty sure the database is to blame
@ypercubeᵀᴹ I log all queries, and the statistics I shared here is from this Splunk search: index=mfe "SQL-Select" "SQL cleanup" | rex "Run time (?<runtime>\d+)ms, total time (?<totaltime>\d+)ms: (?<task>.*)" | eval queuetime = totaltime - runtime | table _time task runtime queuetime totaltime | stats min(runtime) avg(runtime) max(runtime)
 
I'd like to see the SHOW CREATE TABLE players ; output
 
7:30 PM
Your SQL query has been executed successfully.

SHOW CREATE TABLE players



players	CREATE TABLE `players` (
  `player_id` int(11) NOT...
¤"()="¤/%#"/)(=¤"==)"/¤(@£@$£$
Ah, found the button for "full text"
CREATE TABLE `players` (
 `player_id` int(11) NOT NULL AUTO_INCREMENT,
 `display_name` varchar(32) NOT NULL,
 `auth_system` varchar(6) NOT NULL,
 `platform_id` varchar(32) NOT NULL,
 `user_level` int(11) NOT NULL,
 `rating` double NOT NULL,
 PRIMARY KEY (`player_id`),
 KEY `platform_auth` (`auth_system`(1),`platform_id`) USING BTREE,
 KEY `rating` (`rating`)
) ENGINE=InnoDB AUTO_INCREMENT=7132 DEFAULT CHARSET=latin1
 
and this:
SELECT cnt, COUNT(*) AS groups
FROM
( SELECT COUNT(*) AS cnt
  FROM players
  GROUP BY auth_system, platform_id
) AS x
GROUP BY cnt
ORDER BY cnt DESC ;
oh
Why, why is that (1) in that index?
(`auth_system`(1),`platform_id`)
 
@ypercubeᵀᴹ cnt 2 = 14 and cnt 1 = 7103
@ypercubeᵀᴹ Umm... good question? Not sure? Something to do with a compound index thingy?
 
So, the combination of auth_system and platform_id is almost unique. good.
Correct the index or add another one.
 
If I would re-make this table, I would get rid of the auth_system column and just make the platform_id a bit longer, and use the auth_system as the prefix for that field.
 
CREATE INDEX auth_system_platform_id__ix
ON players (auth_system, platform_id) ;
Or reverse order, doesn't matter.
 
7:39 PM
@ypercubeᵀᴹ Done
 
Now check the same query in the next min or hours
 
There's an explanation for the (1) thingy btw ^^
 
what is the explanation?
 
Although I think I should just delete that index now
@ypercubeᵀᴹ No clue.
 
perhaps a misclick when the index was created
 
7:42 PM
Perhaps
I noticed that the cardinality for auth_system is now 10 instead of 6.
Doesn't seem to have helped though:
SQL Cleanup. Run time 736ms, total time 736ms: AuthenticationLookup{authSystem='guest', providerId='guest_v12684'}
SQL Cleanup. Run time 2201ms, total time 2201ms: AuthenticationLookup{authSystem='github', providerId='Zomis'}
Both of these was me.
I'm pretty sure the database connection is just slow and the database instance lousy.
 
is the above the only query run by this authentication lookup?
 
Yes.
As said, my server is in Sweden and database is in U.S.
And web hotel is cheap and probably haven't given the MySQL instance much love.
 
that adds certain latency, yeah
Can you not connect to the db server and run queries though cli?
 
It should still not give 10 minute latency but...
 
for testing I mean, to see how long queries run, without the latency
 
7:46 PM
@ypercubeᵀᴹ Don't have CLI access but I do have phpMyAdmin access
@ypercubeᵀᴹ Showing rows 0 - 0 (1 total, Query took 0.0002 seconds.)
So yeah, indeed quite some latency here...
Might be the C3P0 connection pool that's slowing things down.
Or my bad code
 
are those parameters (:authSystem, :platformId) the same type as the columns,
strings in this case?
 
@ypercubeᵀᴹ Yes.
 
I'm out of ideas then.
Except maybe the connection pool you mention.
If your app is creating a new connection for each, query, not good.
 
@ypercubeᵀᴹ Hmm... Now that I think about it, the connection pool is also testing the instances against a test database before it uses them. Simply just selecting the only record that is in a test-table.
I don't think it creates a new connection for each query
 
8:03 PM
Another thing: is the logging you mentioned in your app or in the database?
 
In my app
Done after it has measured the time the query took
 
so it includes the time to and from the db.
 
You can try profiling queries from phpmyadmin, with something like:
SET profiling = 1;
SELECT ... -- your query ;
SHOW profiles;
 
Would profiling help though, as it's not the queries themselves that are the bottleneck?
 
8:07 PM
pick some parameters that will return data and see what it shows
@SimonForsberg it will show you if the problem is the db or the distance + app
If the query runs in 10 ms in the db, then the problem is either distance or your app or both
If the query runs in 200 ms, yeah, the db has lousy characteristics
 
21 mins ago, by Simon Forsberg
@ypercubeᵀᴹ Showing rows 0 - 0 (1 total, Query took 0.0002 seconds.)
I thought we already settled that?
 
You can also try from your app a silly query, eg SELECT 1 ; and time it. Does it still need 700ms?
@SimonForsberg yeah but I'd prefer a query that returns a row or some. Or a few different queries. Not just one and 1 try.
 
Profiling results ^^
 
Doesn't your query show up?
The one that selects from players?
 
@ypercubeᵀᴹ I did ran that as well, but it's very unclear how this works... I blame phpMyAdmin...
 
8:19 PM
yeah and I can't help much on that. It's years since I last touched phpmyadmin or a mysql database
Some good advice here: code.tutsplus.com/tutorials/…
Not all of it will apply to you as you don't have access to change any settings in the db I suppose
 
Sometimes with free things you get what you pay for.
4
 
@PaulWhite9 While not completely free, I totally agree.
I'm starting to look into Amazon Aurora now.
> For both Single-AZ and Multi-AZ deployments, pricing is per DB instance-hour consumed, from the time a DB instance is launched until it is stopped or deleted. Partial DB instance-hours are billed in one-second increments with a 10 minute minimum charge following a billable status change such as creating, starting, or modifying the DB instance class.
Since I expect this database to be used quite rarely, I think this can be a good (and cheap) solution
 
How do you mean rarely? Isn't the app and db always up?
 
@ypercubeᵀᴹ The application itself is always up, yes. But people are not always connected to it. And even when people are connected, it's not necessarily anyone checking the dedicated statistics page - maybe they're just playing. So my thought is that the app is hosted on my server in Sweden, it connects to DynamoDB for active games.
And then for finished games it can put those into the Amazon Aurora instance, but this can be done on an on-demand basis: When someone goes to the statistics page, or automatically once per day, or when there's 10 new games to add.
 
8:34 PM
Why don't you have the database in the same server, in Sweden
dynamo, mysql, postgres, whatever
 
@ypercubeᵀᴹ Partially because I want to learn more about AWS (or Serverless in general), and move more towards that. Also because I would have to setup backups manually, and do maintenance.
 
 
3 hours later…
11:51 PM
This doesn't work inside an ITVF in SQL Server? OBJECT_SCHEMA_NAME(@@PROCID) + '.' + OBJECT_NAME(@@PROCID)
@@PROCID returns a number, but one of the others is returning NULL.
Not a big deal, all these are wrapped in a build template, so I have a template equivalent of the schema and object name that I can use instead, but just kind of weird.
 

« first day (3344 days earlier)      last day (1538 days later) »