Data centric (SQL)? Domain centric (JPA)? Or both? Let's Discuss!

Copyright (c) 2018 by Data Geekery GmbH and Hypersistence SRL. Slides licensed under CC BY SA 3.0
NoSQL? No, SQL!
Data centric (SQL)?
Domain centric (JPA)?
Or both? Let's Discuss!

Me – @lukaseder
SQL is a device whose
mystery is only exceeded by
its power!
- Founder and CEO at Data Geekery
- Oracle Java Champion
- Oracle ACE

Me – @vlad_mihalcea
JPA is a device whose power
is only exceeded by its
mystery!
- Hibernate Developer Advocate
- Oracle Java Champion
- High-Performance Java Persistence

Our context

Our context
- Different background
- Working on competing
products
- But «surprisingly» not
disagreeing too much ☺

Our context
Now is the time
... to follow us (no pressure)

Domain
Centric
Our context – Paradigm / Mindset
Data
Centric

Record
oriented
Our context – Paradigm / Mindset
Set
oriented

Focus:
OLTP
Our context – Data Processing Style
Focus:
OLAP

Focus:
OLTP
Our context – Data Processing Style
Focus:
OLAP (OLTP)

Object
Orientation
Our context – Statefulness
Functional
Programming

ORM
Our context – Technologies
SQL

JPA /
Hibernate
Our context – Specific products
jOOQ

Our context – No Size Fits All

Two big topics
Architecture
and
Scalability

Enterprise architecture

Universal Scalability Law

Connection pooling
Metric DB_A (ms) DB_B (ms) DB_C (ms) DB_D (ms) HikariCP (ms)
min 11.174 5.441 24.468 0.860 0.001230
max 129.400 26.110 74.634 74.313 1.014051
mean 13.829 6.477 28.910 1.590 0.003458
p99 20.432 9.944 54.952 3.022 0.010263

The anatomy of a database transaction

Preview – The layers go on in the database
Performant logic in the database

Performance is really simple – @carymillsap
https://www.slideshare.net/carymillsap/the-most-important-things-you-should-know-about-oracle/59

Performance is really simple – @carymillsap
This is not specific to databases!

Example: stored procedures
A stored procedure
essentially moves a loop from
the client to the server.
That’s it.

Microservice friendly

Example: Oracle Call Latency
// Version 1: Loop on the server
try (CallableStatement call = c.prepareCall(`
DECLARE
num integer := ?
BEGIN
dbms_output.get_lines(?, num);
END;
`)) {
call.setInt(1, size);
call.registerOutParameter(
2, Types.ARRAY, "DBMSOUTPUT_LINESARRAY");
call.execute();
...
Loop in here

// Version 2: Loop in the client
for (int i = 0; i < size; i++) {
BEGIN
dbms_output.get_line(?, ?);
END;
`)) {
call.registerOutParameter(1, Types.VARCHAR);
call.registerOutParameter(2, Types.INTEGER);
call.execute();
...
Loop

// Version 2: Loop in the client
for (int i = 0; i < size; i++) {
BEGIN
dbms_output.get_line(?, ?);
END;
`)) {
call.registerOutParameter(1, Types.VARCHAR);
call.registerOutParameter(2, Types.INTEGER);
call.execute();
...
JEP 326: Raw String Literals!

More details here:
https://blog.jooq.org/2017/12/18/the-cost-of-jdbc-server-roundtrips/
Measure 1: Loop on the server 2: Loop in the client
Count 50 50
Sum 69.120 🐇 2088.201 🐢
Min 1.068 🐇 33.738 🐢
Avg 1.382 🐇 41.764 🐢
Max 2.455 🐇 64.498 🐢

Example: stored procedures
Still 50 calls
But fast ones

Example: SQL queries
A SQL query moves the
decision whether to loop to
the server.
(e.g. hash join vs nested loop join)

Algorithm adaptation
CC BY-SA 2.0 https://www.flickr.com/photos/arkhangellohim/8553142120/

My other talk
I’ll actually talk about this in my
other talk

Example from my other talk
How much money did we earn
with each film on each day?

SQL: O(N) (as we’re going to run a hash join)
SELECT
title, store_id, payment_date,
SUM(amount)
FROM film
JOIN inventory USING (film_id)
JOIN rental USING (inventory_id)
JOIN payment USING (rental_id)
GROUP BY film_id, title, store_id, payment_date
ORDER BY title, store_id, payment_date
1 query

Java: O(N^3) (worst case N+1)
Map<Film, Map<Integer, Map<LocalDate, BigDecimal>>> result = ...;
List<Film> films = loadAllFilms();
for (Film film : films) {
Map<Integer, Map<LocalDate, BigDecimal>> dailyPerStore =
result.computeIfAbsent(film, k -> new HashMap<>());
for (Inventory inventory : film.getInventories()) {
Map<LocalDate, BigDecimal> daily =
dailyPerStore.computeIfAbsent(
inventory.getStoreId(), k -> new HashMap());
for (Rental rental : inventory.getRentals())
for (Payment p : rental.getPayments())
daily.compute(
p.getPaymentDate(),
(k, v) -> v == null ? p.getAmount() : p.getAmount().add(v));
}
}
1 query
N queries
N*M queries
N*M*O queries

Interesting talk by @ToonKoppelaars
More details here:
https://www.youtube.com/watch?v=8jiJDflpw4Y
Many layers are traversed

Interesting talk by @ToonKoppelaars
More details here:
https://www.youtube.com/watch?v=8jiJDflpw4Y
Set-based JDBC would be OK too

Example: SQL queries
Big improvement
Better algorithm, because SQL

#SmartDB
#SmartDB
(the opposite extreme to doing everything in Java)

A short story on my (@lukaseder) background
(before doing jOOQ)

SQL centric
A lot of business logic
written in SQL views

An extreme execution plan

A typical execution plan
Example: Downloading file
transfer (FT) logs

A typical execution plan FT log stuff
Payment Archive
Currency conversion
Account Access Control
FT Access Control

Orders of magnitude FT log stuff
10M Rows
Payment Archive
1G Rows
Currency conversion
100K Rows
100M Rows
FT Access Control
100M Rows

Around 0.1 sessions are busy
with this query at any time

A single execution takes on
average 10.73ms
Room for
optimisation
This is
actually slow!

But the system is quite fine.
4M total users
300k daily logins

Reminds me of
https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/
Lots of queries
Low response time

Reminds me of
https://www.periscopedata.com/blog/count-distinct-in-mysql-postgres-sql-server-and-oracle

Let’s talk about a core piece of
business logic, implemented in SQL
Remember that Access Control?

Remember that Access Control?Essentially, contracts are granted access
to products owned by customers

Example
CONT_ID PROD_ID CUST_ID
1 10
1 11
1 12
2 10
3 (all of 100’s products) 100

How to do it? With a view!
CREATE VIEW v_contract_rights AS
SELECT cori_id, cont_id, cust_id, prod_id
FROM contract_rights cori
JOIN products prod ON prod.prod_id = cori.prod_id
JOIN customers cust ON cust.cust_id = prod.cust_id
WHERE cont_id = SYS_CONTEXT('APP', 'CONT_ID')
Transaction-scoped variable

I can see only «my data»

UNION ALL
JOIN products prod ON prod.cust_id = cori.cust_id
AND cori.prod_id IS NULL
Individual Product Access
Complete Customer Access

Now join this view everywhere
FT Access Control

In fact, views are everywhere, like functions
View View
View
View
View
View

Advantages
- Very performant (in Oracle)
- Security in the database
- Reusable logic
- Easy to change
Come see my other talk

Caveats
- Very performant (in Oracle)
- Security in the database
- Reusable logic
- Easy to change
Can be done extensively in commercial DBs.
To some extent in PostgreSQL.
Probably not in MySQL.

Key message
These were OLTP queries, not
OLAP / analytics / reports / ETL.
SQL is a very good language for
business logic.

Key message
• Nothing I’ve shown is extra-ordinary.

Key message
• You’re doing this as well.

Key message
• But you’re doing it in Java, not in SQL.

Key message
• But you’re doing it in Java, not in SQL.
• Should you really do it in Java?

Key message
Think outside the box

Key message

This is a recurring pattern
Huge decrease here
Also better, because SQL

But one size doesn’t fit all
But also in this system, not
everything was written in SQL.
In fact, a single user click may
generate dozens of these queries.

But one size doesn’t fit all
And we haven’t discussed
writing to the database yet!

How about Event Sourcing
and CQRS?
Aren’t we suppose to store
immutable events (facts) in
our database?
Is writing data still a thing in 2018?

Google Spanner
https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf

Write-behind caching

Lazy connection acquisition

Decreases this part

Automatic JDBC statement batching
1 10 20 30 40 50 60 70 80 90 100 1000
0
200
400
600
800
1000
1200
1400
1600
Batch size
Time(ms)
DB_A DB_B DB_C DB_D

But there’s more than batch: Bulk DML!
Which one will perform better?
UPDATE post
SET archived = 1
WHERE archived = 0 AND creation_date < DATE '2018-01-01'
try (Statement s = con.createStatement();
ResultSet rs = s.executeQuery(`
SELECT id FROM post WHERE archived = 0 AND creation_date < DATE '2018-01-01'
`);
PreparedStatement u = con.prepareStatement(`
UPDATE post SET archived = 1 WHERE id = ?
`)) {
while (rs.next()) {
u.setInt(1, rs.getInt(1));
u.executeUpdate();
}
}
Single statement
N+1 with JDBC

But there’s more than batch: Bulk DML!
Which one will perform better?
UPDATE post
SET archived = 1
Single statement
FOR rec IN (
SELECT id
FROM post
) LOOP
UPDATE post
SET archived = 1
WHERE id = rec.id;
END LOOP;
Or in PL/SQL

Bulk DML benchmark results
Measure Time
LOOP 0.224 🐓
FORALL 0.100 🐇
BULK UPDATE 0.065 🐇
Measure Time
LOOP (PreparedStatement per iteration) 9.302 🐢
LOOP (PreparedStatement reused) 8.182 🐢
Batch 0.346 🐓
Bulk UPDATE 0.066 🐇
PL/SQL
More details here:
https://blog.jooq.org/2018/04/19/the-performance-difference-between-
sql-row-by-row-updating-batch-updating-and-bulk-updating/
Java

JDBC fetch size
List<PostCommentSummary> summaries =
em.createQuery(
`select new PostCommentSummary(
p.id, p.title, c.review )
from PostComment c
join c.post p`)
.setHint(QueryHints.HINT_FETCH_SIZE, fetchSize)
.getResultList();

JDBC fetch size
1 10 100 1000 10000
0
100
200
300
400
500
600
Fetch size
Time(ms)
DB_A DB_B DB_C DB_D

You’ve guessed it

JDBC Result Set pagination
em.createQuery(
from PostComment c
join c.post p`)
.setFirstResult(pageStart)
.setMaxResults(pageSize)
.getResultList();

JDBC Result Set pagination – 100k vs 100
Fetch all Fetch limit
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Time(ms)
DB_A DB_B DB_C DB_D

What’s wrong with this?
em.createQuery(
from PostComment c
join c.post p`)
.getResultList();

Try not to use OFFSET Pagination
em.createQuery(
from PostComment c
join c.post p`)
.getResultList();

Try not to use OFFSET Pagination
Offset skips rows in:
- O(N) if an index is used for ORDER BY
- O(N log N) otherwise!
Source: https://use-the-index-luke.com/no-offset

Pagination vs Narrowing down

JDBC Result Set column count
SELECT *
FROM post_comment pc
INNER JOIN post p ON p.id = pc.post_id
INNER JOIN post_details pd ON p.id = pd.id
SELECT pc.review, p.title, pd.created_on

JDBC Result Set column count
All columns Custom projection
0
5
10
15
20
25
30
Time(ms)
DB_A DB_B DB_C DB_D

So we’re here again

Let’s look at execution plans, too

CREATE INDEX i ON post (id, title)
No more post table access

No more join on post!

INNER JOIN post_details pd ON pc.post_id = pd.id
As if it were eliminated manually

Rule of thumb
When you need to write data
back to the database, entities
are very useful.
For read-only data, you should
always use projections!

Caching

Application-level concurrency control

Concurrency Control Patterns
- Come see my other talk

… of the
same coin
Concluding...
Two sides

Concluding…
There isn’t really a
better approach or a
better perspective.

Concluding…
It’s just logic,
distributed between
client and server.

Use the right tool
CC BY 2.0 - https://www.flickr.com/photos/justinbaeder/5317820857/

We promised

We gave you

Not that this is any news
FYI: Gavin King: Creator of Hibernate!

Thank you
Check out our trainings:
http://www.jooq.org/training
https://vladmihalcea.com/courses
Coordinates
• Blogs: http://blog.jooq.org, http:// vladmihalcea.com
• Twitter: @lukaseder, @vlad_mihalcea
• Book: https://leanpub.com/high-performance-java-persistence

Data centric (SQL)? Domain centric (JPA)? Or both? Let's Discuss!

Recomendados

Recomendados

Mais conteúdo relacionado

Último

Último (20)

Destaque

Destaque (20)

Data centric (SQL)? Domain centric (JPA)? Or both? Let's Discuss!