ΠΠ°Π½Π°Π» ΠΏΠΎ ΡΡΡΠΈ Π±ΡΠ΄Π΅Ρ Π² ΡΠΎΡΠΌΠ°ΡΠ΅ "ΠΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ Greenplum Π΄Π»Ρ ΡΠ°ΠΉΠ½ΠΈΠΊΠΎΠ²", Π³Π΄Π΅ Π±ΡΠ΄Ρ Π΄Π΅Π»ΠΈΡΡΡΡ
ΡΠ΅ΡΠ΅ΠΏΡΠ°ΠΌΠΈ ΠΊΠ°ΠΊ Π½Π΅ ΡΡΠΎΠΈΡ ΠΏΠΈΡΠ°ΡΡ ΠΊΠΎΠ΄ Π² Π΄Π°Π½Π½ΠΎΠΉ Π‘Π£ΠΠ ΠΊΠ»Π°ΡΡΠ° MPP ( ΠΌΠ°ΡΡΠΈΠ²Π½ΠΎ-ΠΏΠ°ΡΠ°Π»Π»Π΅Π»ΡΠ½Π°Ρ ΠΎΠ±ΡΠ°Π±ΠΎΡΠΊΠ°), ΠΈΡΠ°ΠΊ ΠΏΠΎΠ΅Ρ Π°Π»ΠΈ!
The channel is a kind of "Greenplum for Dummies" where author will share pieces of advice
how to create optimal code in MPP database, let's go!
ΡΠ΅ΡΠ΅ΠΏΡΠ°ΠΌΠΈ ΠΊΠ°ΠΊ Π½Π΅ ΡΡΠΎΠΈΡ ΠΏΠΈΡΠ°ΡΡ ΠΊΠΎΠ΄ Π² Π΄Π°Π½Π½ΠΎΠΉ Π‘Π£ΠΠ ΠΊΠ»Π°ΡΡΠ° MPP ( ΠΌΠ°ΡΡΠΈΠ²Π½ΠΎ-ΠΏΠ°ΡΠ°Π»Π»Π΅Π»ΡΠ½Π°Ρ ΠΎΠ±ΡΠ°Π±ΠΎΡΠΊΠ°), ΠΈΡΠ°ΠΊ ΠΏΠΎΠ΅Ρ Π°Π»ΠΈ!
The channel is a kind of "Greenplum for Dummies" where author will share pieces of advice
how to create optimal code in MPP database, let's go!
π4
Π‘Π΅ΠΊΡΠ΅Ρ 1 (ΠΡΠΎΠ±Π»Π΅ΠΌΠ° ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΡ)
ΠΠ°ΠΆΠ΅ select ΠΈΠ· ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΡ Ρ ΡΠΈΠ»ΡΡΡΠΎΠΌ ΠΏΠΎ ΡΡΠΆΠ΅Π»ΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΠ΅ ΠΌΠΎΠΆΠ΅Ρ Π·Π°Π²ΠΈΡΠ½ΡΡΡ, Π²ΡΠ·Π²Π°Π² ΡΠΏΠΈΠ»Π» (ΡΡΠ΅ΡΠΊΠ° ΠΏΠ°ΠΌΡΡΠΈ)
Secret 1 (Empty table surprise)
Even empty table query filtered by heavy table may run slow and create spill
ΠΠ°ΠΆΠ΅ select ΠΈΠ· ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΡ Ρ ΡΠΈΠ»ΡΡΡΠΎΠΌ ΠΏΠΎ ΡΡΠΆΠ΅Π»ΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΠ΅ ΠΌΠΎΠΆΠ΅Ρ Π·Π°Π²ΠΈΡΠ½ΡΡΡ, Π²ΡΠ·Π²Π°Π² ΡΠΏΠΈΠ»Π» (ΡΡΠ΅ΡΠΊΠ° ΠΏΠ°ΠΌΡΡΠΈ)
Secret 1 (Empty table surprise)
Even empty table query filtered by heavy table may run slow and create spill
--bad code:
SELECT t.a, t.b, t.c
FROM foo t
where t.key not in (select pk from big_tbl)
--good code:
with a as
(select pk from big_tbl intersect select key from foo)
SELECT t.a, t.b, t.c
from foo t
where t.key not in (select pk from a)
Π‘Π΅ΠΊΡΠ΅Ρ 2 (Π£ΡΠΊΠΎΡΡΠ΅ΠΌ GROUP BY)
ΠΠ±ΡΡΠ½ΡΠΉ GROUP BY ΠΏΡΠΈ Π½Π°Π»ΠΈΡΠΈΠΈ Π±ΠΎΠ»ΡΡΠΎΠ³ΠΎ ΡΠΈΡΠ»Π° ΠΌΠ΅ΡΡΠΈΠΊ ΠΌΠΎΠΆΠ½ΠΎ ΡΡΠΊΠΎΡΠΈΡΡ, ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΡ CTE, ΡΠ°ΠΊΠΆΠ΅ ΡΠ½ΠΈΠ·ΠΈΠ² ΡΠ°Π·ΠΌΠ΅Ρ ΡΠΏΠΈΠ»Π»Π°
Secret 2(Speed up GROUP BY)
Simple GROUP BY operator with a lot of aggregates can be boosted using CTE (cutting spill usage as well)
ΠΠ±ΡΡΠ½ΡΠΉ GROUP BY ΠΏΡΠΈ Π½Π°Π»ΠΈΡΠΈΠΈ Π±ΠΎΠ»ΡΡΠΎΠ³ΠΎ ΡΠΈΡΠ»Π° ΠΌΠ΅ΡΡΠΈΠΊ ΠΌΠΎΠΆΠ½ΠΎ ΡΡΠΊΠΎΡΠΈΡΡ, ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΡ CTE, ΡΠ°ΠΊΠΆΠ΅ ΡΠ½ΠΈΠ·ΠΈΠ² ΡΠ°Π·ΠΌΠ΅Ρ ΡΠΏΠΈΠ»Π»Π°
Secret 2(Speed up GROUP BY)
Simple GROUP BY operator with a lot of aggregates can be boosted using CTE (cutting spill usage as well)
--sub-optimal code:
select dim1, dim2, dim3, sum(m1), sum(m2), ..., sum(m50)
from foo
group by 1, 2, 3
--good code
with a as
(select dim1, dim2, dim3, sum(m1), sum(m2), ..., sum(m25)
group by 1, 2, 3),
b as
(select dim1, dim2, dim3, sum(m26) m26_agg, sum(m27) m27_agg, ..., sum(m50) m50_agg
group by 1, 2, 3)
select a.*, b.m26_agg, b.m27_agg, ..., b.m50_agg
from a, b
where a.dim(1, 2, 3) = b.dim(1, 2, 3) -- ΠΊΡΠ°ΡΠΊΠ°Ρ ΡΠΎΡΠΌΠ° Π·Π°ΠΏΠΈΡΠΈ Π΄Π»Ρ join-Π° ΠΏΠΎ Π²ΡΠ΅ΠΌ ΠΈΠ·ΠΌΠ΅ΡΠ΅Π½ΠΈΡΠΌ / short form of a.dim1 = b.dim1 and ... and a.dim3 = b.dim3
π₯1
Π‘Π΅ΠΊΡΠ΅Ρ 3 (ΠΠ΅Π·Π΄Π΅ΡΡΡΠ°Ρ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠ°)
Π§Π°ΡΡΠΎΠΉ ΠΏΡΠΈΡΠΈΠ½ΠΎΠΉ Π·Π°Π²ΠΈΡΠ°Π½ΠΈΡ Π·Π°ΠΏΡΠΎΡΠ° ΠΈ/ΠΈΠ»ΠΈ Π³Π΅Π½Π΅ΡΠ°ΡΠΈΠΈ ΡΠΏΠΈΠ»Π»Π° ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΎΡΡΡΡΡΡΠ²ΠΈΠ΅ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ΠΏΠΎ ΡΠ°Π±Π»ΠΈΡΠ°ΠΌ Π·Π°ΠΏΡΠΎΡΠ°.
ΠΡΠ»ΠΈ Π²Π°Ρ ETL ΠΎΡΠ½ΠΎΠ²Π°Π½ Π½Π° PL/pgSQL ΡΡΠ½ΠΊΡΠΈΡΡ , ΡΠΎ ΡΠ΅ΠΆΠΈΠΌΠΎΠΌ ΡΠ±ΠΎΡΠ° ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ( Π΅ΡΠ»ΠΈ Π½Π΅ Π·Π°Π΄Π°Π½ Π½Π° ΡΡΠΎΠ²Π½Π΅ GUC - Global User Configuration )
ΡΡΠ»ΠΈΡ ΠΏΠ°ΡΠ°ΠΌΠ΅ΡΡ gp_autostats_mode_in_functions, ΠΊΠΎΡΠΎΡΡΠΉ Π΄ΠΎΠ»ΠΆΠ΅Π½ Π±ΡΡΡ ΡΡΡΠ°Π½ΠΎΠ²Π»Π΅Π½ Π² ΡΠ°ΠΌΠΎΠΉ ΡΡΠ½ΠΊΡΠΈΠΈ ( Π½Π΅ Π΄ΠΎ Π΅Π΅ Π²ΡΠ·ΠΎΠ²Π° )
Secret 3 (Omnipresent statistics)
A common reason for a query to hang and/or generate a spill is the lack of statistics on the query tables.
If your ETL is built on PL/pgSQL functions, then the statistics gather mode ( if not set by GUC - Global User Configuration )
is managed by gp_autostats_mode_in_functions parameter which must be set in the function itself ( otherwise it doesn't matter )
Π§Π°ΡΡΠΎΠΉ ΠΏΡΠΈΡΠΈΠ½ΠΎΠΉ Π·Π°Π²ΠΈΡΠ°Π½ΠΈΡ Π·Π°ΠΏΡΠΎΡΠ° ΠΈ/ΠΈΠ»ΠΈ Π³Π΅Π½Π΅ΡΠ°ΡΠΈΠΈ ΡΠΏΠΈΠ»Π»Π° ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΎΡΡΡΡΡΡΠ²ΠΈΠ΅ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ΠΏΠΎ ΡΠ°Π±Π»ΠΈΡΠ°ΠΌ Π·Π°ΠΏΡΠΎΡΠ°.
ΠΡΠ»ΠΈ Π²Π°Ρ ETL ΠΎΡΠ½ΠΎΠ²Π°Π½ Π½Π° PL/pgSQL ΡΡΠ½ΠΊΡΠΈΡΡ , ΡΠΎ ΡΠ΅ΠΆΠΈΠΌΠΎΠΌ ΡΠ±ΠΎΡΠ° ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ( Π΅ΡΠ»ΠΈ Π½Π΅ Π·Π°Π΄Π°Π½ Π½Π° ΡΡΠΎΠ²Π½Π΅ GUC - Global User Configuration )
ΡΡΠ»ΠΈΡ ΠΏΠ°ΡΠ°ΠΌΠ΅ΡΡ gp_autostats_mode_in_functions, ΠΊΠΎΡΠΎΡΡΠΉ Π΄ΠΎΠ»ΠΆΠ΅Π½ Π±ΡΡΡ ΡΡΡΠ°Π½ΠΎΠ²Π»Π΅Π½ Π² ΡΠ°ΠΌΠΎΠΉ ΡΡΠ½ΠΊΡΠΈΠΈ ( Π½Π΅ Π΄ΠΎ Π΅Π΅ Π²ΡΠ·ΠΎΠ²Π° )
Secret 3 (Omnipresent statistics)
A common reason for a query to hang and/or generate a spill is the lack of statistics on the query tables.
If your ETL is built on PL/pgSQL functions, then the statistics gather mode ( if not set by GUC - Global User Configuration )
is managed by gp_autostats_mode_in_functions parameter which must be set in the function itself ( otherwise it doesn't matter )
CREATE OR REPLACE FUNCTION public.fn_foo()
RETURNS boolean
LANGUAGE plpgsql
AS $function$
begin
set gp_autostats_mode_in_functions = on_no_stats; -- Π±Π΅Π·ΡΡΠ»Π»ΠΎΠ²Π½ΡΠΉ ΡΠ±ΠΎΡ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ΠΏΠΎ ΡΠ°Π±Π»ΠΈΡΠ°ΠΌ, ΠΌΠΎΠ΄ΠΈΡΠΈΡΠΈΡΡΠ΅ΠΌΡΠΌ Π² ΡΡΠ½ΠΊΡΠΈΠΈ / always gather stat for all tables modified by function
-- ΠΡΡΠ³ΠΈΠ΅ Π·Π½Π°ΡΠ΅Π½ΠΈΡ / other values:
-- none - Π½Π΅ ΡΠΎΠ±ΠΈΡΠ°ΡΡ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΡ / don't gather stat,
-- on_change - ΡΠΎΠ±ΠΈΡΠ°ΡΡ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΡ Π΅ΡΠ»ΠΈ ΡΠΈΡΠ»ΠΎ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½Π½ΡΡ Π·Π°ΠΏΠΈΡΠ΅ΠΉ ΠΏΡΠ΅Π²ΡΡΠΈΠ»ΠΎ Π·Π°Π΄Π°Π½Π½ΡΠΉ ΠΏΠΎΡΠΎΠ³ / gather stat if the number of modified records exceeds specified threshold
...
π3π₯1
Π‘Π΅ΠΊΡΠ΅Ρ 4 (ΠΡΡΠΎΡΠΎΠΆΠ½ΠΎ - Π Π΅ΠΊΡΡΡΠΈΡ!)
Π‘Π»Π΅Π΄ΡΠ΅Ρ ΠΈΠ·Π±Π΅Π³Π°ΡΡ ΡΠ΅ΠΊΡΡΡΠΈΠ²Π½ΡΡ Π·Π°ΠΏΡΠΎΡΠΎΠ² , Ρ.ΠΊ. join ΡΠ°Π±Π»ΠΈΡΡ Ρ Π½Π΅ΠΉ ΠΆΠ΅ Π½Π΅ΡΠΎΠ²ΠΌΠ΅ΡΡΠΈΠΌ Ρ ΠΊΠΎΠ½ΡΠ΅ΠΏΡΠΈΠ΅ΠΉ shared-nothing.
ΠΠ½ΡΠΌΠΈ ΡΠ»ΠΎΠ²Π°ΠΌΠΈ, ΡΠ°ΠΊΠ°Ρ ΠΎΠΏΠ΅ΡΠ°ΡΠΈΡ ΠΏΡΠΈΠ²ΠΎΠ΄ΠΈΡ ΠΊ ΡΠΈΡΠ°ΠΆΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΡΠ°Π±Π»ΠΈΡΡ Π½Π° ΠΊΠ°ΠΆΠ΄ΡΠΉ ΡΠ·Π΅Π» ΠΊΠ»Π°ΡΠ΅ΡΠ°, Ρ.ΠΊ.
Π½Π΅ ΠΌΠΎΠΆΠ΅Ρ Π²ΡΠΏΠΎΠ»Π½ΡΡΡΡΡ Π»ΠΎΠΊΠ°Π»ΡΠ½ΠΎ (ΡΠΎΠ»ΡΠΊΠΎ Π½Π° ΡΠ²ΠΎΠ΅ΠΌ) Π²Π²ΠΈΠ΄Ρ ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ ΡΠ°Π±Π»ΠΈΡΡ ΡΠΎΠ»ΡΠΊΠΎ ΠΏΠΎ ΠΎΠ΄Π½ΠΎΠΌΡ ΠΈΠ· ΠΊΠ»ΡΡΠ΅ΠΉ ΠΏΡΠ΅Π΄ΠΈΠΊΠ°ΡΠ° Π·Π°ΠΏΡΠΎΡΠ°.
ΠΠ΄Π½Π°ΠΊΠΎ, Π΅ΡΠ»ΠΈ join-ΠΈΡΡ ΡΠ°Π±Π»ΠΈΡΡ Π½Π΅ ΡΠ°ΠΌΡ Π½Π° ΡΠ΅Π±Ρ ( ΡΠ°ΠΏΡΠ΅Π΄Π΅Π»Π΅Π½Π½ΡΡ ΠΏΠΎ AGREEMENT_PK Π² ΠΏΡΠΈΠΌΠ΅ΡΠ΅ Π½ΠΈΠΆΠ΅), Π° Ρ Π΅Π΅ ΠΊΠΎΠΏΠΈΠ΅ΠΉ, ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½Π½ΠΎΠΉ ΠΏΠΎ PREVIOUS_AGREEMENT_PK( 2Π΅ ΠΏΠ»Π΅ΡΠΎ ΠΏΡΠ΅Π΄ΠΈΠΊΠ°ΡΠ°),
ΡΠ°Π·ΠΌΠ΅Ρ ΡΠΏΠΈΠ»Π»Π° ΠΌΠΎΠΆΠ½ΠΎ Π·Π°ΠΌΠ΅ΡΠ½ΠΎ ΡΠΎΠΊΡΠ°ΡΠΈΡΡ Π½Π° ΠΏΠΎΡΡΠ΄ΠΎΠΊ.
Secret 4 (Beware - Recursion!)
Recursive queries should be avoided, because joining a table with itself is incompatible with the shared-nothing concept.
In other words, such an operation leads to replicating the table to each cluster node, because it cannot be performed locally (only on its own) due to the distribution of the table only by one of the keys of the query predicate.
However, If you join the table not to itself (distributed by AGREEMENT_PK in the example below), but with its copy distributed by PREVIOUS_AGREEMENT_PK (the parent key in the predicate),
the size of the spill can be significantly reduced by an order of magnitude.
Π‘Π»Π΅Π΄ΡΠ΅Ρ ΠΈΠ·Π±Π΅Π³Π°ΡΡ ΡΠ΅ΠΊΡΡΡΠΈΠ²Π½ΡΡ Π·Π°ΠΏΡΠΎΡΠΎΠ² , Ρ.ΠΊ. join ΡΠ°Π±Π»ΠΈΡΡ Ρ Π½Π΅ΠΉ ΠΆΠ΅ Π½Π΅ΡΠΎΠ²ΠΌΠ΅ΡΡΠΈΠΌ Ρ ΠΊΠΎΠ½ΡΠ΅ΠΏΡΠΈΠ΅ΠΉ shared-nothing.
ΠΠ½ΡΠΌΠΈ ΡΠ»ΠΎΠ²Π°ΠΌΠΈ, ΡΠ°ΠΊΠ°Ρ ΠΎΠΏΠ΅ΡΠ°ΡΠΈΡ ΠΏΡΠΈΠ²ΠΎΠ΄ΠΈΡ ΠΊ ΡΠΈΡΠ°ΠΆΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΡΠ°Π±Π»ΠΈΡΡ Π½Π° ΠΊΠ°ΠΆΠ΄ΡΠΉ ΡΠ·Π΅Π» ΠΊΠ»Π°ΡΠ΅ΡΠ°, Ρ.ΠΊ.
Π½Π΅ ΠΌΠΎΠΆΠ΅Ρ Π²ΡΠΏΠΎΠ»Π½ΡΡΡΡΡ Π»ΠΎΠΊΠ°Π»ΡΠ½ΠΎ (ΡΠΎΠ»ΡΠΊΠΎ Π½Π° ΡΠ²ΠΎΠ΅ΠΌ) Π²Π²ΠΈΠ΄Ρ ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ ΡΠ°Π±Π»ΠΈΡΡ ΡΠΎΠ»ΡΠΊΠΎ ΠΏΠΎ ΠΎΠ΄Π½ΠΎΠΌΡ ΠΈΠ· ΠΊΠ»ΡΡΠ΅ΠΉ ΠΏΡΠ΅Π΄ΠΈΠΊΠ°ΡΠ° Π·Π°ΠΏΡΠΎΡΠ°.
ΠΠ΄Π½Π°ΠΊΠΎ, Π΅ΡΠ»ΠΈ join-ΠΈΡΡ ΡΠ°Π±Π»ΠΈΡΡ Π½Π΅ ΡΠ°ΠΌΡ Π½Π° ΡΠ΅Π±Ρ ( ΡΠ°ΠΏΡΠ΅Π΄Π΅Π»Π΅Π½Π½ΡΡ ΠΏΠΎ AGREEMENT_PK Π² ΠΏΡΠΈΠΌΠ΅ΡΠ΅ Π½ΠΈΠΆΠ΅), Π° Ρ Π΅Π΅ ΠΊΠΎΠΏΠΈΠ΅ΠΉ, ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½Π½ΠΎΠΉ ΠΏΠΎ PREVIOUS_AGREEMENT_PK( 2Π΅ ΠΏΠ»Π΅ΡΠΎ ΠΏΡΠ΅Π΄ΠΈΠΊΠ°ΡΠ°),
ΡΠ°Π·ΠΌΠ΅Ρ ΡΠΏΠΈΠ»Π»Π° ΠΌΠΎΠΆΠ½ΠΎ Π·Π°ΠΌΠ΅ΡΠ½ΠΎ ΡΠΎΠΊΡΠ°ΡΠΈΡΡ Π½Π° ΠΏΠΎΡΡΠ΄ΠΎΠΊ.
Secret 4 (Beware - Recursion!)
Recursive queries should be avoided, because joining a table with itself is incompatible with the shared-nothing concept.
In other words, such an operation leads to replicating the table to each cluster node, because it cannot be performed locally (only on its own) due to the distribution of the table only by one of the keys of the query predicate.
However, If you join the table not to itself (distributed by AGREEMENT_PK in the example below), but with its copy distributed by PREVIOUS_AGREEMENT_PK (the parent key in the predicate),
the size of the spill can be significantly reduced by an order of magnitude.
-- bad code
with recursive
agreement_chain as
(
select
AGREEMENT_PK,
PREVIOUS_AGREEMENT_PK,
counter_rec,
from foo
where is_first_agreement = 'y'
union all
select
t1.AGREEMENT_PK,
t2.AGREEMENT_PK as PREVIOUS_AGREEMENT_PK,
t2.counter_rec + 1,
from foo t1
inner join agreement_chain t2 on t2.AGREEMENT_PK = t1.PREVIOUS_AGREEMENT_PK
and t2.counter_rec < 10 -- ΠΎΠ³ΡΠ°Π½ΠΈΡΠ΅Π½ΠΈΠ΅ ΠΏΠΎ ΡΠ΅ΠΊΡΡΡΠΈΠΈ ΡΡΠΎΠ±Ρ Π½Π΅ ΡΠΉΡΠΈ Π² Π±Π΅ΡΠΊΠΎΠ½Π΅ΡΠ½ΡΠΉ ΡΠΈΠΊΠ» / limitation on recursion to avoid an infinite loop
)
select
AGREEMENT_PK,
PREVIOUS_AGREEMENT_PK
from agreement_chain
-- good code
create table foo_mirror as select * from foo distribued by (PREVIOUS_AGREEMENT_PK);
with recursive
agreement_chain as
(
select
AGREEMENT_PK,
PREVIOUS_AGREEMENT_PK,
counter_rec,
from foo
where is_first_agreement = 'y'
union all
select
t1.AGREEMENT_PK,
t2.AGREEMENT_PK as PREVIOUS_AGREEMENT_PK,
t2.counter_rec + 1,
from foo_mirror t1
inner join agreement_chain t2 on t2.AGREEMENT_PK = t1.PREVIOUS_AGREEMENT_PK
and t2.counter_rec < 10
)
select
AGREEMENT_PK,
PREVIOUS_AGREEMENT_PK
from agreement_chain
Π‘Π΅ΠΊΡΠ΅Ρ 5 (Join Π±Π΅Π· ΠΊΠ»ΡΡΠ°)
ΠΡΠ΅ ΠΎΠ΄ΠΈΠ½ ΠΏΡΠΈΠΌΠ΅Ρ ΠΏΠΎΡΠ΅Π½ΡΠΈΠ°Π»ΡΠ½ΠΎΠΉ Π±ΠΎΠΌΠ±Ρ - ΡΡΠΎ Π·Π°ΠΏΡΠΎΡΡ Π±Π΅Π· '=' Π² ΠΏΡΠ΅Π΄ΠΈΠΊΠ°ΡΠ΅ ΡΠΎΠ΅Π΄ΠΈΠ½Π΅Π½ΠΈΡ ΡΠ°Π±Π»ΠΈΡ,
Π² ΡΠ°ΡΡΠ½ΠΎΡΡΠΈ ΡΠ°ΠΊΠΎΠΉ, Π³Π΄Π΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΠΎΠΏΠ΅ΡΠ°ΡΠΈΡ between Π² join-Π΅ ΡΠ°Π±Π»ΠΈΡ Π±Π΅Π· ΠΊΠ»ΡΡΠ°, ΠΊΠΎΡΠΎΡΠ°Ρ Π² GP Π²ΡΠΏΠΎΠ»Π½ΡΠ΅ΡΡΡ ΠΊΡΠ°ΠΉΠ½Π΅ Π½Π΅ΠΎΠΏΡΠΈΠΌΠ°Π»ΡΠ½ΠΎ ( ΡΠ΅ΡΠ΅Π· Nested Loop, Π° Π½Π΅ ΡΠ΅ΡΠ΅Π· Π±ΡΡΡΡΡΠΉ Hash Join )
ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, Π΅ΡΠ»ΠΈ Π² ΡΠ°Π±Π»ΠΈΡΠ΅ ΠΏΡΠΎΠ΄Π°ΠΆ foo ΠΎΡΡΡΡΡΡΠ²ΡΠ΅Ρ ΠΊΠ»ΡΡ ΠΈΠ·ΠΌΠ΅ΡΠ΅Π½ΠΈΡ dim_periods, ΠΏΡΡΠΌΠΎΠ΅ ΡΠ΅ΡΠ΅Π½ΠΈΠ΅ Π² Π»ΠΎΠ± Π²ΡΡΠ°ΡΠΈΡΡ ΠΏΠ΅ΡΠΈΠΎΠ΄ Π΄Π»Ρ ΠΊΠ°ΠΆΠ΄ΠΎΠΉ ΡΡΡΠΎΠΊΠΈ foo - ΡΡΠΎ Π»ΠΎΠ²ΡΡΠΊΠ°.
ΠΠ΄Π½Π°ΠΊΠΎ, Π΅ΡΠ»ΠΈ Π² foo ΡΡΠΎΠ²Π΅Π½Ρ Π΄Π΅ΡΠ°Π»ΠΈΠ·Π°ΡΠΈΠΈ Π΄Π°Π½Π½ΡΡ - Π΄Π΅Π½Ρ,
ΡΠΎ Π² ΠΎΠ±ΡΠ΅ΠΌ ΡΠ»ΡΡΠ°Π΅ ΡΠΈΡΠ»ΠΎ ΡΡΡΠΎΠΊ Π±ΡΠ΄Π΅Ρ Π½Π° ΠΏΠΎΡΡΠ΄ΠΊΠΈ Π±ΠΎΠ»ΡΡΠ΅ ΡΠΈΡΠ»Π° Π΄Π½Π΅ΠΉ, Π² ΠΊΠΎΡΠΎΡΡΠ΅ Π±ΡΠ»ΠΈ ΠΏΡΠΎΠ΄Π°ΠΆΠΈ ΠΈ
Π² ΡΠ°ΠΊΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π»ΠΎΠ²ΡΡΠΊΠΈ ΠΌΠΎΠΆΠ½ΠΎ ΠΈΠ·Π±Π΅ΠΆΠ°ΡΡ, ΠΏΡΠΈΠ²Π΅Π΄Ρ Π·Π°ΠΏΡΠΎΡ ΠΊ ΡΠΎΡΠΌΠ΅ equi-join:
Secret 5(Non-Equi Join)
Another example of a potential bomb is queries without '=' in the table join predicate.
Particularly, one that uses the between operation in a join of tables without a key, which in GP is performed extremely suboptimally (through Nested Loop, and not through fast Hash Join)
For example, if the sales table foo is missing the dim_periods dimension key, a straight forward solution to pull out period for each row of foo is a trap.
However, if foo has a data granularity of day,
then in the general case the number of rows will be orders of magnitude greater than the number of days in which there were sales and
in this case, the pitfall can be avoided by transforming the query in equi-join form:
ΠΡΠ΅ ΠΎΠ΄ΠΈΠ½ ΠΏΡΠΈΠΌΠ΅Ρ ΠΏΠΎΡΠ΅Π½ΡΠΈΠ°Π»ΡΠ½ΠΎΠΉ Π±ΠΎΠΌΠ±Ρ - ΡΡΠΎ Π·Π°ΠΏΡΠΎΡΡ Π±Π΅Π· '=' Π² ΠΏΡΠ΅Π΄ΠΈΠΊΠ°ΡΠ΅ ΡΠΎΠ΅Π΄ΠΈΠ½Π΅Π½ΠΈΡ ΡΠ°Π±Π»ΠΈΡ,
Π² ΡΠ°ΡΡΠ½ΠΎΡΡΠΈ ΡΠ°ΠΊΠΎΠΉ, Π³Π΄Π΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΠΎΠΏΠ΅ΡΠ°ΡΠΈΡ between Π² join-Π΅ ΡΠ°Π±Π»ΠΈΡ Π±Π΅Π· ΠΊΠ»ΡΡΠ°, ΠΊΠΎΡΠΎΡΠ°Ρ Π² GP Π²ΡΠΏΠΎΠ»Π½ΡΠ΅ΡΡΡ ΠΊΡΠ°ΠΉΠ½Π΅ Π½Π΅ΠΎΠΏΡΠΈΠΌΠ°Π»ΡΠ½ΠΎ ( ΡΠ΅ΡΠ΅Π· Nested Loop, Π° Π½Π΅ ΡΠ΅ΡΠ΅Π· Π±ΡΡΡΡΡΠΉ Hash Join )
ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, Π΅ΡΠ»ΠΈ Π² ΡΠ°Π±Π»ΠΈΡΠ΅ ΠΏΡΠΎΠ΄Π°ΠΆ foo ΠΎΡΡΡΡΡΡΠ²ΡΠ΅Ρ ΠΊΠ»ΡΡ ΠΈΠ·ΠΌΠ΅ΡΠ΅Π½ΠΈΡ dim_periods, ΠΏΡΡΠΌΠΎΠ΅ ΡΠ΅ΡΠ΅Π½ΠΈΠ΅ Π² Π»ΠΎΠ± Π²ΡΡΠ°ΡΠΈΡΡ ΠΏΠ΅ΡΠΈΠΎΠ΄ Π΄Π»Ρ ΠΊΠ°ΠΆΠ΄ΠΎΠΉ ΡΡΡΠΎΠΊΠΈ foo - ΡΡΠΎ Π»ΠΎΠ²ΡΡΠΊΠ°.
ΠΠ΄Π½Π°ΠΊΠΎ, Π΅ΡΠ»ΠΈ Π² foo ΡΡΠΎΠ²Π΅Π½Ρ Π΄Π΅ΡΠ°Π»ΠΈΠ·Π°ΡΠΈΠΈ Π΄Π°Π½Π½ΡΡ - Π΄Π΅Π½Ρ,
ΡΠΎ Π² ΠΎΠ±ΡΠ΅ΠΌ ΡΠ»ΡΡΠ°Π΅ ΡΠΈΡΠ»ΠΎ ΡΡΡΠΎΠΊ Π±ΡΠ΄Π΅Ρ Π½Π° ΠΏΠΎΡΡΠ΄ΠΊΠΈ Π±ΠΎΠ»ΡΡΠ΅ ΡΠΈΡΠ»Π° Π΄Π½Π΅ΠΉ, Π² ΠΊΠΎΡΠΎΡΡΠ΅ Π±ΡΠ»ΠΈ ΠΏΡΠΎΠ΄Π°ΠΆΠΈ ΠΈ
Π² ΡΠ°ΠΊΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π»ΠΎΠ²ΡΡΠΊΠΈ ΠΌΠΎΠΆΠ½ΠΎ ΠΈΠ·Π±Π΅ΠΆΠ°ΡΡ, ΠΏΡΠΈΠ²Π΅Π΄Ρ Π·Π°ΠΏΡΠΎΡ ΠΊ ΡΠΎΡΠΌΠ΅ equi-join:
Secret 5(Non-Equi Join)
Another example of a potential bomb is queries without '=' in the table join predicate.
Particularly, one that uses the between operation in a join of tables without a key, which in GP is performed extremely suboptimally (through Nested Loop, and not through fast Hash Join)
For example, if the sales table foo is missing the dim_periods dimension key, a straight forward solution to pull out period for each row of foo is a trap.
However, if foo has a data granularity of day,
then in the general case the number of rows will be orders of magnitude greater than the number of days in which there were sales and
in this case, the pitfall can be avoided by transforming the query in equi-join form:
--bad code (ΠΌΠ΅ΡΠΎΠ΄ ΡΠΎΠ΅Π΄ΠΈΠ½Π΅Π½ΠΈΡ/ join method - Nested Loop )
select foo.*, d.date_from, d.date_to
from foo
join dim_periods d on
foo.dt between d.date_from d.and date_to
--good code ( Π²ΠΌΠ΅ΡΡΠΎ NL ΠΎΠΏΡΠΈΠΌΠΈΠ·Π°ΡΠΎΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅Ρ Π² ΠΈΡΠΎΠ³Π΅ Hash Join / Instead of NL optimizer finally uses Hash Join )
with a as
(select distinct dt from foo),
x as ( a.dt, s.* from dim_periods s, a
where a.dt between s.date_from s.and date_to ) -- ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΠΌ NL Π»ΠΈΡΡ Π΄Π»Ρ ΠΌΠ°Π»ΠΎΠΉ Π΄ΠΎΠ»ΠΈ ΠΎΡ foo / use NL path only for tiny % of foo
select foo.*, x.date_from, x.date_to
from foo join x on foo.dt = x.dt
π3
Π‘Π΅ΠΊΡΠ΅Ρ 6 (ΠΡΠΎΠ±Π»Π΅ΠΌΠ° ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΡ, Ρ.2)
Π‘Π΅ΠΊΡΠ΅Ρ 1 ΠΏΠΎΠ²Π΅Π΄Π°Π» Π½Π°ΠΌ, ΡΡΠΎ select ΠΈΠ· ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΡ ΠΌΠΎΠΆΠ΅Ρ ΠΏΠΎΠ΄Π²ΠΈΡΠ°ΡΡ, ΡΠΎΠ·Π΄Π°Π²Π°Ρ ΡΠΏΠΈΠ»Π».
ΠΠΊΠ°Π·ΡΠ²Π°Π΅ΡΡΡ, Π·Π°ΠΏΡΠΎΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΡΡΠΈΠΉ ΠΏΡΡΡΡΡ ΡΠ°Π±Π»ΠΈΡΡ Π² ΠΊΠ°ΡΠ΅ΡΡΠ²Π΅ ΡΠΈΠ»ΡΡΡΠ°, ΠΏΠΎ ΠΊΠΎΡΠΎΡΠΎΠΉ Π½Π΅Ρ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ΠΌΠΎΠΆΠ΅Ρ ΡΠΎΠ·Π΄Π°ΡΡ ΡΠ΅ ΠΆΠ΅ ΠΏΡΠΎΠ±Π»Π΅ΠΌΡ.
Π ΠΏΡΠΈΠΌΠ΅ΡΠ΅ Π½ΠΈΠΆΠ΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΡΠ΅Π°Π»ΡΠ½ΡΠΉ ΠΊΠ΅ΠΉΡ Π² ΠΊΡΡΠΏΠ½ΠΎΠΌ DWH Π½Π° ΡΠΎΡΠ½ΠΈ TB Π³Π΄Π΅ ΠΈΠ΄Π΅Ρ select ΠΈΠ· ΠΈΠ½ΡΠ΅ΡΡΠ΅ΠΉΡΠ½ΠΎΠΉ ΡΡΠ½ΠΊΡΠΈΠΈ ΠΏΠΎ Π΄ΠΈΠ°ΠΏΠ°Π·ΠΎΠ½Ρ Π΄Π°Ρ Ρ ΡΠΈΠ»ΡΡΡΠ°ΡΠΈΠ΅ΠΉ ΠΏΠΎ ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΠ΅.
Π‘Π±ΠΎΡ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ΠΏΠΎ ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΠ΅ foo ΡΠ΅ΡΠ°Π΅Ρ ΠΏΡΠΎΠ±Π»Π΅ΠΌΡ.
Π ΡΠ»ΠΎΠ²Ρ, Π² ΠΈΠ΄Π΅Π°Π»Π΅, Π½Π΅ ΡΡΠΎΠΈΡ Π·Π»ΠΎΡΠΏΠΎΡΡΠ΅Π±Π»ΡΡΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΠ΅ΠΌ ΡΡΠ½ΠΊΡΠΈΠΉ Π²ΠΌΠ΅ΡΡΠΎ ΡΠ°Π±Π»ΠΈΡ Π² Π·Π°ΠΏΡΠΎΡΠ°Ρ , Ρ.ΠΊ. Π² ΡΡΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ Legacy optimizer ΠΎΡ Postgres,
Π° Π½Π΅ Π½Π°ΡΠΈΠ²Π½ΡΠΉ GPORCA
Secret 6 (Empty table problem, part 2)
Secret 1 told us that select from an empty table can hang, creating a spill.
It turns out that a query using an empty table as a filter for which there are no statistics can also fall into the same problem.
The example below uses a real case in a large DWH for hundreds of TB, where there is a select from an interface function by a range of versions with filtering by an empty table.
Collecting statistics on an empty table foo solves the problem.
By the way, ideally, you should avoid the use of functions instead of tables in queries, because in this case the Legacy optimizer from Postgres is used, and not the native GPORCA.
Π‘Π΅ΠΊΡΠ΅Ρ 1 ΠΏΠΎΠ²Π΅Π΄Π°Π» Π½Π°ΠΌ, ΡΡΠΎ select ΠΈΠ· ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΡ ΠΌΠΎΠΆΠ΅Ρ ΠΏΠΎΠ΄Π²ΠΈΡΠ°ΡΡ, ΡΠΎΠ·Π΄Π°Π²Π°Ρ ΡΠΏΠΈΠ»Π».
ΠΠΊΠ°Π·ΡΠ²Π°Π΅ΡΡΡ, Π·Π°ΠΏΡΠΎΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΡΡΠΈΠΉ ΠΏΡΡΡΡΡ ΡΠ°Π±Π»ΠΈΡΡ Π² ΠΊΠ°ΡΠ΅ΡΡΠ²Π΅ ΡΠΈΠ»ΡΡΡΠ°, ΠΏΠΎ ΠΊΠΎΡΠΎΡΠΎΠΉ Π½Π΅Ρ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ΠΌΠΎΠΆΠ΅Ρ ΡΠΎΠ·Π΄Π°ΡΡ ΡΠ΅ ΠΆΠ΅ ΠΏΡΠΎΠ±Π»Π΅ΠΌΡ.
Π ΠΏΡΠΈΠΌΠ΅ΡΠ΅ Π½ΠΈΠΆΠ΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΡΠ΅Π°Π»ΡΠ½ΡΠΉ ΠΊΠ΅ΠΉΡ Π² ΠΊΡΡΠΏΠ½ΠΎΠΌ DWH Π½Π° ΡΠΎΡΠ½ΠΈ TB Π³Π΄Π΅ ΠΈΠ΄Π΅Ρ select ΠΈΠ· ΠΈΠ½ΡΠ΅ΡΡΠ΅ΠΉΡΠ½ΠΎΠΉ ΡΡΠ½ΠΊΡΠΈΠΈ ΠΏΠΎ Π΄ΠΈΠ°ΠΏΠ°Π·ΠΎΠ½Ρ Π΄Π°Ρ Ρ ΡΠΈΠ»ΡΡΡΠ°ΡΠΈΠ΅ΠΉ ΠΏΠΎ ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΠ΅.
Π‘Π±ΠΎΡ ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ ΠΏΠΎ ΠΏΡΡΡΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΠ΅ foo ΡΠ΅ΡΠ°Π΅Ρ ΠΏΡΠΎΠ±Π»Π΅ΠΌΡ.
Π ΡΠ»ΠΎΠ²Ρ, Π² ΠΈΠ΄Π΅Π°Π»Π΅, Π½Π΅ ΡΡΠΎΠΈΡ Π·Π»ΠΎΡΠΏΠΎΡΡΠ΅Π±Π»ΡΡΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΠ΅ΠΌ ΡΡΠ½ΠΊΡΠΈΠΉ Π²ΠΌΠ΅ΡΡΠΎ ΡΠ°Π±Π»ΠΈΡ Π² Π·Π°ΠΏΡΠΎΡΠ°Ρ , Ρ.ΠΊ. Π² ΡΡΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ Legacy optimizer ΠΎΡ Postgres,
Π° Π½Π΅ Π½Π°ΡΠΈΠ²Π½ΡΠΉ GPORCA
Secret 6 (Empty table problem, part 2)
Secret 1 told us that select from an empty table can hang, creating a spill.
It turns out that a query using an empty table as a filter for which there are no statistics can also fall into the same problem.
The example below uses a real case in a large DWH for hundreds of TB, where there is a select from an interface function by a range of versions with filtering by an empty table.
Collecting statistics on an empty table foo solves the problem.
By the way, ideally, you should avoid the use of functions instead of tables in queries, because in this case the Legacy optimizer from Postgres is used, and not the native GPORCA.
select deal_rk
from fn_foo( p_dt_from :='2024-09-01', p_dt_to :='2024-09-02')
where 1 = 1
and src_cd = 'IMOEX'
and coalesce(add_info_04, '&&&') not like '%NON_RESIDENT%'
and id in (select risk_contract_id from foo) -- ΠΏΠ»ΠΎΡ ΠΎΠΉ ΡΠΈΠ»ΡΡΡ Π΅ΡΠ»ΠΈ Π΄Π°ΠΆΠ΅ ΡΠ°Π±Π»ΠΈΡΠ° ΠΏΡΡΡΠ°Ρ, Π½ΠΎ Π±Π΅Π· ΡΡΠ°ΡΠΈΡΡΠΈΠΊΠΈ / bad filter if even the table is empty but without statistics
π3
Π‘Π΅ΠΊΡΠ΅Ρ 7(ΠΠΎΠ»ΠΈΠ²Π°Ρ Π½Π΅ Π²ΡΠ΄Π΅ΡΠΆΠΈΡ Π΄Π²ΠΎΠΈΡ
ΠΈΠ»ΠΈ ΠΏΡΠΎΠ±Π»Π΅ΠΌΠ° ΠΏΠ΅ΡΠ΅ΠΊΠΎΡΠ°)
ΠΠ° Π΄Π½ΡΡ ΡΠ»ΠΎΠ²ΠΈΠ»ΠΈ Π½Π° ΠΏΡΠΎΠ΄Π΅ Π½Π΅ΠΎΡΠ΅Π²ΠΈΠ΄Π½ΡΡ ΠΎΡΠΈΠ±ΠΊΡ ΠΏΡΠΈ left join 2Ρ ΡΠ°Π±Π»ΠΈΡ ΡΠΎΠ²Π΅ΡΡΠ΅Π½Π½ΠΎ ΠΎΠ±ΡΡΠ½ΡΡ Π΄Π»Ρ Π½Π°ΡΠ΅Π³ΠΎ Π₯Π ΡΠ°Π·ΠΌΠ΅ΡΠΎΠ²
ΠΡΠΈ ΡΡΠΎΠΌ Π°Π½Π°Π»ΠΎΠ³ΠΈΡΠ½ΡΠΉ Π·Π°ΠΏΡΠΎΡ Ρ inner join ΠΎΡΡΠ°Π±Π°ΡΡΠ²Π°Π΅Ρ Π±Π΅Π· ΠΏΡΠΎΠ±Π»Π΅ΠΌ :
Secret 7 (Bolivar can't bring two or the problem of skew)
Recently we caught a non-obvious error in production when left joining 2 tables of completely normal sizes for our data warehouse.
At the same time, a similar query with inner join works without problems:
ΠΡΠΊΡΡΡΠΈΠ΅ ΠΏΠΎΠΊΠ°Π·Π°Π»ΠΎ, ΡΡΠΎ Π² ΡΠ°Π±Π»ΠΈΡΠ΅ small_tbl ΠΏΠΎΠ»ΠΎΠ²ΠΈΠ½Π° ΠΊΠ»ΡΡΠ΅ΠΉ ΠΎΠΊΠ°Π·Π°Π»ΠΈΡΡ NULL + addr_fk ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΊΠ»ΡΡΠΎΠΌ ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ Π² ΡΠ°Π±Π»ΠΈΡΠ΅.
ΠΡΠΎΠ±Π»Π΅ΠΌΡ ΡΠ΅ΡΠΈΠ»ΠΈ ΠΌΠ°Π»ΠΎΠΉ ΠΊΡΠΎΠ²ΡΡ, Π·Π°ΠΌΠ΅Π½ΠΈΠ² left join ΠΎΠ±ΡΠ΅Π΄ΠΈΠ½Π΅Π½ΠΈΠ΅ΠΌ Π΄Π²ΡΡ ΠΌΠ½ΠΎΠΆΠ΅ΡΡΠ²:
The analysis showed that in the small_tbl table half of the keys were NULL + addr_fk is the distribution key in the table. The problem was solved with little effort by replacing left join with a union of two sets:
ΠΠ° Π΄Π½ΡΡ ΡΠ»ΠΎΠ²ΠΈΠ»ΠΈ Π½Π° ΠΏΡΠΎΠ΄Π΅ Π½Π΅ΠΎΡΠ΅Π²ΠΈΠ΄Π½ΡΡ ΠΎΡΠΈΠ±ΠΊΡ ΠΏΡΠΈ left join 2Ρ ΡΠ°Π±Π»ΠΈΡ ΡΠΎΠ²Π΅ΡΡΠ΅Π½Π½ΠΎ ΠΎΠ±ΡΡΠ½ΡΡ Π΄Π»Ρ Π½Π°ΡΠ΅Π³ΠΎ Π₯Π ΡΠ°Π·ΠΌΠ΅ΡΠΎΠ²
ΠΡΠΈ ΡΡΠΎΠΌ Π°Π½Π°Π»ΠΎΠ³ΠΈΡΠ½ΡΠΉ Π·Π°ΠΏΡΠΎΡ Ρ inner join ΠΎΡΡΠ°Π±Π°ΡΡΠ²Π°Π΅Ρ Π±Π΅Π· ΠΏΡΠΎΠ±Π»Π΅ΠΌ :
Secret 7 (Bolivar can't bring two or the problem of skew)
Recently we caught a non-obvious error in production when left joining 2 tables of completely normal sizes for our data warehouse.
At the same time, a similar query with inner join works without problems:
-- bad query
select a.*, b.address
from small_tbl a -- 7 ΠΌΠ»Π½ ΡΡΡΠΎΠΊ ( 7 mln rows )
LEFT JOIN big_tbl b -- 1 ΠΌΠ»ΡΠ΄ ΡΡΡΠΎΠΊ ( 1 bln rows )
on a.addr_fk = b.addr_pk
ERROR: Canceling query because of high VMEM usage.
-- friendly query
select a.*, b.address
from small_tbl a JOIN big_tbl b
on a.addr_fk = b.addr_pk
ΠΡΠΊΡΡΡΠΈΠ΅ ΠΏΠΎΠΊΠ°Π·Π°Π»ΠΎ, ΡΡΠΎ Π² ΡΠ°Π±Π»ΠΈΡΠ΅ small_tbl ΠΏΠΎΠ»ΠΎΠ²ΠΈΠ½Π° ΠΊΠ»ΡΡΠ΅ΠΉ ΠΎΠΊΠ°Π·Π°Π»ΠΈΡΡ NULL + addr_fk ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΊΠ»ΡΡΠΎΠΌ ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ Π² ΡΠ°Π±Π»ΠΈΡΠ΅.
ΠΡΠΎΠ±Π»Π΅ΠΌΡ ΡΠ΅ΡΠΈΠ»ΠΈ ΠΌΠ°Π»ΠΎΠΉ ΠΊΡΠΎΠ²ΡΡ, Π·Π°ΠΌΠ΅Π½ΠΈΠ² left join ΠΎΠ±ΡΠ΅Π΄ΠΈΠ½Π΅Π½ΠΈΠ΅ΠΌ Π΄Π²ΡΡ ΠΌΠ½ΠΎΠΆΠ΅ΡΡΠ²:
The analysis showed that in the small_tbl table half of the keys were NULL + addr_fk is the distribution key in the table. The problem was solved with little effort by replacing left join with a union of two sets:
select a.*, b.address
from small_tbl a
left join big_tbl b
on a.addr_fk = b.addr_pk
where a.addr_fk is not null
union all
select a.*, null
from small_tbl a
where a.addr_fk is null
Π‘Π΅ΠΊΡΠ΅Ρ 8(Π‘ΠΊΠΎΠ»ΡΠΊΠΎ Π²Π΅ΡΠΈΡ ΡΠ°Π±Π»ΠΈΡΠ° .. ΠΈΠ»ΠΈ ΠΎ Π΄ΠΎΠ±ΡΠΎΠΌ cross-join Π·Π°ΠΌΠΎΠ»Π²ΠΈΡΠ΅ ΡΠ»ΠΎΠ²ΠΎ!)
ΠΠ°ΠΊ-ΡΠΎ Π½Π°Ρ DBA ΠΏΠΎΠ΄Π³ΠΎΡΠΎΠ²ΠΈΠ» ΠΎΡΡΠ΅Ρ ΠΎ ΡΠ°Π·ΠΌΠ΅ΡΠ΅ ΡΠ°Π±Π»ΠΈΡ ΠΏΠΎ ΡΡΠ΅Π±ΡΠ΅ΠΌΠΎΠΌΡ ΡΠΏΠΈΡΠΊΡ ΠΈ ΠΊΡΠ°ΠΉΠ½Π΅ ΡΠ΄ΠΈΠ²ΠΈΠ»ΡΡ.
ΠΠΊΠ°Π·Π°Π»ΠΎΡΡ, ΡΡΠΎ ΡΡΠ½ΠΊΡΠΈΡ pg_relation_size Π΄Π»Ρ Π²ΡΡΠΈΡΠ»Π΅Π½ΠΈΡ ΡΠ°Π·ΠΌΠ΅ΡΠ° ΡΠ°Π±Π»ΠΈΡΡ Π΄Π°Π΅Ρ ΡΠ°Π·Π½ΡΠΉ ΡΠ΅Π·ΡΠ»ΡΡΠ°Ρ Π² Π·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΠΈ ΠΎΡ ΠΊΠΎΠ½ΡΠ΅ΠΊΡΡΠ° Π΅Π΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΡ.
ΠΡΠΎΠ²Π΅Π΄Π΅ΠΌ ΠΏΡΠΎΡΡΠΎΠΉ ΡΠΊΡΠΏΠ΅ΡΠΈΠΌΠ΅Π½Ρ
Secret 8 (How much does a table weigh... or say a word about a good cross-join!)
Once our DBA prepared a report on the size of tables for the required list and was extremely surprised.
It turned out that the pg_relation_size function for calculating the size of a table gives different results depending on the context of its use.
Let's conduct a simple experiment
ΠΠΏΡΠ΅Π΄Π΅Π»ΠΈΠΌ ΡΠ°Π·ΠΌΠ΅Ρ : Let's determine its size:
ΠΡΠΎ ΠΊΠΎΡΡΠ΅ΠΊΡΠ½ΡΠΉ ΡΠ΅Π·ΡΠ»ΡΡΠ°Ρ.
Π‘ΠΎΠ·Π΄Π°Π΄ΠΈΠΌ ΡΠΏΠΈΡΠΎΠΊ ΠΈΠ· 1 ΡΠ»Π΅ΠΌΠ΅Π½ΡΠ° Ρ Π½Π°Π·Π²Π°Π½ΠΈΠ΅ΠΌ ΡΠ°Π±Π»ΠΈΡΡ Π²ΡΡΠ΅
This is the correct result.
Let's create a list of 1 element with the table name above
ΠΠΏΡΠ΅Π΄Π΅Π»ΠΈΠΌ ΡΠ°Π·ΠΌΠ΅Ρ ΡΠ°Π±Π»ΠΈΡ ΡΠΏΠΈΡΠΊΠ°: Let's determine its size:
ΠΠ°ΠΊ ΠΆΠ΅ ΡΠ°ΠΊ ?
ΠΠ΅Π»ΠΎ Π² ΡΠΎΠΌ, ΡΡΠΎ Π·Π°ΠΏΡΠΎΡ ΠΏΠΎΠΊΠ°Π·ΡΠ²Π°Π΅Ρ ΡΠ°Π·ΠΌΠ΅Ρ Π½Π° ΡΠΎΠΌ ΡΠ΅Π³ΠΌΠ΅Π½ΡΠ΅, Π³Π΄Π΅ Π»Π΅ΠΆΠΈΡ ΡΡΡΠΎΡΠΊΠ° Ρ Π΅Π΅ Π½Π°Π·Π²Π°Π½ΠΈΠ΅ΠΌ Π² public.foo_2.
Π Π΅ΡΠ΅Π½ΠΈΠ΅ΠΌ Π² Π»ΠΎΠ± Π±ΡΠ΄Π΅Ρ join cΠΏΠΈΡΠΊΠ° c pg_class
The point is that the query shows the size of the public.foo part on the segment where the line with its name in public.foo_2 is located.
The straightforward solution would be to join the list with pg_class
ΠΠ΄Π½Π°ΠΊΠΎ ΠΎΠ½ΠΎ Π½Π΅ ΡΠ°ΠΌΠΎΠ΅ ΠΎΠΏΡΠΈΠΌΠ°Π»ΡΠ½ΠΎΠ΅, Ρ.ΠΊ. ΡΠΈΠ»ΡΠ½ΠΎ Π½Π°Π³ΡΡΠΆΠ°Π΅Ρ ΠΊΠ°ΡΠ°Π»ΠΎΠ³.
Π‘ΡΡΠ΅ΡΡΠ²ΡΠ΅Ρ Π±ΠΎΠ»Π΅Π΅ ΡΠ»Π΅Π³Π°Π½ΡΠ½ΡΠΉ Π²Π°ΡΠΈΠ°Π½Ρ, ΠΊΠΎΡΠΎΡΡΠΉ ΡΡΠΎΠΈΡ ΡΠΌΠ΅Π»ΠΎ Π±ΡΠ°ΡΡ Π½Π° Π²ΠΎΠΎΡΡΠΆΠ΅Π½ΠΈΠ΅.
However, it is not the most optimal, as it heavily loads the catalog.
There is a more elegant option that is A MUST!
ΠΠ°ΠΊ-ΡΠΎ Π½Π°Ρ DBA ΠΏΠΎΠ΄Π³ΠΎΡΠΎΠ²ΠΈΠ» ΠΎΡΡΠ΅Ρ ΠΎ ΡΠ°Π·ΠΌΠ΅ΡΠ΅ ΡΠ°Π±Π»ΠΈΡ ΠΏΠΎ ΡΡΠ΅Π±ΡΠ΅ΠΌΠΎΠΌΡ ΡΠΏΠΈΡΠΊΡ ΠΈ ΠΊΡΠ°ΠΉΠ½Π΅ ΡΠ΄ΠΈΠ²ΠΈΠ»ΡΡ.
ΠΠΊΠ°Π·Π°Π»ΠΎΡΡ, ΡΡΠΎ ΡΡΠ½ΠΊΡΠΈΡ pg_relation_size Π΄Π»Ρ Π²ΡΡΠΈΡΠ»Π΅Π½ΠΈΡ ΡΠ°Π·ΠΌΠ΅ΡΠ° ΡΠ°Π±Π»ΠΈΡΡ Π΄Π°Π΅Ρ ΡΠ°Π·Π½ΡΠΉ ΡΠ΅Π·ΡΠ»ΡΡΠ°Ρ Π² Π·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΠΈ ΠΎΡ ΠΊΠΎΠ½ΡΠ΅ΠΊΡΡΠ° Π΅Π΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΡ.
ΠΡΠΎΠ²Π΅Π΄Π΅ΠΌ ΠΏΡΠΎΡΡΠΎΠΉ ΡΠΊΡΠΏΠ΅ΡΠΈΠΌΠ΅Π½Ρ
Secret 8 (How much does a table weigh... or say a word about a good cross-join!)
Once our DBA prepared a report on the size of tables for the required list and was extremely surprised.
It turned out that the pg_relation_size function for calculating the size of a table gives different results depending on the context of its use.
Let's conduct a simple experiment
create table public.foo WITH (appendonly=true,orientation=column,compresstype=zstd,compresslevel=1)
as select generate_series(1,1000000) n;
ΠΠΏΡΠ΅Π΄Π΅Π»ΠΈΠΌ ΡΠ°Π·ΠΌΠ΅Ρ : Let's determine its size:
SQL> select pg_relation_size('public.foo')
pg_relation_size
------------------------------
3 209 024 Π±Π°ΠΉΡΠΡΠΎ ΠΊΠΎΡΡΠ΅ΠΊΡΠ½ΡΠΉ ΡΠ΅Π·ΡΠ»ΡΡΠ°Ρ.
Π‘ΠΎΠ·Π΄Π°Π΄ΠΈΠΌ ΡΠΏΠΈΡΠΎΠΊ ΠΈΠ· 1 ΡΠ»Π΅ΠΌΠ΅Π½ΡΠ° Ρ Π½Π°Π·Π²Π°Π½ΠΈΠ΅ΠΌ ΡΠ°Π±Π»ΠΈΡΡ Π²ΡΡΠ΅
This is the correct result.
Let's create a list of 1 element with the table name above
create table public.foo_2
as select 'public.foo'::text as tbl_nm;
ΠΠΏΡΠ΅Π΄Π΅Π»ΠΈΠΌ ΡΠ°Π·ΠΌΠ΅Ρ ΡΠ°Π±Π»ΠΈΡ ΡΠΏΠΈΡΠΊΠ°: Let's determine its size:
SQL> select pg_relation_size(tbl_nm) from public.foo_2
pg_relation_size
------------------------------
4 640 Π±Π°ΠΉΡ
ΠΠ°ΠΊ ΠΆΠ΅ ΡΠ°ΠΊ ?
ΠΠ΅Π»ΠΎ Π² ΡΠΎΠΌ, ΡΡΠΎ Π·Π°ΠΏΡΠΎΡ ΠΏΠΎΠΊΠ°Π·ΡΠ²Π°Π΅Ρ ΡΠ°Π·ΠΌΠ΅Ρ Π½Π° ΡΠΎΠΌ ΡΠ΅Π³ΠΌΠ΅Π½ΡΠ΅, Π³Π΄Π΅ Π»Π΅ΠΆΠΈΡ ΡΡΡΠΎΡΠΊΠ° Ρ Π΅Π΅ Π½Π°Π·Π²Π°Π½ΠΈΠ΅ΠΌ Π² public.foo_2.
Π Π΅ΡΠ΅Π½ΠΈΠ΅ΠΌ Π² Π»ΠΎΠ± Π±ΡΠ΄Π΅Ρ join cΠΏΠΈΡΠΊΠ° c pg_class
The point is that the query shows the size of the public.foo part on the segment where the line with its name in public.foo_2 is located.
The straightforward solution would be to join the list with pg_class
select pg_relation_size(t.tbl_nm) from
pg_class c, pg_namespace s, public.foo_2 t
where 1=1
and s.oid = c.relnamespace
and s.nspname || '.' || relname = t.tbl_nm
ΠΠ΄Π½Π°ΠΊΠΎ ΠΎΠ½ΠΎ Π½Π΅ ΡΠ°ΠΌΠΎΠ΅ ΠΎΠΏΡΠΈΠΌΠ°Π»ΡΠ½ΠΎΠ΅, Ρ.ΠΊ. ΡΠΈΠ»ΡΠ½ΠΎ Π½Π°Π³ΡΡΠΆΠ°Π΅Ρ ΠΊΠ°ΡΠ°Π»ΠΎΠ³.
Π‘ΡΡΠ΅ΡΡΠ²ΡΠ΅Ρ Π±ΠΎΠ»Π΅Π΅ ΡΠ»Π΅Π³Π°Π½ΡΠ½ΡΠΉ Π²Π°ΡΠΈΠ°Π½Ρ, ΠΊΠΎΡΠΎΡΡΠΉ ΡΡΠΎΠΈΡ ΡΠΌΠ΅Π»ΠΎ Π±ΡΠ°ΡΡ Π½Π° Π²ΠΎΠΎΡΡΠΆΠ΅Π½ΠΈΠ΅.
However, it is not the most optimal, as it heavily loads the catalog.
There is a more elegant option that is A MUST!
SQL> select sum(pg_relation_size(y.tbl_nm || decode(q.content, q.content, '')))
from public.foo_2 y
cross join gp_dist_random('gp_id') q
sum
------------------------------
3 209 024 Π±Π°ΠΉΡ
π2
