bigqueryに関するこの日常の在庫バランス計算を改善することができます

Question

1日の在庫残高を計算するために次のクエリを思いつきました。クエリが機能し、予想される結果が得られますが、約2mio行のトランザクションテーブルのサブセットで実行するには200秒以上かかります。 bigqueryに新しいこと私はこれを行うためのより良い/より効率的な方法があるのだろうかと思いますか？

いくつかのサンプルデータを含むコードは以下のとおりです。 ご迷惑をおかけして申し訳ありません。

#### Generate a continuous date range 
WITH days AS 
    (
    SELECT day 
    FROM UNNEST(
    GENERATE_DATE_ARRAY(DATE('2011-01-01'), CURRENT_DATE(), INTERVAL 1 DAY)) AS day 
), 

#### Transactional information of inventory movements. Simple example 
movements AS 
    (
    SELECT 1 AS ItemID 
     ,1 AS Location 
     ,DATE('2017-12-01') AS TransactionDate 
     ,0 AS Quantity 
     UNION ALL SELECT 1, 1, DATE('2017-12-03'), 10 
     UNION ALL SELECT 1, 1, DATE('2017-12-06'), 100 
     UNION ALL SELECT 1, 1, DATE('2017-12-12'), 1000 
), 

#### Calculate cumulative sum for each item and location based on the transaction date 
cumsum AS 
    (
    SELECT ItemID 
     ,TransactionDate 
     ,Location 
     ,SUM(Quantity) OVER (PARTITION BY ItemID, Location ORDER BY TransactionDate ROWS UNBOUNDED PRECEDING) as cumulative_quantity 
    FROM movements 
), 

#### Cross join with the date range to backfill cumulative values for each day 
#### This will return multiple lines for a day when there are multiple transaction date balances 
cross_sum AS 
    (
    SELECT m.ItemID 
     ,m.Location 
     ,d.day 
     ,m.TransactionDate 
     ,m.cumulative_quantity 
    FROM days d 
    CROSS JOIN cumsum m 
    WHERE m.TransactionDate <= d.day 
), 

#### Get just one line per day, based on the latest transaction date 
filtered AS 
    (
    SELECT ItemID 
     ,Location 
     ,CAST (day AS datetime) AS BalanceDate 
     ,ARRAY_AGG(cumulative_quantity ORDER BY TransactionDate DESC LIMIT 1) AS InventoryBalance 
    FROM cross_sum 
    GROUP BY 1,2,3 
) 

#### Final result, flattened out 
SELECT ItemID 
     ,Location 
     ,BalanceDate 
     ,(SELECT SUM(InventoryBalance) FROM UNNEST(InventoryBalance) AS InventoryBalance) AS InventoryBalance 
FROM filtered 
ORDER BY 1,2,3 

Answer 1

これを行うには良い/より効率的な方法がある場合、私は疑問に思って？ days、cumsumとcross_sumは/修正に最適化され、残りはちょうど解消されています

以下はあなたが見ることができるようBigQueryの標準SQL

ためです。それは、より効率的に良いポテンシャルを持っていますが、実際のデータでテストする必要があります - ので、あなたが

#standardSQL 
#### Transactional information of inventory movements. Simple example 
WITH movements AS (
    SELECT 1 AS ItemID, 1 AS Location, DATE('2017-12-01') AS TransactionDate, 0 AS Quantity UNION ALL 
    SELECT 1, 1, DATE('2017-12-03'), 10 UNION ALL 
    SELECT 1, 1, DATE('2017-12-06'), 100 UNION ALL 
    SELECT 1, 1, DATE('2017-12-12'), 1000 
), days AS (
    SELECT day, ItemID, Location 
    FROM UNNEST(GENERATE_DATE_ARRAY((SELECT MIN(TransactionDate) AS d FROM movements), CURRENT_DATE(), INTERVAL 1 DAY)) AS day 
    CROSS JOIN (SELECT DISTINCT ItemID, Location FROM movements) 
), cumsum AS (
    SELECT ItemID 
     ,TransactionDate 
     ,Location 
     ,LEAD(TransactionDate) OVER(PARTITION BY ItemID, Location ORDER BY TransactionDate) AS NextTransactionDate 
     ,SUM(Quantity) OVER(PARTITION BY ItemID, Location ORDER BY TransactionDate ROWS UNBOUNDED PRECEDING) AS cumulative_quantity 
    FROM movements 
), cross_sum AS (
    SELECT d.ItemID 
     ,d.Location 
     ,d.day AS BalanceDate 
     ,m.cumulative_quantity 
    FROM days d 
    JOIN cumsum m 
    ON d.day >= IFNULL(m.TransactionDate, d.day) 
    AND d.day < IFNULL(m.NextTransactionDate, CURRENT_DATE()) 
) 
SELECT ItemID 
    ,Location 
    ,BalanceDate 
    ,cumulative_quantity 
FROM cross_sum 
ORDER BY 1,2,3 

結果はmovements`テーブル `で

ItemID Location BalanceDate cumulative_quantity 
1   1   2017-12-01  0  
1   1   2017-12-02  0  
1   1   2017-12-03  10  
1   1   2017-12-04  10  
1   1   2017-12-05  10  
1   1   2017-12-06  110 
1   1   2017-12-07  110 
1   1   2017-12-08  110 
1   1   2017-12-09  110 
1   1   2017-12-10  110 
1   1   2017-12-11  110 
1   1   2017-12-12  1110 
1   1   2017-12-13  1110 
1   1   2017-12-14  1110 
1   1   2017-12-15  1110