INSERT

Applies to: Databricks SQL Databricks Runtime

Inserts new rows into a table and optionally truncates the table or partitions. You specify the inserted rows by value expressions or the result of a query.

Databricks does not support INSERT for Hive Avro tables if the timestamp-millis type is present in the table schema.

Syntax

[ common_table_expression ]
  INSERT { OVERWRITE | INTO } [ TABLE ] table_name
    [ PARTITION clause ]
    [ ( column_name [, ...] ) | BY NAME ]
    query

[ common_table_expression ]
  INSERT INTO [ TABLE ] table_name
    [ REPLACE WHERE predicate | REPLACE USING ( column_name [, ...] ) ]
    query

[ common_table_expression ]
  INSERT INTO [ TABLE ] table_name [target_alias]
    REPLACE ON boolean_expression
    { (query) [source_alias] | query }

Parameters

• common table expression

  Common table expressions (CTE) are one or more named queries which can be reused multiple times within the main query block to avoid repeated computations or to improve readability of complex, nested queries.

• INTO or OVERWRITE

  If you specify OVERWRITE the following applies:

  • Without a partition_spec the table is truncated before inserting the first row.
  • Otherwise, all partitions matching the partition_spec are truncated before inserting the first row.

  If you specify INTO all rows inserted are additive to the existing rows.

• table_name

  Identifies the table to be inserted to. The name must not include a temporal specification. If the table cannot be found Azure Databricks raises a TABLE_OR_VIEW_NOT_FOUND error.

  table_name must not be a foreign table.

  Applies to: Databricks Runtime 16.0 and above

  table_name may include an options specification.

• PARTITION clause

  An optional parameter that specifies a target partition for the insert. You may also only partially specify the partition.

  When specifying a static partition (column = value), this column must not be repeated in the insert column list.

  When specifying a dynamic partition for INSERT OVERWRITE, the table_name must be a Delta Lake table.

• ( column_name [, …] )

  An optional list of columns in the table. The insert command may specify any particular column from the table at most once.

  Applies to: Databricks SQL Databricks Runtime 12.2 LTS and above

  • If this command omits a column, Databricks SQL assigns the corresponding default value instead.
  • If the target table schema does not define any default value for the inserted column, Databricks SQL assigns NULL if the column is nullable. Otherwise, Databricks SQL raises an error.

  Providing no column list is equivalent to specifying all columns, except for those with assigned values in the PARTITION clause, in the order defined in the table.

• BY NAME

  Applies to: Databricks SQL Databricks Runtime 13.3 LTS and above

  When this clause is used instead of an explicit column list, the command uses the exposed column names of query to produce the column list in the order of query. As with an explicit column list, each column must exist in the target table, and must not be duplicated. If a column present in table_name is not part of the implied column list the DEFAULT value is used instead.

  BY NAME also matches attributes of structs by name.

  No column in query may match a column specified in the PARTITION clause or any column that is generated.

• REPLACE WHERE boolean_expression

  Applies to: Databricks SQL Databricks Runtime 12.2 LTS and above

  If table_name is a Delta Lake table, delete rows matching boolean_expression before inserting any rows matching boolean-expression specified in query. Rows in query which do not match boolean_expression are ignored.

  boolean_expression can only reference attributes from the table.

  See Arbitrary selective overwrite with replaceWhere.

• REPLACE USING ( column_name [, …] )

  Applies to: Databricks SQL Databricks Runtime 16.3 and above

  Atomically deletes rows from table_name that match a row in query on the specified columns, then inserts the rows from query. Matching is based on equality comparison of the specified column values in the table and query.

  For Databricks Runtime 16.3 to 17.1, this feature requires that the specified columns include all of the table’s partition columns.

  Note

  Similar to JOIN USING, REPLACE USING matches rows using regular equality comparison, where NULL values are treated as not equal to anything. Rows with NULL values in the specified columns won't match and therefore aren't removed from the target table.

• query

  A query that produces the rows to be inserted.

  You must match the number of columns returned by the query with the specified or implied insert column list.

  If a data type cannot be safely cast to the matching column data type, a runtime exception is thrown.

  Applies to: Databricks SQL Databricks Runtime 11.3 LTS and above

  • If query consists of a VALUES clause the expression can be DEFAULT.
  • If query consists of a SELECT clause the named_expression can be DEFAULT.
  • DEFAULT will insert the explicitly defined DEFAULT expression of the corresponding column in table_name, or NULL if none is defined.

  If schema evolution is enabled, new columns can exist as the last columns of your schema (or nested columns) for the schema to evolve.

• target_alias

  An optional table alias for the table for REPLACE ON. The alias must not include a column list.

• source_alias

  An optional query alias for the query for REPLACE ON. The alias must not include a column list.

• REPLACE ON boolean_expression

  Applies to: Databricks SQL Databricks Runtime 17.1 and above

  If table_name is a Delta Lake table, REPLACE ON atomically deletes any rows in the table that match the query rows based on the specified boolean_expression, then inserts the query rows. Unlike REPLACE WHERE’s boolean_expression, REPLACE ON’s boolean_expression can reference columns from both the table and the query. If the query is empty, no deletions occur because there are no query rows to match.

Examples

In this section:

• INSERT INTO
• Insert with a column list
• Insert with both a partition spec and a column list
• INSERT using the BY NAME clause
• REPLACE WHERE
• REPLACE USING
• REPLACE ON
• INSERT OVERWRITE

INSERT INTO

INSERT using VALUES

> CREATE TABLE students (name VARCHAR(64), address VARCHAR(64) DEFAULT 'unknown', student_id INT)
  PARTITIONED BY (student_id);

-- Single row insert using a `VALUES` clause specifying all columns.
> INSERT INTO students VALUES
    ('Amy Smith', '123 Park Ave, San Jose', 111111);

-- Single row insert using an implicit default for address
> INSERT INTO students(name, student_id) VALUES('Grayson Miller', 222222);

-- Single row insert using an explicit DEFAULT keyword for address
> INSERT INTO students VALUES('Youna Kim', DEFAULT, 333333);

--  Multi-row insert using a `VALUES` clause
> INSERT INTO students VALUES
    ('Bob Brown', '456 Taylor St, Cupertino', 444444),
    ('Cathy Johnson', '789 Race Ave, Palo Alto', 555555);

-- Multi-row insert using a mix of DEFAULT and literals
> INSERT INTO students VALUES
    ('Gwyneth Zhao', '120 Main St, Rockport', 666666),
    ('Jackson Peterson', DEFAULT, 777777);

> SELECT * FROM students;
            name                  address student_id
 ---------------- ------------------------ ----------
        Amy Smith   123 Park Ave, San Jose     111111
   Grayson Miller                  unknown     222222
        Youna Kim                  unknown     333333
        Bob Brown 456 Taylor St, Cupertino     444444
    Cathy Johnson  789 Race Ave, Palo Alto     555555
     Gwyneth Zhao    120 Main St, Rockport     666666
 Jackson Peterson                  unknown     777777

Insert using a subquery

-- Assuming the persons table has already been created and populated.
> SELECT * FROM persons;
          name                   address       ssn
 ------------- ------------------------- ---------
 Dora Williams 134 Forest Ave, Melo Park 123456789
   Eddie Davis   245 Market St, Milpitas 345678901

> INSERT INTO students PARTITION (student_id = 444444)
    SELECT name, address FROM persons WHERE name = "Dora Williams";

> SELECT * FROM students;
          name                   address student_id
 ------------- ------------------------- ----------
     Amy Smith    123 Park Ave, San Jose     111111
     Bob Brown  456 Taylor St, Cupertino     222222
 Cathy Johnson   789 Race Ave, Palo Alto     333333
 Dora Williams 134 Forest Ave, Melo Park     444444

-- Use an option specification to influence the write behavior
INSERT INTO t WITH ('write.split-size' = 10) SELECT * FROM s;

Insert using a TABLE clause

-- Assuming the visiting_students table has already been created and populated.
> SELECT * FROM visiting_students;
          name               address student_id
 ------------- --------------------- ----------
 Fleur Laurent 345 Copper St, London     777777
 Gordon Martin  779 Lake Ave, Oxford     888888

> INSERT INTO students TABLE visiting_students;

> SELECT * FROM students;
          name                   address student_id
 ------------- ------------------------- ----------
     Amy Smith     123 Park Ave,San Jose     111111
     Bob Brown  456 Taylor St, Cupertino     222222
 Cathy Johnson   789 Race Ave, Palo Alto     333333
 Dora Williams 134 Forest Ave, Melo Park     444444
 Fleur Laurent     345 Copper St, London     777777
 Gordon Martin      779 Lake Ave, Oxford     888888

Insert into a directory

> CREATE TABLE students (name VARCHAR(64), address VARCHAR(64), student_id INT)
    PARTITIONED BY (student_id)
    LOCATION "/path/to/students_table";

> INSERT INTO delta.`/path/to/students_table` VALUES
    ('Amy Smith', '123 Park Ave, San Jose', 111111);
> SELECT * FROM students;
          name                   address student_id
 ------------- ------------------------- ----------
     Amy Smith    123 Park Ave, San Jose     111111

Insert with a column list

> INSERT INTO students (address, name, student_id) VALUES
    ('Hangzhou, China', 'Kent Yao', 11215016);
> SELECT * FROM students WHERE name = 'Kent Yao';
      name                address student_id
 --------- ---------------------- ----------
 Kent Yao         Hangzhou, China   11215016

Insert with both a partition spec and a column list

> INSERT INTO students PARTITION (student_id = 11215017) (address, name) VALUES
    ('Hangzhou, China', 'Kent Yao Jr.');
> SELECT * FROM students WHERE student_id = 11215017;
         name                address student_id
 ------------ ---------------------- ----------
 Kent Yao Jr.        Hangzhou, China   11215017

INSERT using the BY NAME clause

> CREATE TABLE target(n INT, text STRING, s STRUCT<a INT, b INT>);
> INSERT INTO target BY NAME SELECT named_struct('b', 2, 'a', 1) AS s, 0 AS n, 'data' AS text;
> SELECT * FROM target;
  0  data  {"a":1,"b":2}

> CREATE OR REPLACE TABLE target(n INT, arr ARRAY<STRUCT<a INT, b INT>>);
> INSERT INTO target BY NAME SELECT array(named_struct('b', 2, 'a', 1)) AS arr, 0 AS n;
> INSERT INTO target BY NAME SELECT array(named_struct('b', 2, 'a', 1)) AS arr;
> SELECT * FROM target;
  0     [{"a":1,"b":2}]
  NULL  [{"a":1,"b":2}]

> INSERT INTO target BY NAME SELECT array(named_struct('b', 2, 'a', 1)) AS arr, 0 AS badname;
Error

> INSERT INTO target BY NAME SELECT array(named_struct('b', 2, 'a', 1)) AS arr, 0 AS n, 1 AS n;
 Error: INSERT_COLUMN_ARITY_MISMATCH.TOO_MANY_DATA_COLUMNS

REPLACE WHERE

> CREATE TABLE sales(tx_date DATE, amount INTEGER);
> INSERT INTO sales VALUES
   (DATE'2022-10-01', 1234),
   (DATE'2022-10-02', 2345),
   (DATE'2022-10-03', 3456),
   (DATE'2022-11-01', 3214);

-- Replace any rows with a transaction date in October 2022.
> INSERT INTO sales REPLACE WHERE tx_date BETWEEN '2022-10-01' AND '2022-10-31'
   VALUES (DATE'2022-10-01', 1237),
          (DATE'2022-10-02', 2378),
          (DATE'2022-10-04', 2456),
          (DATE'2022-10-05', 6328);
> SELECT * FROM sales ORDER BY  tx_date;
 tx_date    amount
 ---------- ------
 2022-10-01   1237
 2022-10-02   2378
 2022-10-04   2456
 2022-10-05   6328
 2022-11-01   3214

REPLACE USING

> CREATE TABLE students (name STRING, country STRING) PARTITIONED BY (country);
> CREATE TABLE new_students (name STRING, country STRING);

> INSERT INTO students VALUES
    ('Dylan', 'US'),
    ('Doug', 'UK'),
    ('Julia', 'IT'),
    ('David', 'DE'),
    ('Liz', 'CN'),
    ('Adam', NULL);

> INSERT INTO new_students VALUES
    ('Peter', 'FR'),
    ('Jennie', 'UK'),
    ('Eva', NULL);

-- Replace any rows in `students` that have matching `country` values in `new_students`.
> INSERT INTO TABLE students
    REPLACE USING (country)
    SELECT * FROM new_students;

> SELECT * FROM students;
   name       country
  ------      -------
   Dylan           US
  Jennie           UK
   Julia           IT
   David           DE
     Liz           CN
   Peter           FR
    Adam         NULL
     Eva         NULL

REPLACE ON

> CREATE TABLE students (name STRING, row_origin STRING)
> CREATE TABLE people (name STRING, row_origin STRING)

> INSERT INTO students VALUES ('Alice', 'table'), (NULL, 'table'), ('Bob', 'table')
> INSERT INTO people VALUES ('Alice', 'query'), (NULL, 'query'), ('Delta', 'query')

> INSERT INTO TABLE students AS t
  REPLACE ON t.name <=> s.name
  (SELECT * FROM people) AS s

> SELECT * FROM students
  name        row_origin
 -----        ----------
 Alice             query
  NULL             query
   Bob             table
 Delta             query

INSERT OVERWRITE

Insert using a VALUES clause

-- Assuming the students table has already been created and populated.
> SELECT * FROM students;
          name                   address student_id
 ------------- ------------------------- ----------
     Amy Smith    123 Park Ave, San Jose     111111
     Bob Brown  456 Taylor St, Cupertino     222222
 Cathy Johnson   789 Race Ave, Palo Alto     333333
 Dora Williams 134 Forest Ave, Melo Park     444444
 Fleur Laurent     345 Copper St, London     777777
 Gordon Martin      779 Lake Ave, Oxford     888888
   Helen Davis 469 Mission St, San Diego     999999
    Jason Wang     908 Bird St, Saratoga     121212

> INSERT OVERWRITE students VALUES
    ('Ashua Hill', '456 Erica Ct, Cupertino', 111111),
    ('Brian Reed', '723 Kern Ave, Palo Alto', 222222);

> SELECT * FROM students;
       name                 address student_id
 ---------- ----------------------- ----------
 Ashua Hill 456 Erica Ct, Cupertino     111111
 Brian Reed 723 Kern Ave, Palo Alto     222222

Insert using a subquery

-- Assuming the persons table has already been created and populated.
> SELECT * FROM persons;
          name                   address       ssn
 ------------- ------------------------- ---------
 Dora Williams 134 Forest Ave, Melo Park 123456789
   Eddie Davis    245 Market St,Milpitas 345678901

> INSERT OVERWRITE students PARTITION (student_id = 222222)
    SELECT name, address FROM persons WHERE name = "Dora Williams";

> SELECT * FROM students;
          name                   address student_id
 ------------- ------------------------- ----------
    Ashua Hill   456 Erica Ct, Cupertino     111111
 Dora Williams 134 Forest Ave, Melo Park     222222

Insert using a TABLE clause

-- Assuming the visiting_students table has already been created and populated.
> SELECT * FROM visiting_students;
          name               address student_id
 ------------- --------------------- ----------
 Fleur Laurent 345 Copper St, London     777777
 Gordon Martin  779 Lake Ave, Oxford     888888

> INSERT OVERWRITE students TABLE visiting_students;

> SELECT * FROM students;
          name               address student_id
 ------------- --------------------- ----------
 Fleur Laurent 345 Copper St, London     777777
 Gordon Martin  779 Lake Ave, Oxford     888888

Insert overwrite a directory

> CREATE TABLE students (name VARCHAR(64), address VARCHAR(64), student_id INT)
    PARTITIONED BY (student_id)
    LOCATION "/path/to/students_table";

> INSERT OVERWRITE delta.`/path/to/students_table` VALUES
    ('Amy Smith', '123 Park Ave, San Jose', 111111);
> SELECT * FROM students;
          name                   address student_id
 ------------- ------------------------- ----------
     Amy Smith    123 Park Ave, San Jose     111111

Related articles

• common table expression
• COPY
• DELETE
• MERGE
• PARTITION
• query
• UPDATE