Here is part of my flyway.toml. I am providing the table setting.
[flyway]
#We want all scripts to run in a transaction together
group = true
#We do not want the clean command to be possible
cleanDisabled = false
########################################
######## Schema management #############
########################################
#Schema name for Flyway to create/use for its tracking purposes
defaultSchema = "demo"
#Table name for Flyway to create/use for its tracking purposes
#table = "flyway_schema_history"
table = "flywayScriptLog"
environment="Development"
locations = ["filesystem:src/ReleaseScripts"]
mixed = true
outOfOrder = true
validateMigrationNaming = true
baselineOnMigrate = true
###########################################################
############ Environment-specific configuration ###########
###########################################################
[environments.Development]
url = "jdbc:databricks://adb-1641939143605626.6.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/3cf9bee346a8668d;ConnCatalog=prod-engineering-central;EnableArrow=0;"
user = "token"
password = "<token>"
schemas = [ "bronze", "silver", "gold" ]
defaultSchema = "demo"
[environments.Staging]
url = "jdbc:databricks://adb-1641939143605626.6.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/3cf9bee346a8668d;ConnCatalog=prod-engineering-central;EnableArrow=0;"
user = "token"
password = "<token>"
schemas = [ "bronze", "silver", "gold" ]
defaultSchema = "demo"
[environments.Production]
url = "jdbc:databricks://adb-1641939143605626.6.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/3cf9bee346a8668d;ConnCatalog=prod-engineering-central;EnableArrow=0;"
user = "token"
password = "<token>"
schemas = [ "bronze", "silver", "gold" ]
defaultSchema = "demo"
The flyway product is failing since the flyway is submitting the create history table command twice. Here is the error.
Successfully validated 1 migration (execution time 00:00.918s)
All configured schemas are empty; baseline operation skipped. A baseline or migration script with a lower version than the baseline version may execute if available. Check the Schemas parameter if this is not intended.
Creating Schema History table `demo`.`flywayScriptLog` ...
Creating Schema History table `demo`.`flywayScriptLog` ...
SQL State : 42P07
Error Code : 500051
Message : [Databricks][JDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: 42P07, Query: CREATE TAB***, Error message from Server: org.apache.hive.service.cli.HiveSQLException: Error running query: [TABLE_OR_VIEW_ALREADY_EXISTS] org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException: [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view `demo`.`flywayScriptLog` because it already exists.
Choose a different name, drop the existing object, add the IF NOT EXISTS clause to tolerate pre-existing objects, add the OR REPLACE clause to replace the existing materialized view, or add the OR REFRESH clause to refresh the existing streaming table. SQLSTATE: 42P07
Flyway Command:
flyway migrate -configFiles="flyway.toml" -environment=Development -locations=filesystem:./src/ReleaseScripts/PreReleaseScripts -user=token -password=<pwd> -ignoreMigrationPatterns="*:missing,*:future" --enable-native-access=ALL-UNNAMED
Here is part of my flyway.toml. I am providing the table setting.
[flyway]
#We want all scripts to run in a transaction together
group = true
#We do not want the clean command to be possible
cleanDisabled = false
########################################
######## Schema management #############
########################################
#Schema name for Flyway to create/use for its tracking purposes
defaultSchema = "demo"
#Table name for Flyway to create/use for its tracking purposes
#table = "flyway_schema_history"
table = "flywayScriptLog"
environment="Development"
locations = ["filesystem:src/ReleaseScripts"]
mixed = true
outOfOrder = true
validateMigrationNaming = true
baselineOnMigrate = true
###########################################################
############ Environment-specific configuration ###########
###########################################################
[environments.Development]
url = "jdbc:databricks://adb-1641939143605626.6.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/3cf9bee346a8668d;ConnCatalog=prod-engineering-central;EnableArrow=0;"
user = "token"
password = "<token>"
schemas = [ "bronze", "silver", "gold" ]
defaultSchema = "demo"
[environments.Staging]
url = "jdbc:databricks://adb-1641939143605626.6.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/3cf9bee346a8668d;ConnCatalog=prod-engineering-central;EnableArrow=0;"
user = "token"
password = "<token>"
schemas = [ "bronze", "silver", "gold" ]
defaultSchema = "demo"
[environments.Production]
url = "jdbc:databricks://adb-1641939143605626.6.azuredatabricks.net:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/3cf9bee346a8668d;ConnCatalog=prod-engineering-central;EnableArrow=0;"
user = "token"
password = "<token>"
schemas = [ "bronze", "silver", "gold" ]
defaultSchema = "demo"
The flyway product is failing since the flyway is submitting the create history table command twice. Here is the error.
Successfully validated 1 migration (execution time 00:00.918s)
All configured schemas are empty; baseline operation skipped. A baseline or migration script with a lower version than the baseline version may execute if available. Check the Schemas parameter if this is not intended.
Creating Schema History table `demo`.`flywayScriptLog` ...
Creating Schema History table `demo`.`flywayScriptLog` ...
SQL State : 42P07
Error Code : 500051
Message : [Databricks][JDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: 42P07, Query: CREATE TAB***, Error message from Server: org.apache.hive.service.cli.HiveSQLException: Error running query: [TABLE_OR_VIEW_ALREADY_EXISTS] org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException: [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view `demo`.`flywayScriptLog` because it already exists.
Choose a different name, drop the existing object, add the IF NOT EXISTS clause to tolerate pre-existing objects, add the OR REPLACE clause to replace the existing materialized view, or add the OR REFRESH clause to refresh the existing streaming table. SQLSTATE: 42P07
Flyway Command:
flyway migrate -configFiles="flyway.toml" -environment=Development -locations=filesystem:./src/ReleaseScripts/PreReleaseScripts -user=token -password=<pwd> -ignoreMigrationPatterns="*:missing,*:future" --enable-native-access=ALL-UNNAMED