Remove old GC table name format (#18981)

Signed-off-by: Shlomi Noach <[email protected]>

Author: shlomi-noach

doc/design-docs/SafeLazyDropTables.md

@@ -21,10 +21,10 @@ We wish to manage `DROP` requests. Most managed `DROP` requests will _wait_ befo
 
 This is done by first issuing a `RENAME TABLE my_table TO something_else`. To the app, it seems like the table is gone; but the user may easily restore it by running the revert query: `RENAME TABLE something_else TO my_table`.
 
-That `something_else` name can be e.g. `_vt_HOLD_2201058f_f266_11ea_bab4_0242c0a8b007_20200910113042`.
+That `something_else` name can be e.g. `_vt_hld_2201058f_f266_11ea_bab4_0242c0a8b007_20200910113042_`.
 
 At some point we decide that we can destroy the data. The "hold" period can either be determined by vitess or explicitly by the user. e.g. On a successful schema migration completion, Vitess can choose to purge the "old" table right away.
-At that stage we rename the table to e.g. `_vt_PURGE_63b5db0c_f25c_11ea_bab4_0242c0a8b007_20200911070228`.
+At that stage we rename the table to e.g. `_vt_prg_63b5db0c_f25c_11ea_bab4_0242c0a8b007_20200911070228_`.
 A table by that name is eligible to have its data purged.
 
 By experience (see `gh-ost` issue above), a safe method to purge data is to slowly remove rows, until the table is empty. Note:
@@ -35,9 +35,9 @@ By experience (see `gh-ost` issue above), a safe method to purge data is to slow
 
 It's important to note that the `DELETE` statement actually causes table pages to _load into the buffer pool_, which works against our objective.
 
-Once all rows are purged from a table, we rename it again to e.g. `_vt_DROP_8a797518_f25c_11ea_bab4_0242c0a8b007_20210211234156`. At this time we point out that `20200911234156` is actually a readable timestamp, and stands for `2021-02-11 23:41:56`. That timestamp can tell us when the table was last renamed. 
+Once all rows are purged from a table, we rename it again to e.g. `_vt_drp_8a797518_f25c_11ea_bab4_0242c0a8b007_20210211234156_`. At this time we point out that `20200911234156` is actually a readable timestamp, and stands for `2021-02-11 23:41:56`. That timestamp can tell us when the table was last renamed. 
 
-Vitess can then run an actual `DROP TABLE` for `_vt_DROP_...` tables whose timestamp is older than, say, 2 days. As mentioned above, purging the table actually caused the table to load onto the buffer pool, and we need to wait for it to naturally get evicted, before dropping it.
+Vitess can then run an actual `DROP TABLE` for `_vt_drp_...` tables whose timestamp is older than, say, 2 days. As mentioned above, purging the table actually caused the table to load onto the buffer pool, and we need to wait for it to naturally get evicted, before dropping it.
 
 ## Suggested implementation: table lifecycle aka table garbage collection
 
@@ -69,11 +69,11 @@ The way to support the above is by introducing user-defined states. The general
 Vitess will transition the table through these states, _in order_. But it will also support skipping some states. Let's first explain the meaning of the states:
 
 - Real table (alive): Table is in use in production.
-- `HOLD`: Table is renamed to something like `_vt_HOLD_6ace8bcef73211ea87e9f875a4d24e90_20200915120410`. Vitess will not make changes to the table, will not drop data. The table is put away for safe keeping for X hours/days. If it turns out the app still needs the table, the user can `RENAME` is back to its original name, taking it out of this game.
-- `PURGE`: Table renamed to e.g `_vt_PURGE_6ace8bcef73211ea87e9f875a4d24e90_20200916080539`. Vitess purges (or will purge, based on workload and prior engagements) rows from `PURGE` tables. Data is lost and the user may not resurrect the table anymore.
+- `HOLD`: Table is renamed to something like `_vt_hld_6ace8bcef73211ea87e9f875a4d24e90_20200915120410_`. Vitess will not make changes to the table, will not drop data. The table is put away for safe keeping for X hours/days. If it turns out the app still needs the table, the user can `RENAME` is back to its original name, taking it out of this game.
+- `PURGE`: Table renamed to e.g `_vt_prg_6ace8bcef73211ea87e9f875a4d24e90_20200916080539_`. Vitess purges (or will purge, based on workload and prior engagements) rows from `PURGE` tables. Data is lost and the user may not resurrect the table anymore.
 Most likely we will settle for a `SQL_LOG_BIN=0`, ie purging will not go through replication. The replicas are not so badly affected by `DROP` statements as a `primary`.
-- `EVAC`: Table renamed to e.g. `_vt_EVAC_6ace8bcef73211ea87e9f875a4d24e90_20200918192031`. The table sits still for Y houtrs/days. I'm thinking this period will be pre-defined by vitess. The purpose of this state is to wait a _reasonable_ amount of time so that tabe's pages are evacuated from the innodb buffer pool by the natural succession of production IO/memory activity.
-- `DROP`: Table renamed to e.g. `_vt_DROP_6ace8bcef73211ea87e9f875a4d24e90_20200921193202`. Vitess will `DROP TABLE` this table _imminently_.  
+- `EVAC`: Table renamed to e.g. `_vt_evc_6ace8bcef73211ea87e9f875a4d24e90_20200918192031_`. The table sits still for Y houtrs/days. I'm thinking this period will be pre-defined by vitess. The purpose of this state is to wait a _reasonable_ amount of time so that tabe's pages are evacuated from the innodb buffer pool by the natural succession of production IO/memory activity.
+- `DROP`: Table renamed to e.g. `_vt_drp_6ace8bcef73211ea87e9f875a4d24e90_20200921193202_`. Vitess will `DROP TABLE` this table _imminently_.  
 - gone: end of lifecycle
 
 ## Transitioning and skipping of states

go/test/endtoend/tabletmanager/tablegc/tablegc_test.go

@@ -320,137 +320,118 @@ func TestPopulateTable(t *testing.T) {
 	validateTableDoesNotExist(t, "no_such_table")
 }
 
-func generateRenameStatement(newFormat bool, fromTableName string, state schema.TableGCState, tm time.Time) (statement string, toTableName string, err error) {
-	if newFormat {
-		return schema.GenerateRenameStatement(fromTableName, state, tm)
-	}
-	return schema.GenerateRenameStatementOldFormat(fromTableName, state, tm)
+func generateRenameStatement(fromTableName string, state schema.TableGCState, tm time.Time) (statement string, toTableName string, err error) {
+	return schema.GenerateRenameStatement(fromTableName, state, tm)
 }
 
 func TestHold(t *testing.T) {
-	for _, newNameFormat := range []bool{false, true} {
-		t.Run(fmt.Sprintf("new format=%t", newNameFormat), func(t *testing.T) {
-			populateTable(t)
-			query, tableName, err := generateRenameStatement(newNameFormat, "t1", schema.HoldTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
-			assert.NoError(t, err)
+	populateTable(t)
+	query, tableName, err := generateRenameStatement("t1", schema.HoldTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
+	assert.NoError(t, err)
 
-			_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)
-			assert.NoError(t, err)
+	_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)
+	assert.NoError(t, err)
 
-			validateTableDoesNotExist(t, "t1")
-			validateTableExists(t, tableName)
+	validateTableDoesNotExist(t, "t1")
+	validateTableExists(t, tableName)
 
-			time.Sleep(tableTransitionExpiration / 2)
-			{
-				// Table was created with +10s timestamp, so it should still exist
-				validateTableExists(t, tableName)
+	time.Sleep(tableTransitionExpiration / 2)
+	{
+		// Table was created with +10s timestamp, so it should still exist
+		validateTableExists(t, tableName)
 
-				checkTableRows(t, tableName, 1024)
-			}
+		checkTableRows(t, tableName, 1024)
+	}
 
-			time.Sleep(tableTransitionExpiration)
-			// We're now both beyond table's timestamp as well as a tableGC interval
-			validateTableDoesNotExist(t, tableName)
-			if fastDropTable {
-				validateAnyState(t, -1, schema.DropTableGCState, schema.TableDroppedGCState)
-			} else {
-				validateAnyState(t, -1, schema.PurgeTableGCState, schema.EvacTableGCState, schema.DropTableGCState, schema.TableDroppedGCState)
-			}
-		})
+	time.Sleep(tableTransitionExpiration)
+	// We're now both beyond table's timestamp as well as a tableGC interval
+	validateTableDoesNotExist(t, tableName)
+	if fastDropTable {
+		validateAnyState(t, -1, schema.DropTableGCState, schema.TableDroppedGCState)
+	} else {
+		validateAnyState(t, -1, schema.PurgeTableGCState, schema.EvacTableGCState, schema.DropTableGCState, schema.TableDroppedGCState)
 	}
 }
 
 func TestEvac(t *testing.T) {
-	for _, newNameFormat := range []bool{false, true} {
-		t.Run(fmt.Sprintf("new format=%t", newNameFormat), func(t *testing.T) {
-			var tableName string
-			t.Run("setting up EVAC table", func(t *testing.T) {
-				populateTable(t)
-				var query string
-				var err error
-				query, tableName, err = generateRenameStatement(newNameFormat, "t1", schema.EvacTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
-				assert.NoError(t, err)
-
-				_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)
-				assert.NoError(t, err)
-
-				validateTableDoesNotExist(t, "t1")
-			})
-
-			t.Run("validating before expiration", func(t *testing.T) {
-				time.Sleep(tableTransitionExpiration / 2)
-				// Table was created with +10s timestamp, so it should still exist
-				if fastDropTable {
-					// EVAC state is skipped in mysql 8.0.23 and beyond
-					validateTableDoesNotExist(t, tableName)
-				} else {
-					validateTableExists(t, tableName)
-					checkTableRows(t, tableName, 1024)
-				}
-			})
-
-			t.Run("validating rows evacuated", func(t *testing.T) {
-				// We're now both beyond table's timestamp as well as a tableGC interval
-				validateTableDoesNotExist(t, tableName)
-				// Table should be renamed as _vt_DROP_... and then dropped!
-				validateAnyState(t, 0, schema.DropTableGCState, schema.TableDroppedGCState)
-			})
-		})
-	}
+	var tableName string
+	t.Run("setting up EVAC table", func(t *testing.T) {
+		populateTable(t)
+		var query string
+		var err error
+		query, tableName, err = generateRenameStatement("t1", schema.EvacTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
+		assert.NoError(t, err)
+
+		_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)
+		assert.NoError(t, err)
+
+		validateTableDoesNotExist(t, "t1")
+	})
+
+	t.Run("validating before expiration", func(t *testing.T) {
+		time.Sleep(tableTransitionExpiration / 2)
+		// Table was created with +10s timestamp, so it should still exist
+		if fastDropTable {
+			// EVAC state is skipped in mysql 8.0.23 and beyond
+			validateTableDoesNotExist(t, tableName)
+		} else {
+			validateTableExists(t, tableName)
+			checkTableRows(t, tableName, 1024)
+		}
+	})
+
+	t.Run("validating rows evacuated", func(t *testing.T) {
+		// We're now both beyond table's timestamp as well as a tableGC interval
+		validateTableDoesNotExist(t, tableName)
+		// Table should be renamed as _vt_drp_... and then dropped!
+		validateAnyState(t, 0, schema.DropTableGCState, schema.TableDroppedGCState)
+	})
 }
 
 func TestDrop(t *testing.T) {
-	for _, newNameFormat := range []bool{false, true} {
-		t.Run(fmt.Sprintf("new format=%t", newNameFormat), func(t *testing.T) {
-			populateTable(t)
-			query, tableName, err := generateRenameStatement(newNameFormat, "t1", schema.DropTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
-			assert.NoError(t, err)
+	populateTable(t)
+	query, tableName, err := generateRenameStatement("t1", schema.DropTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
+	assert.NoError(t, err)
 
-			_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)
-			assert.NoError(t, err)
+	_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)
+	assert.NoError(t, err)
 
-			validateTableDoesNotExist(t, "t1")
+	validateTableDoesNotExist(t, "t1")
 
-			time.Sleep(tableTransitionExpiration)
-			time.Sleep(2 * gcCheckInterval)
-			// We're now both beyond table's timestamp as well as a tableGC interval
-			validateTableDoesNotExist(t, tableName)
-		})
-	}
+	time.Sleep(tableTransitionExpiration)
+	time.Sleep(2 * gcCheckInterval)
+	// We're now both beyond table's timestamp as well as a tableGC interval
+	validateTableDoesNotExist(t, tableName)
 }
 
 func TestPurge(t *testing.T) {
-	for _, newNameFormat := range []bool{false, true} {
-		t.Run(fmt.Sprintf("new format=%t", newNameFormat), func(t *testing.T) {
-			populateTable(t)
-			query, tableName, err := generateRenameStatement(newNameFormat, "t1", schema.PurgeTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
-			require.NoError(t, err)
-
-			_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)
-			require.NoError(t, err)
-
-			validateTableDoesNotExist(t, "t1")
-			if !fastDropTable {
-				validateTableExists(t, tableName)
-				checkTableRows(t, tableName, 1024)
-			}
-			if !fastDropTable {
-				time.Sleep(5 * gcPurgeCheckInterval) // wait for table to be purged
-			}
-			validateTableDoesNotExist(t, tableName) // whether purged or not, table should at some point transition to next state
-			if fastDropTable {
-				// if MySQL supports fast DROP TABLE, TableGC completely skips the PURGE state. Rows are not purged.
-				validateAnyState(t, 1024, schema.DropTableGCState, schema.TableDroppedGCState)
-			} else {
-				validateAnyState(t, 0, schema.EvacTableGCState, schema.DropTableGCState, schema.TableDroppedGCState)
-			}
-		})
+	populateTable(t)
+	query, tableName, err := generateRenameStatement("t1", schema.PurgeTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
+	require.NoError(t, err)
+
+	_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)
+	require.NoError(t, err)
+
+	validateTableDoesNotExist(t, "t1")
+	if !fastDropTable {
+		validateTableExists(t, tableName)
+		checkTableRows(t, tableName, 1024)
+	}
+	if !fastDropTable {
+		time.Sleep(5 * gcPurgeCheckInterval) // wait for table to be purged
+	}
+	validateTableDoesNotExist(t, tableName) // whether purged or not, table should at some point transition to next state
+	if fastDropTable {
+		// if MySQL supports fast DROP TABLE, TableGC completely skips the PURGE state. Rows are not purged.
+		validateAnyState(t, 1024, schema.DropTableGCState, schema.TableDroppedGCState)
+	} else {
+		validateAnyState(t, 0, schema.EvacTableGCState, schema.DropTableGCState, schema.TableDroppedGCState)
 	}
 }
 
 func TestPurgeView(t *testing.T) {
 	populateTable(t)
-	query, tableName, err := generateRenameStatement(true, "v1", schema.PurgeTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
+	query, tableName, err := generateRenameStatement("v1", schema.PurgeTableGCState, time.Now().UTC().Add(tableTransitionExpiration))
 	require.NoError(t, err)
 
 	_, err = primaryTablet.VttabletProcess.QueryTablet(query, keyspaceName, true)

go/test/endtoend/vreplication/config_test.go

@@ -36,7 +36,7 @@ import (
 //  2. Column and table names with special characters in them, namely a dash
 //  3. Identifiers using reserved words, as lead is a reserved word in MySQL 8.0+ (https://dev.mysql.com/doc/refman/8.0/en/keywords.html)
 //
-// The internal table _vt_PURGE_4f9194b43b2011eb8a0104ed332e05c2_20221210194431 should be ignored by vreplication
+// The internal table _vt_prg_4f9194b43b2011eb8a0104ed332e05c2_20221210194431_ should be ignored by vreplication
 // The db_order_test table is used to ensure vreplication and vdiff work well with complex non-integer PKs, even across DB versions.
 // The db_order_test table needs to use a collation that exists in all versions for cross version tests as we use the collation for the PK
 // based merge sort in VDiff. The table is using a non-default collation for any version with utf8mb4 as 5.7 does NOT show the default
@@ -66,7 +66,7 @@ create table customer2(cid int, name varchar(128), typ enum('individual','soho',
 create table customer_seq2(id int, next_id bigint, cache bigint, primary key(id)) comment 'vitess_sequence';
 create table `+"`Lead`(`Lead-id`"+` binary(16), name varbinary(16), date1 datetime not null default '0000-00-00 00:00:00', date2 datetime not null default '2021-00-01 00:00:00', primary key (`+"`Lead-id`"+`), key (date1));
 create table `+"`Lead-1`(`Lead`"+` binary(16), name varbinary(16), date1 datetime not null default '0000-00-00 00:00:00', date2 datetime not null default '2021-00-01 00:00:00', primary key (`+"`Lead`"+`), key (date2));
-create table _vt_PURGE_4f9194b43b2011eb8a0104ed332e05c2_20221210194431(id int, val varbinary(128), primary key(id), key(val));
+create table _vt_prg_4f9194b43b2011eb8a0104ed332e05c2_20221210194431_(id int, val varbinary(128), primary key(id), key(val));
 create table db_order_test (c_uuid varchar(64) not null default '', created_at datetime not null, dstuff varchar(128), dtstuff text, dbstuff blob, cstuff char(32), primary key (c_uuid,created_at), key (dstuff)) CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
 create table vdiff_order (order_id varchar(50) collate utf8mb4_unicode_ci not null, primary key (order_id), key (order_id)) charset=utf8mb4 COLLATE=utf8mb4_unicode_ci;
 create table datze (id int, dt1 datetime not null default current_timestamp, dt2 datetime not null, ts1 timestamp default current_timestamp, primary key (id), key (dt1));
@@ -83,10 +83,10 @@ create table ukTable (id1 int not null, id2 int not null, name varchar(20), uniq
 	internalSchema = `
  create table _1e275eef_3b20_11eb_a38f_04ed332e05c2_20201210204529_gho(id int, val varbinary(128), primary key(id));
  create table _0e8a27c8_1d73_11ec_a579_0aa0c75a6a1d_20210924200735_vrepl(id int, val varbinary(128), primary key(id));
- create table _vt_PURGE_1f9194b43b2011eb8a0104ed332e05c2_20201210194431(id int, val varbinary(128), primary key(id));
- create table _vt_EVAC_6ace8bcef73211ea87e9f875a4d24e90_29990915120410(id int, val varbinary(128), primary key(id));
- create table _vt_DROP_2bce8bcef73211ea87e9f875a4d24e90_20200915120410(id int, val varbinary(128), primary key(id));
- create table _vt_HOLD_4abe6bcef73211ea87e9f875a4d24e90_20220115120410(id int, val varbinary(128), primary key(id));
+ create table _vt_prg_6ace8bcef73211ea87e9f875a4d24e90_20200915120410_(id int, val varbinary(128), primary key(id));
+ create table _vt_evc_6ace8bcef73211ea87e9f875a4d24e90_20200915120410_(id int, val varbinary(128), primary key(id));
+ create table _vt_drp_6ace8bcef73211ea87e9f875a4d24e90_20200915120410_(id int, val varbinary(128), primary key(id));
+ create table _vt_hld_6ace8bcef73211ea87e9f875a4d24e90_20200915120410_(id int, val varbinary(128), primary key(id));
  `
 
 	initialProductVSchema = `

go/test/endtoend/vreplication/resharding_workflows_v2_test.go

@@ -596,7 +596,7 @@ func testMoveTablesV2Workflow(t *testing.T) {
 
 	// The purge table should get skipped/ignored
 	// If it's not then we'll get an error as the table doesn't exist in the vschema
-	createMoveTablesWorkflow(t, "customer,loadtest,vdiff_order,reftable,_vt_PURGE_4f9194b43b2011eb8a0104ed332e05c2_20221210194431")
+	createMoveTablesWorkflow(t, "customer,loadtest,vdiff_order,reftable,_vt_prg_4f9194b43b2011eb8a0104ed332e05c2_20221210194431_")
 	waitForWorkflowState(t, vc, defaultKsWorkflow, binlogdatapb.VReplicationWorkflowState_Running.String())
 	validateReadsRouteToSource(t, "replica,rdonly")
 	validateWritesRouteToSource(t)