VMware SQL with MySQL
for Tanzu Application
Service 2.9
VMware SQL with MySQL for Tanzu Application Service 2.9
You can find the most up-to-date technical documentation on the VMware by Broadcom website at:
https://docs.vmware.com/
VMware by Broadcom
3401 Hillview Ave.
Palo Alto, CA 94304
www.vmware.com
Copyright © 2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its
subsidiaries. For more information, go to https://www.broadcom.com. All trademarks, trade names, service
marks, and logos referenced herein belong to their respective companies. Copyright and trademark
information.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
2
Contents
VMware Tanzu SQL with MySQL for VMs 18
About Tanzu SQL for VMs 18
VMware Tanzu SQL with MySQL for VMs 19
About Tanzu SQL for VMs 19
Product Snapshot 19
On-Demand Service 19
On-Demand Service Plan 20
Enterprise-Ready Checklist 20
Feedback 21
Release Notes 22
v2.9.4 22
Features 22
Resolved Issues 23
Known Issues 23
Compatibility 23
v2.9.3 24
Features 24
Known Issues 24
Compatibility 24
v2.9.2 25
Features 25
Known Issues 26
Compatibility 26
v2.9.1 26
Features 27
Resolved Issues 27
Known Issues 28
Compatibility 28
v2.9.0 28
Breaking Changes 28
Features 29
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
3
Known Issues 30
Compatibility 30
View Release Notes for Another Version 31
Tanzu SQL for VMs - Architecture and Planning Guide 32
On-Demand Networking 32
Service Network Requirement 32
Default Network and Service Network 33
Required Networking Rules for On-Demand Services 34
Required Networking Rules for Tanzu SQL for VMs 35
Required Networking Rules for Leader-Follower Plans 35
Required Networking Rules for Highly Available (HA) Cluster Plans 36
Required Networking Rules for Multi‑Site Replication 36
Availability Options 37
VMware Tanzu SQL with MySQL for VMs Topologies 37
Pros and Cons 37
RPOs and RTOs 38
RPOs 38
RTOs 38
Risk 39
About Leader-Follower 39
Overview 39
About Failover 40
About Synchronous Replication 40
About Leader-Follower Errands 40
Infrastructure Requirements for Leader-Follower Deployments 41
Capacity Planning 41
Availability Zones 41
Networking Rules 41
About Highly Available Clusters 41
Highly Available Cluster Topology 41
Infrastructure Requirements for Highly Available Deployments 43
Capacity Planning 43
Availability Zones 43
Networking Rules 43
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
4
Highly Available Cluster Limitations 43
Storage Engine Limitations 44
Architecture Limitations 44
Networking Limitations 44
MySQL Server Defaults for HA Components 45
State Snapshot Transfer Process 45
Large Data File Splitting Enabled 45
Maximum Transaction Sizes 45
MySQL Proxy 45
About Multi-Site Replication 46
Overview 46
Multi‑Site Replication Benefits 47
About Active-Passive Topology 47
About App-Layer Active-Active Topology 48
About Failover and Switchover 49
About Failover 50
About Active-Passive Switchover 51
About App-Layer Active-Active Switchover 51
Infrastructure Requirements for Multi‑Site Replication 52
Capacity Planning 52
Networking Requirements 52
Multi‑Site Replication Limitations 53
VMware Tanzu SQL with MySQL for VMs Recommended Usage and
Limitations
53
Recommended Use Cases 53
About On-Demand Plans 53
Resource Usage Planning for On-Demand Plans 53
Availability Using Multiple AZs 53
Downtime During Redeploys 54
Persistent Disk Usage 54
Single Node and Leader-Follower 54
Multi-Site Replication 55
HA Cluster Jumpbox 55
HA Cluster Node 56
VMware Tanzu SQL with MySQL for VMs - Operator Guide 57
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
5
VMware Tanzu SQL with MySQL for VMs - Operator Getting Started 57
Installing and Configuring VMware Tanzu SQL with MySQL for VMs 58
Role-Based Access in Ops Manager 58
Prerequisites 58
Create an App Security Group for Tanzu SQL for VMs 58
Enable the BOSH Resurrector 59
Download and Import the Tile 60
Configure the Tile 61
Configure AZs and Networks 61
Configure Service Plans 62
About Creating Plans for Restoring Multi-Node Service Instances 62
Procedure for Configuring Service Plans 62
(Optional) Deactivate Service Plan 65
Configure Global Settings 65
Configure MySQL 66
Configure Backups 68
Configure Security 69
Configure TLS 69
Configure Secure Service Instance Credentials 69
Configure Monitoring 71
Configure System Logging 72
Configure Service Instance Upgrades 75
Review Errands (Optional) 76
Verify Stemcell Version and Apply All Changes 79
Preparing for Multi-Site Replication 79
Prerequisites 79
Enable Multi‑Site Replication 80
Preparing for TLS 80
Overview 81
TLS Required or Optional? 82
Generated or Provided CA Certificate 82
Workflow 82
If Using the Generated CA Certificate 82
If Providing Your Own CA Certificate 82
Find the CredHub Credentials in Ops Manager 83
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
6
Set a Custom CA Certificate 84
Add the CA Certificate 85
Enable TLS in Tanzu SQL for VMs 85
Setting Limits for On-Demand Service Instances 86
Create Global-level Quotas 87
Create Plan-level Quotas 87
Create and Set Org-level Quotas 87
Create and Set Space-level Quotas 88
View Current Org and Space-level Quotas 89
Monitor Quota Use and Service Instance Count 89
Calculate Resource Costs for On-Demand Plans 89
Calculate Maximum Resource Cost Per On-Demand Plan 90
Calculate Maximum Resource Cost for All On-Demand Plans 90
Calculate Actual Resource Cost of all On-Demand Plans 91
Create Global-level Quotas 91
Create Plan-level Quotas 91
Create and Set Org-level Quotas 92
Create and Set Space-level Quotas 92
View Current Org and Space-level Quotas 93
Monitor Quota Use and Service Instance Count 93
Calculate Resource Costs for On-Demand Plans 94
Calculate Maximum Resource Cost Per On-Demand Plan 95
Calculate Maximum Resource Cost for All On-Demand Plans 95
Calculate Actual Resource Cost of all On-Demand Plans 95
Controlling Access to Service Plans by Org 96
Control Access to Service Plan by Orgs 96
VMware Tanzu SQL with MySQL for VMs - Managing Tanzu MySQL for
VMs
97
Upgrading VMware Tanzu SQL with MySQL for VMs 97
Upgrade Tanzu SQL for VMs 97
About Individual Service Instance Upgrades 98
Service Interruptions 99
Stemcell or Service Update 99
Plan Change 99
Service Broker Deployments 99
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
7
Configuring Automated Backups 100
About Configuring Automated Backups 100
Back Up Using SCP 100
Create a Public and Private Key‑Pair 101
Configure Backups in Ops Manager 101
Back Up to Amazon S3 or Ceph 103
Create a Custom Policy and Access Key 103
Configure Backups in Ops Manager 104
Back Up to GCS 106
Create a Service Account and Private Key 106
Configure Backups in Ops Manager 107
Back Up to Azure Storage 109
Create a Storage Account and Access Key 109
Configure Backups in Ops Manager 109
Manually Restoring from Backup 111
Overview 111
Identify and Download the Backup Artifact 112
If Restoring a Backup from a Deleted Service Instance 112
If Restoring to a Different Foundation 113
Retrieve Backup Encryption Key 113
Create and Prepare a New Service Instance for Restore 115
Restore the Service Instance 116
Restage the Service Instance 117
Accessing a Database as an Admin User 118
Overview 118
Connect to MySQL with BOSH SSH 118
Connect to MySQL with CredHub Credentials 119
Rotating Certificates 120
Rotate Services TLS Certificate Authority 121
Certificates Used by Tanzu SQL for VMs 121
Resolving Service Interruptions 122
Stemcell or Service Update 122
Plan Change 122
VM Process Failure 122
VM Failure 123
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
8
AZ Failure 123
Region Failure 123
Running Errands 124
Overview 124
Run an Errand 124
Available Errands 125
find-deprecated-bindings 125
smoke-tests 125
configure-leader-follower 126
make-leader 126
make-read-only 126
upgrade-all-service-instances 126
register-broker 127
delete-all-service-instances-and-deregister-broker 127
recreate-all-service-instances 127
Troubleshooting VMware Tanzu SQL with MySQL for VMs 128
Troubleshoot Errors 128
Common Services Errors 128
Leader-Follower Service Instance Errors 132
Inoperable App and Database Errors 140
Highly Available Cluster Errors 141
Failed Backups 143
Troubleshoot Components 144
BOSH Problems 144
Large BOSH Queue 144
Configuration 144
Service Instances in Failing State 144
Configuration 144
Service Instances in Failing State 145
Authentication 145
UAA Changes 145
Networking 145
Validate Service Broker Connectivity to Service Instances 146
Networking 146
Validate Service Broker Connectivity to Service Instances 146
Validate App Access to Service Instance 147
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
9
Quotas 147
Plan Quota Issues 147
Global Quota Issues 147
Failing Jobs and Unhealthy Instances 148
AZ or Region Failure 148
Techniques for Troubleshooting 148
Parse a Cloud Foundry (CF) Error Message 148
Access Broker and Instance Logs and VMs 149
Access Broker Logs and VMs 149
Access Service Instance Logs and VMs 150
Run Service Broker Errands to Manage Brokers and Instances 150
Register Broker 151
Deregister Broker 151
Upgrade All Service Instances 151
Delete All Service Instances 151
Detect Orphaned Service Instances 152
View Resource Saturation and Scaling 154
Identify Apps using a Service Instance 154
Monitor Quota Saturation and Service Instance Count 154
Techniques for Troubleshooting Highly Available Clusters 155
Techniques for Troubleshooting 155
Parse a Cloud Foundry (CF) Error Message 155
Access Broker and Instance Logs and VMs 156
Access Broker Logs and VMs 156
Access Service Instance Logs and VMs 156
Run Service Broker Errands to Manage Brokers and Instances 157
Register Broker 157
Deregister Broker 157
Upgrade All Service Instances 158
Delete All Service Instances 158
Detect Orphaned Service Instances 159
Reinstall a Tile 160
View Resource Saturation and Scaling 160
Identify Apps using a Service Instance 160
Monitor Quota Saturation and Service Instance Count 161
Techniques for Troubleshooting Highly Available Clusters 161
Force a Node to Rejoin a Highly Available Cluster Manually 162
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
10
Re-create a Corrupted VM in a Highly Available Cluster 163
Check Replication Status in a Highly Available Cluster 163
Tools for Troubleshooting 166
download-logs 166
Tools for Troubleshooting 167
download-logs 167
mysql-diag 168
Knowledge Base (Community) 168
File a Support Ticket 168
About Data Migration in VMware Tanzu SQL with MySQL for VMs 169
Triggering a Leader-Follower Failover 169
Overview 169
Retrieve Information 170
Promote the Follower 172
Clean Up Former Leader VM (Optional) 173
Configure the New Follower 174
Unbind and Rebind the App 175
HA Tanzu MySQL for VMs Clusters Procedures 176
Bootstrapping 176
When to Bootstrap 177
Run the Bootstrap Errand 178
Determine Type of Cluster Failure 178
Scenario 1: VMs Running, Cluster Disrupted 179
Scenario 2: VMs Terminated or Lost 179
Re-create the Missing VMs 180
Run the Bootstrap Errand 181
Restore the BOSH Configuration 182
Bootstrap Manually 183
Shut Down MySQL 183
Determine which Node to Bootstrap 184
Bootstrap the First Node 185
Restart Remaining Nodes 186
Monit Restart 186
Manual Redeploy 187
Verify that the Nodes Have Joined the Cluster 187
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
11
Running mysql-diag 187
Run mysql-diag Using the BOSH Command Line Interface (CLI) 188
Example Healthy Output 188
Example Unhealthy Output 189
About the Replication Canary 190
Overview 190
Sample Notification Email 190
Determine If the Cluster Is Accessible 191
VMware Tanzu SQL with MySQL for VMs - Developer Guide 193
VMware Tanzu SQL with MySQL for VMs - Developer Guide - Getting
Started
193
Using VMware Tanzu SQL with MySQL for VMs 194
Overview 194
Prerequisites 194
Confirm the Tanzu SQL for VMs Service Availability 195
Check Service Availability in the Marketplace 195
Check that an Instance is Running in the Space 195
Create a Service Instance 196
Bind a Service Instance to your App 197
Use the MySQL Service in your App 197
Use Custom Schemas 198
MySQL Environment Variables 198
Manage a Service Instance 199
Migrate Data to a Different Plan 200
Upgrade an Individual Service Instance 200
Share Service Instances 201
Unbind an App from a Service Instance 202
Delete a Service Instance 202
Purge a Service Instance 203
Using VMware Tanzu SQL with MySQL for VMs for Multi-Site Replication 203
Overview 203
Prerequisites 204
Create a Multi‑Site Replication Leader-Follower Service Instance 204
Create Multi‑Site Replication Service Instances 204
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
12
Configure Multi‑Site Replication 206
Workflow for Configuring Multi‑Site Replication 206
Procedure for Configuring Multi‑Site Replication 207
Bind a Multi‑Site Replication Leader-Follower Service Instance to Your App 211
Using TLS 211
Overview 211
Activate TLS 212
Prerequisites 212
Activate TLS for Java and Spring Apps 212
Rebind your App 213
Activate TLS for Non-Spring Apps 213
Establish a TLS Connection to a Service Instance 215
About Data Migration in VMware Tanzu SQL with MySQL for VMs 216
The migrate Command 216
Use Cases 216
Use Cases Requiring the migrate Command 216
Use Cases Not Requiring the migrate Command 217
Omitted Data 217
Migrating Data in VMware Tanzu SQL with MySQL for VMs 218
Overview 218
Prerequisites 218
Resource Planning 219
Install the mysql-tools CF CLI Plug-in 219
Enable Source Access 220
Source Access across Spaces 220
Source Access Off-Platform 220
Migrate Data 221
Stop and Unbind Apps 221
Migrate Data to Destination Instance 222
Validate Data 223
Rebind and Re-Stage Apps 223
About MySQL Server Defaults 224
Overview 224
Server Defaults for All Plans 225
Server Defaults for Single Node and Leader-Follower Plans 226
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
13
Server Defaults for Highly Available Cluster Plans 227
Server Defaults for Multi‑Site Replication Plans 227
Changing Defaults Using Arbitrary Parameters 228
Overview 228
Set Optional Parameters 228
Workloads 229
Workload Profile Types 229
Lowercase Table Names 230
Character Sets 231
Synchronous Replication for Leader-Follower 231
Synchronous Replication Timeout 232
Backup Schedule 232
Optimize for Short Words 232
WSREP Applier Threads 233
VMware Tanzu SQL with MySQL for VMs - Developer Guide - Managing
MySQL
234
Developer Guide - Managing MySQL for VMs - Connecting To MySQL 234
Customizing Database Credentials 234
Overview 234
Create Read-only Access Credentials 234
Create Custom Username Credentials 236
Using Management Tools for VMware Tanzu SQL with MySQL for VMs 237
Overview 237
MySQLWeb Database Management App 238
cf CLI MySQL Plug-in 238
Desktop Tools 239
Establishing a Connection from Outside a Deployment 240
Overview 240
Connect Using an SSH Tunnel 240
Prerequisite 240
Procedure 240
Connect Using an IP Address 241
Obtain the IP Address Using cf SSH 241
Prerequisites 241
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
14
Procedure 241
Obtain the IP Address Using the DNS Lookup App 242
Obtain the IP Address Using the BOSH CLI 242
Prerequisite 242
Procedure 242
Triggering Multi-Site Replication Failover and Switchover 244
Overview 244
Verify Follower Health 245
Trigger a Failover 246
Promote the Follower 246
Delete or Purge the Former Leader 247
Create a Follower 248
Reconfigure Multi‑Site Replication 248
Trigger a Switchover 249
Promote the Follower 249
Reconfigure Multi‑Site Replication 250
Workflow for Reconfiguring Multi‑Site Replication 251
Procedure for Reconfiguring Multi‑Site Replication 251
Backing Up and Restoring with mysqldump 256
Overview 256
Back Up and Restore a Tanzu SQL for VMs Logical Backup 257
Create a Tanzu SQL for VMs Logical Backup 257
Restore from a Tanzu SQL for VMs Logical Backup 259
Restore from an Off-Platform Logical Backup 259
Monitoring Node Health 260
Monitor Node Health 260
Prerequisite 260
Monitor Node Health Using the Dashboard 261
Monitor Node Health Using the API 262
Node Health Status 263
Healthy 263
Unhealthy 263
Unresponsive 264
Troubleshooting On-Demand Instances 264
Troubleshoot Errors 264
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
15
Common Service Errors 264
If Instances or Database are Inaccessible 264
Failed Backup or Restore with the adbr Plug-in 267
Persistent Disk Usage Is Increasing 269
Techniques for Troubleshooting 269
Understand a Cloud Foundry (CF) Error Message 269
Find Information about Your Service Instance 270
Use the Knowledge Base (Community) 271
File a Support Ticket 271
Backing Up and Restoring VMware Tanzu SQL with MySQL for VMs 272
Overview 272
About Backups 273
About Backup Files and Metadata 273
Prerequisite: adbr Plug in 274
Manually Back Up a Service Instance 274
Restore a Service Instance 275
About Restoring a Service Instance 275
The Topology of the Instance You Restore To 275
When to Use a Different Restore Method 275
The New Database Must Be Empty 276
Procedure to Restore a Service Instance 276
Troubleshooting the adbr Plug-in 278
Monitor Backups 278
Use the adbr Plug-in to List Backups 278
Use Healthwatch to Confirm Automated Backups 279
Monitoring and KPIs for VMware Tanzu SQL with MySQL for VMs 280
About Metrics 280
Access MySQL Metrics 280
Use Grafana 280
Use the Indicator Protocol Dashboard 281
Use Log Cache 282
KPIs for MySQL Service Instances 283
Server Availability 283
Persistent Disk Used 284
Ephemeral Disk Used 284
CPU Utilization Percent 285
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
16
Connections 285
Queries Delta 286
Highly Available Cluster WSREP Ready 286
Highly Available Cluster WSREP Cluster Size 287
Highly Available Cluster WSREP Cluster Status 287
Hours Since Last Successful Backup 288
Component Metrics 288
MySQL Metrics 288
Disk Metrics 294
Leader-Follower Metrics 295
Highly Available Cluster Metrics 296
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
17
VMware Tanzu SQL with MySQL for VMs
VMware Tanzu™ SQL with MySQL for VMs is an on-demand service. Users can provision dedicated
instances of MySQL using the cf CLI or Apps Manager.
About Tanzu SQL for VMs
Tanzu SQL for VMs enables developers to provision and use dedicated instances of MySQL
database on demand. When you install Tanzu SQL for VMs, the tile deploys and maintains a single
service broker that integrates Tanzu SQL for VMs with Ops Manager.
Tanzu SQL for VMs is configured with sensible settings by default so that the service meets user
expectations for a general use relational database.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
18
VMware Tanzu SQL with MySQL for VMs
This topic provides an overview of the on-demand VMware Tanzu SQL with MySQL for VMs
service.
About Tanzu SQL for VMs
Tanzu SQL for VMs enables developers to provision and use dedicated instances MySQL database
on demand. When you install Tanzu SQL for VMs, the tile deploys and maintains a single service
broker that integrates Tanzu SQL for VMs with Ops Manager.
Tanzu SQL for VMs is configured with sensible settings by default so that the service meets user
expectations for a general-use relational database.
Tanzu SQL for VMs supports the following VM topologies:
Single node.
Leader-follower. For more information, see About Leader-Follower.
Highly available (HA) cluster. For highly available clusters Tanzu SQL for VMs uses a patched
Galera Cluster named Percona XtraDB Cluster (PXC). For more information about PXC, see
Percona XtraDB Cluster.
Multi‑Site Replication. For more information, see About Multi‑Site Replication.
Product Snapshot
The following table provides version and version-support information about Tanzu SQL for VMs.
Element Details
Version 2.9.4
Release date May 28, 2021
Compatible Ops Manager versions 2.10, 2.9, 2.8, and 2.7
Compatible VMware Tanzu Application Service for VMs (TAS for VMs)
versions
2.7 and later
IaaS support AWS, Azure, GCP, OpenStack, and
vSphere
IPsec support Yes
On-Demand Service
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
19
Tanzu SQL for VMs is an on-demand service. This means that the service provides dedicated
instances of MySQL that developers can provision on-demand using the cf CLI or Apps Manager.
With Tanzu SQL for VMs you have more options for how and when instances are provisioned.
Tanzu SQL for VMs enables both the operator and developer to configure MySQL settings and
resource use.
The Tanzu SQL for VMs on-demand service uses the on-demand service broker. For information
about the on-demand service broker, see On-Demand Services SDK for VMware Tanzu.
On-Demand Service Plan
Tanzu SQL for VMs offers an on-demand service plan called p.mysql. Operators can configure and
update the plan settings.
When operators update the VM or disk size these settings are applied to all existing instances. If an
operator decreases the disk size, data loss might occur in existing instances.
Enterprise-Ready Checklist
Review the following table to determine if Tanzu SQL for VMs has the features needed to support
your enterprise:
Plans and Instances More Information
On-Demand,
Dedicated-
VM Plans
Tanzu SQL for VMs provides on-demand service plans. On-Demand Networking
Customizable
Plans
Operators can customize the VM, disk size, and availability zone for
service plans.
Configure Service Plans
Custom
Schemas
Tanzu SQL for VMs supports custom schemas. Using custom
schemas enables apps to share a MySQL service instance and
isolate app data by schema.
Use Custom Schemas
Share Service
Instances
Tanzu SQL for VMs supports sharing service instances between
different orgs and spaces.
Share Service Instances
Installation and Upgrades More Information
Product
Upgrades
Tanzu SQL for VMs can be upgraded within the v2 tile series. Upgrading Tanzu SQL for VMs
Deployment
Smoke Tests
Tanzu SQL for VMs installations and upgrades run a post-
deployment BOSH errand that validates basic MySQL operations.
smoke-tests
Maintenance and Backups More Information
Operational
Monitoring
and Logging
Tanzu SQL for VMs provides metrics for monitoring service plan
usage, service quotas, and MySQL component metrics. Tanzu SQL
for VMs can also forward metrics to an external service.
Monitoring Tanzu SQL for VMs
Backup and
Restore
Tanzu SQL for VMs provides backups to an external storage solution
on a configurable schedule. Tanzu SQL for VMs also provides a
restore process.
Configuring Automated
Backups and Backing Up and
Restoring Tanzu SQL for VMs
Scale and Availability More Information
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
20
Scaling Operators can scale the size of VMs up, but not down. Installing Configuring Tanzu
SQL for VMs
Rolling
Deployments
Tanzu SQL for VMs supports rolling deployments when upgrading
HA clusters. However, single node and leader-follower service
instances are unavailable during upgrades.
Upgrading Tanzu SQL for VMs
AZ Support Tanzu SQL for VMs can be deployed to multiple zones to ensure
availability if an unplanned outage of a zone occurs.
About Availability Zones
Encryption More Information
Transport
Layer Security
(TLS)
Tanzu SQL for VMs supports TLS. Enabling TLS provisions a MySQL
server with a certificate so that apps and clients can establish an
encrypted connection with the data service.
Preparing for TLS
Encrypted
Communicati
on in Transit
Tanzu SQL for VMs has been tested successfully with the BOSH
IPsec Add-on.
IPsec Add-on
Feedback
Please send any issues, feature requests, or questions to the Feedback list.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
21
Release Notes
VMware recommends that you upgrade to the latest version of your current minor line, then
upgrade to the latest available version of the next minor line. For example, if you use an older
v2.8.x version, upgrade to the latest v2.8.x version before upgrading to the latest v2.9.x version.
For product versions and upgrade paths, see Upgrade Planner.
Because VMware uses the Percona Distribution for MySQL, expect a time lag between Oracle
releasing a MySQL patch and VMware releasing Tanzu SQL with MySQL for VMs containing that
patch.
v2.9.4
Release Date: May 28, 2021
Features
New features in this release:
List as many backups as needed: Previously, the Application Developer Backup and
Restore (ADBR) plugin always listed the five most recent backups.
Now, the ADBR plugin accepts an integer parameter to specify how many backups to list.
The plugin continues to list five backups by default.
To install the new plugin, see ApplicationDataBackupRestore on the Cloud Foundry
Community cf CLI Plugins website. For how to use the new limit parameter, see Procedure
to Restore a Service Instance in
Backing Up and Restoring
.
Configure backup location with bucket path: The tile now accepts an optional field for
bucket path. This allows backups to be stored with a path like: p.mysql > BUCKET-PATH >
service-instance_GUID > yyyy > mm > dd.
Caution: This release has been removed from General Support.
Warning: Do not use stemcell v621.141 or later with VMware Tanzu SQL with
MySQL for VMs.
Breaking Changes: The breaking changes listed for v2.9.0 also apply if you are
upgrading from v2.8 to v2.9.4. See Breaking Changes below.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
22
Previously, the Application Developer Backup and Restore (ADBR) feature stored backup
artifacts with a path like: p.mysql > service-instance_GUID > yyyy > mm > dd.
Existing backup artifacts can still be accessed through the cf CLI using the adbr plugin.
To configure a bucket path, follow the procedure for your external storage solution in
Configuring Automated Backups.
Resolved Issues
This release has the following fix:
ADBR API only accepts TLS 1.2 connections: Previously, the ADBR API accepted all TLS
connections, regardless of version.
Known Issues
This release has the following issues:
Multi-Site Replication is unsupported for TAS for VMs v2.5 and earlier: Multi-Site
Replication requires Cloud Foundry API (CAPI) v3 as a dependency, which is introduced in
TAS for VMs v2.6.
Plan 1 must be active: If you set Plan 1 to Inactive in the VMware Tanzu SQL with MySQL
for VMs tile, your installation fails when you apply changes. To fix this issue, ensure that
Plan 1 is always configured.
For Ops Manager v2.6 and earlier, automated backups using SCP silently fail: For
workaround instructions, see MySQL SCP backups are failing when prompting for the SSH
key passphrase in PCF in the VMware Tanzu Network knowledge base.
During restore, the ADBR plugin incorrectly indicates the database is unavailable: The
plugin should report that the restore is in progress.
Stemcells v621.141 and later are not compatible with this release. To mitigate this issue,
use a stemcell version between 621.136 and the minimum version indicated in the
Compatibility table below.
Backups larger than 50 GB fail: Backups larger than 50 GB fail to complete. As of v2.10.5,
backups can be as large as 1000 GB.
Compatibility
The following components are compatible with this release:
Component Version
Stemcell 621.125*
Percona Server 5.7.33–36*
Percona XtraDB Cluster 5.7.33–36*
Percona XtraBackup 2.4.21
mysql-backup-release 2.16.0
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
23
mysql-monitoring-release 9.15.0
pxc-release 0.34.0*
*
Components marked with an asterisk have been updated
v2.9.3
Release Date: March 11, 2021
Features
New feature in this release:
Backup Artifacts are organized under Subfolders in External Storage: Previously, the
Application Developer Backup and Restore (ADBR) feature stored backup artifacts in the
root directory in external storage. For easier navigation, backup artifacts are now stored
under p.mysql > service-instance_GUID > yyyy > mm > dd.
Old backup artifacts stored in the root directory can still be accessed through the cf CLI
using the adbr plugin.
Known Issues
This release has the following issues:
Multi-Site Replication is unsupported for TAS for VMs v2.5 and earlier: Multi-Site
Replication requires Cloud Foundry API (CAPI) v3 as a dependency, which is introduced in
TAS for VMs v2.6.
Plan 1 must be active: If you set Plan 1 to Inactive in the VMware Tanzu SQL with MySQL
for VMs tile, your installation fails when you apply changes. To fix this issue, ensure that
Plan 1 is always configured.
For Ops Manager v2.6 and earlier, automated backups using SCP silently fail: For
workaround instructions, see MySQL SCP backups are failing when prompting for the SSH
key passphrase in PCF in the VMware Tanzu Network knowledge base.
During restore, the ADBR plugin incorrectly indicates the database is unavailable: The
plugin should report that the restore is in progress.
Stemcells v621.141 and later are not compatible with this release. To mitigate this issue,
use a stemcell version between 621.136 and the minimum version indicated in the
Compatibility table below.
Backups larger than 50 GB fail: Backups larger than 50 GB fail to complete. As of v2.10.5,
backups can be as large as 1000 GB.
Compatibility
Breaking Changes: The breaking changes listed for v2.9.0 also apply if you are
upgrading from v2.8 to v2.9.3. See Breaking Changes below.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
24
The following components are compatible with this release:
Component Version
Stemcell 621.101*
Percona Server 5.7.32–35*
Percona XtraDB Cluster 5.7.32–31*
Percona XtraBackup 2.4.21*
mysql-backup-release 2.16.0*
mysql-monitoring-release 9.15.0
pxc-release 0.33.0*
*
Components marked with an asterisk have been updated
v2.9.2
Release Date: November 6, 2020
Features
New features and changes in this release:
Multi-Site Replication GA: The Multi-Site Replication feature is officially generally available
in this release. There have been no changes to the feature since v2.9.1.
Removed Global service instance quota limit: Previously, operators could not set the
maximum service instances quota higher than 500 service instances. This upper limit has
been removed.
Percona Server: Upgraded to v5.7.31-34
Breaking Change: If you store automated backups in an S3-compatible bucket that
requires a path-style URL to address the bucket, then select Force path style
access to bucket before upgrading.
This is necessary because the default style for addressing S3-compatible buckets
changed between v2.8.x, v2.9.0, and v2.9.1:
* In v2.9.1 and later, the default bucket address style is virtual hosted.
* In v2.9.0, the default bucket address style is path.
* In v2.8.x, the bucket address style is detected automatically.
For how to select Force path style access to bucket, see Upgrading VMware
Tanzu SQL with MySQL for VMs.
Breaking Changes: The breaking changes listed for v2.9.0 also apply if you are
upgrading from v2.8 to v2.9.2. See Breaking Changes below.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
25
Percona XtraDB Cluster: Upgraded to v5.7.31-34
This Percona XtraDB Cluster version contains a patch for a remote code execution
vulnerability, CVE-2020-15180. For more information, see Percona’s blog post.
Known Issues
This release has the following issues:
Multi-Site Replication is unsupported for TAS for VMs v2.5 and earlier: Multi-Site
Replication requires Cloud Foundry API (CAPI) v3 as a dependency, which is introduced in
TAS for VMs v2.6.
Plan 1 must be active: If you set Plan 1 to Inactive in the VMware Tanzu SQL with MySQL
for VMs tile, your installation fails when you apply changes. To fix this issue, ensure that
Plan 1 is always configured.
For Ops Manager v2.6 and earlier, automated backups using SCP silently fail: For
workaround instructions, see MySQL SCP backups are failing when prompting for the SSH
key passphrase in PCF in the VMware Tanzu Network knowledge base.
During restore, the ADBR plugin incorrectly indicates the database is unavailable: The
plugin should report that the restore is in progress.
Stemcells v621.141 and later are not compatible with this release. To mitigate this issue,
use a stemcell version between 621.136 and the minimum version indicated in the
Compatibility table below.
Backups larger than 50 GB fail: Backups larger than 50 GB fail to complete. As of v2.10.5,
backups can be as large as 1000 GB.
Compatibility
The following components are compatible with this release:
Component Version
Stemcell 621.90*
Percona Server 5.7.31-34*
Percona XtraDB Cluster 5.7.31-34*
Percona XtraBackup 2.4.20
mysql-backup-release 2.15.0
mysql-monitoring-release 9.15.0
pxc-release 0.30.0*
*
Components marked with an asterisk have been updated
v2.9.1
Release Date: October 29, 2020
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
26
Features
New features and changes in this release:
Support for both S3 path-style URLs and virtual hosted-style URLs: Amazon S3 plans to
deprecate S3 path-style URLs in favor of virtual hosted-style URLs. However, some S3-
compatible buckets from other providers still use path-style URLs. If the S3-compatible
bucket that you use for backup storage requires a path-style URL, select Force path style
access to bucket in Ops Manager. See Configure Backups in Ops Manager.
For general information about the deprecation of S3 path-style URLs, see AWS blog posts:
Amazon S3 Path Deprecation Plan – The Rest of the Story and the subsequent Update to
Amazon S3 Path Deprecation Plan.
Support for tuning MySQL full-text search: Developers who want to optimize MySQL’s
full-text search to allow for shorter words can set the optimize_for_short_words arbitrary
parameter to true. This parameter sets the MySQL innodb_ft_min_token_size server
setting to 1. For more information, see Optimize for Short Words in
Changing Defaults
Using Arbitrary Parameters
.
Resolved Issues
This release has the following fixes:
Uppercase letters in the service network no longer prevents configuration of Multi-
Site Replication: Previously, if you had configured the service network name with
uppercase letters in your environment, multi-site replication could not be configured
successfully.
The cf adbr list-backups command no longer fails to list backups for non-AWS S3-
compatible blobstores. For example, the command now supports listing backups in Minio
S3-compatible storage.
Breaking Change: If you store automated backups in an S3-compatible bucket that
requires a path-style URL to address the bucket, then select Force path style
access to bucket before upgrading.
This is necessary because the default style for addressing S3-compatible buckets
changed between v2.8.x, v2.9.0, and v2.9.1:
* In v2.9.1, the default bucket address style is virtual hosted.
* In v2.9.0, the default bucket address style is path.
* In v2.8.x, the bucket address style is detected automatically.
For how to select Force path style access to bucket, see Upgrading VMware
Tanzu SQL with MySQL for VMs.
Breaking Changes: The breaking changes listed for v2.9.0 also apply if you are
upgrading from v2.8 to v2.9.1. See Breaking Changes below.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
27
Known Issues
This release has the following issues:
Multi-Site Replication is unsupported for TAS for VMs v2.5 and earlier: Multi-Site
Replication requires Cloud Foundry API (CAPI) v3 as a dependency, which is introduced in
TAS for VMs v2.6.
Plan 1 must be active: If you set Plan 1 to Inactive in the VMware Tanzu SQL with MySQL
for VMs tile, your installation fails when you apply changes. To fix this issue, ensure that
Plan 1 is always configured.
For Ops Manager v2.6 and earlier, automated backups using SCP silently fail: For
workaround instructions, see MySQL SCP backups are failing when prompting for the SSH
key passphrase in PCF in the VMware Tanzu Network knowledge base.
During restore, the ADBR plugin incorrectly indicates the database is unavailable: The
plugin should report that the restore is in progress.
Stemcells v621.141 and later are not compatible with this release. To mitigate this issue,
use a stemcell version between 621.136 and the minimum version indicated in the
Compatibility table below.
Backups larger than 50 GB fail: Backups larger than 50 GB fail to complete. As of v2.10.5,
backups can be as large as 1000 GB.
Compatibility
The following components are compatible with this release:
Component Version
Stemcell 621.85*
Percona Server 5.7.30–33
Percona XtraDB Cluster 5.7.30–31.43
Percona XtraBackup 2.4.20
mysql-backup-release 2.15.0
mysql-monitoring-release 9.15.0
pxc-release 0.29.0*
*
Components marked with an asterisk have been updated
v2.9.0
Release Date: September 14, 2020
Breaking Changes
Breaking changes in this release:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
28
Additional Networking Port for Backup: Before upgrading from v2.8 to v2.9, ensure that
port 2345 is open from TAS for VMs to ODB.
For the complete list of ports that need to be open for Tanzu SQL for VMs, see Required
Networking Rules for Tanzu SQL for VMs in
On-Demand Networking
.
Location of backup artifact: Backups are no longer stored in a subdirectory in the storage
destination. Instead, they are stored at the root-level.
Removed fields from tile configuration: The following properties have been removed
from the tile configuration. Remove them from any automation that configures the tile.
.properties.backups_selector.s3.enable_email_alerts
.properties.backups_selector.scp.enable_email_alerts
.properties.backups_selector.gcs.enable_email_alerts
.properties.backups_selector.azure.enable_email_alerts
.properties.backups_selector.s3.path
.properties.backups_selector.azure.path
Cron expressions for automated backups are validated: If your cron schedule is invalid in
v2.8, the upgrade to v2.9 fails. For information about cron schedules for automated
backups, see Configuring Automated Backups.
Update the MySQL Connector/J: If you use MySQL Connector/J v8.0.13–8.0.21 and TLS,
then update the MySQL Connector to v8.0.22 or later before upgrading Tanzu SQL for
VMs. The older versions of the MySQL Connector do not support verifying TLS connections
with wildcard domains.
For more information, see Apps Requiring TLS Cannot Connect to the Database after
Upgrade from v2.8 in
Troubleshooting Instances
.
Features
New features and changes in this release:
Improved backup and restore workflow: Operators and developers can use the
ApplicationDataBackupRestore (adbr) plugin with the Cloud Foundry Command Line
Interface (cf CLI) to back up and restore service instances. For more information, see
Backing Up and Restoring VMware Tanzu SQL with MySQL for VMs and Manually Restoring
from Backup.
VMware strongly recommends that if operators offer an HA cluster plan, they also
offer a Multi‑Site Replication plan. To restore an HA cluster service instance from backup,
developers need access to a Multi‑Site Replication plan. For more information, see Restore
a Service Instance in
Backing Up and Restoring VMware Tanzu SQL with MySQL for VMs
and Configure Service Plans.
A developer can change the backup schedule for an individual service instance using
an arbitrary parameter: For more information, see Backup Schedule.
Each backup artifact has a unique encryption key: Each backup of any MySQL service
instance now uses a unique encryption key.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
29
SCP backups use SFTP: When Secure copy protocol (SCP) is configured as the external
storage to upload backups, it now relies on SSH File Transfer Protocol (SFTP).
All service instances are provisioned with a DNS alias: This DNS alias is used as the
hostname in new service keys. Old service keys, which use the DNS address for the
hostname, continue to work.
The legacy workflow to enable TLS on a service instance has been removed: In the
legacy workflow, which was deprecated after MySQL for PCF v2.4, developers had to run
cf update-service to enable TLS on each service instance. This workflow is no longer
needed because MySQL connectors can use the DNS alias to connect to the service
instance over TLS.
Known Issues
This release has the following issues:
The cf adbr list-backups command fails to list backups saved in non-AWS S3-
compatible blobstores. For example, the command shows an error message if the
backups are in Minio S3-compatible storage.
Uppercase letters in the service network name blocks Multi-Site Replication setup:
BOSH DNS can only resolve lowercase letters. Because the service network name is only
used to reference the network configuration in BOSH, change the network name to
lowercase letters.
Multi-Site Replication is unsupported for TAS for VMs v2.5 and earlier: Multi-Site
Replication requires Cloud Foundry API (CAPI) v3 as a dependency, which is introduced in
TAS for VMs v2.6.
Plan 1 must be active: If you set Plan 1 to Inactive in the VMware Tanzu SQL with MySQL
for VMs tile, your installation fails when you apply changes. To fix this issue, ensure that
Plan 1 is always configured.
For Ops Manager v2.6 and earlier, automated backups using SCP silently fail: For
workaround instructions, see MySQL SCP backups are failing when prompting for the SSH
key passphrase in PCF in the VMware Tanzu Network knowledge base.
During restore, the ADBR plugin incorrectly indicates the database is unavailable: The
plugin should report that the restore is in progress.
Stemcells v621.141 and later are not compatible with this release. To mitigate this issue,
use a stemcell version between 621.136 and the minimum version indicated in the
Compatibility table below.
Backups larger than 50 GB fail: Backups larger than 50 GB fail to complete. As of v2.10.5,
backups can be as large as 1000 GB.
Compatibility
The following components are compatible with this release:
Component Version
Stemcell 621.78*
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
30
Percona Server 5.7.30–33
Percona XtraDB Cluster 5.7.30–31.43
Percona XtraBackup 2.4.20
mysql-backup-release 2.15.0
mysql-monitoring-release 9.15.0
pxc-release 0.28.0
*
Components marked with an asterisk have been updated
View Release Notes for Another Version
To view the release notes for another product version, select the version from dropdown at the top
of this page.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
31
Tanzu SQL for VMs - Architecture and
Planning Guide
This topic covers the following areas:
On-Demand Networking
Availability Options
About Leader-Follower
About Highly Available Clusters
About Multi‑Site Replication
Recommended Usage and Limitations
On-Demand Networking
Service Network Requirement
Default Network and Service Network
Required Networking Rules for On-Demand Services
Required Networking Rules for Tanzu SQL for VMs
Required Networking Rules for Leader-Follower Plans
Required Networking Rules for Highly Available (HA) Cluster Plans
Required Networking Rules for Multi‑Site Replication
This topic describes networking for on-demand services, including VMware Tanzu SQL with MySQL
for VMs.
Service Network Requirement
When you deploy VMware Tanzu Application Service for VMs (TAS for VMs), you must create a
statically defined network to host the component VMs that make up the infrastructure.
Components, such as Cloud Controller and UAA, run on this infrastructure network.
On-demand services might require you to host them on a separate network from the default
network. You can also deploy on-demand services on a separate service networks to meet your
own security requirements.
TAS for VMs supports dynamic networking. Operators can use dynamic networking with
asynchronous service provisioning to define dynamically-provisioned service networks. For more
information, see Default Network and Service Network below.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
32
On-demand services are enabled by default on all networks. Operators can optionally create
separate networks to host services in BOSH Director. Operators can select which network hosts
on-demand service instances when they configure the tile for that service.
Default Network and Service Network
On-demand Tanzu SQL for VMs services use BOSH to dynamically deploy VMs and create single-
tenant service instances in a dedicated network. On-demand services use the dynamically-
provisioned service network to host single-tenant worker VMs. These worker VMs run as service
instances within development spaces.
This on-demand architecture has the following advantages:
Developers can provision IaaS resources for their services instances when the instances are
created. This removes the need for operators to pre-provision a fixed amount of IaaS
resources when they deploy the service broker.
Service instances run on a dedicated VM and do not share VMs with unrelated processes.
This removes the "noisy neighbor" problem, where an app monopolizes resources on a
shared cluster.
Single-tenant services can support regulatory compliances where sensitive data must be
separated across different machines.
An on-demand service separates operations between the default network and the service network.
Shared service components, such as executive controllers and databases, Cloud Controller, UAA,
and other on-demand components, run on the default network. Worker pools deployed to specific
spaces run on the service network.
The following diagram shows worker VMs in an on-demand service instance running on a separate
services network, while other components run on the default network.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
33
View a larger version of this image
Required Networking Rules for On-Demand Services
Before deploying a service tile that uses the on-demand service broker (ODB), you must create
networking rules to enable components to communicate with ODB. For instructions for creating
networking rules, see the documentation for your IaaS.
The following table lists key components and their responsibilities in the on-demand architecture.
Key Components Component Responsibilities
BOSH Director Creates and updates service instances as instructed by ODB.
BOSH Agent Adds an agent on every VM that it deploys. The agent listens for instructions from the BOSH
Director and executes those instructions. The agent receives job specifications from the BOSH
Director and uses them to assign a role or job to the VM.
BOSH UAA Issues OAuth2 tokens for clients to use when they act on behalf of BOSH users.
VMware Tanzu
Application
Service for VMs
Contains the apps that consume services.
ODB Instructs BOSH to create and update services. Connects to services to create bindings.
Deployed service
instance
Runs the given service. For example, a deployed Tanzu SQL for VMs service instance runs the
Tanzu SQL for VMs service.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
34
Required Networking Rules for Tanzu SQL for VMs
Regardless of the specific network layout, the operator must set network rules.
To ensure that connections are open, see the following table:
Source
Component
Destination Component Default
TCP Port
Notes
BOSH Agent BOSH Director 4222 The BOSH Agent runs on every VM in the system, including
the BOSH Director VM. The BOSH Agent initiates the
connection with the BOSH Director.
The default port is not configurable.
The communication between these components is two-
way.
Broker and
service
instances
Doppler on VMware Tanzu
Application Service for VMs
(TAS for VMs)
8082 This port is for metrics.
Deployed
apps on TAS
for VMs
MySQL service instances 3306 This port is for general use, app-specific tasks. In addition
to configuring your IaaS, create a security group for the
MySQL service instance.
ODB BOSH Director
BOSH UAA
25555
(BOSH
Director)
8443
(UAA)
8844
(CredHub
)
The default ports are not configurable.
ODB MySQL service instances 8443
3306
This connection is for administrative tasks. Avoid opening
general use, app-specific ports for this connection.
ODB TAS for VMs 8443 The default port is not configurable.
TAS for VMs ODB 8080 This port allows TAS for VMs to communicate with the
ODB component.
TAS for VMs ODB 2345 This port allows TAS for VMs to communicate with the
ODB component so that the
ApplicationDataBackupRestore (adbr) API can take
backups.
Deployed
apps on TAS
for VMs
Runtime CredHub 8844 This port is needed if secure service binding credentials are
enabled. For information, see Configure Security.
TAS for VMs MySQL service instances 8853 This port is for DNS to run health checks against services
instances.
Required Networking Rules for Leader-Follower Plans
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
35
If you are using a leader-follower service plan, the operator must set network rules in addition to
the networking rules required for Tanzu SQL for VMs.
To ensure that connections are open, see the following table:
Source
VM
Destination
VM
Default TCP
Port
Notes
Leader
VM
Follower
VM
8443
8081
This port is needed if leader-follower is enabled. For more information, see
Networking Rules.
The communication between these VMs is two-way.
Required Networking Rules for Highly Available (HA) Cluster Plans
If you are using a HA cluster service plan, the operator must set network rules in addition to the
networking rules required for Tanzu SQL for VMs.
To ensure that connections are open, see the following table:
Source
VM
Destination
VM
Default
TCP
Port
Notes
TAS for
VMs
MySQL
service
instances
8083 This port is needed to monitor cluster health with Switchboard. For more
information, see Monitoring Node Health (HA Cluster).
Jumpbox
VM
TAS for VMs
UAA
8443 This port is needed so that the replication canary can create a UAA client for
sending email notifications. For more information, see About the Replication
Canary.
HA
cluster
node
HA cluster
node
4567
4568
4444
This port is needed to maintain network connectivity between nodes in an HA
cluster. For more information, see Firewall Configuration in the Percona
documentation.
The communication between these VMs is two way.
Galera
healthche
ck
Galera
healthcheck
9200 This port is for monitoring the health of nodes in an HA cluster.
The communication between these VMs is two way.
Required Networking Rules for Multi‑Site Replication
If you are using multi‑site replication, the operator must set network rules for both foundations in
addition to the networking rules required for Tanzu SQL for VMs.
To ensure that connections are open in both foundations, see the following table:
Source
VM
Destination
VM
Default TCP
Port
Notes
Leader
VM
Follower
VM
8081
8443
18443
3306
These ports are needed to enable replication between services instance in
two different foundations.
The communication between these VMs is two way.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
36
Availability Options
This topic describes and contrasts the topologies for VMware Tanzu SQL with MySQL for VMs and
explains the kind of availability each provides.
VMware Tanzu SQL with MySQL for VMs Topologies
The topologies offered by Tanzu SQL for VMs are:
Single node: This one-VM topology is inexpensive and good for testing. You can use this
topology for apps where high availability is not important.
Leader-follower: This two-VM topology gives you redundancy through failover of the
leader VM to the follower VM. For more information, see About Leader-Follower.
Highly available (HA) cluster: This three-VM plus jumpbox cluster uses a patched Galera
cluster, Percona XtraDB Cluster (PXC), to provide the greatest availability of the topologies.
For more information, see About Highly Available Clusters.
Multi‑Site Replication: This two-VM topology enables you to provision a leader and
follower VM in two different foundations. For more information about how to provision a
leader and follower VM, see About Multi‑Site Replication.
Pros and Cons
The following table lists some key characteristics of the topologies to help you decide which one is
best for your developers.
Topology VMs needed Pros Cons
Single node One
Simple
Least expensive
Straightforward operator experience
Easy for the developer
No redundancy
All data since last backup
can be lost
Data recovery requires
restore from backup
Leader-
follower
Two
Two copies of the data
Data recovery is faster than single
node
Less flexible tuning than
single node
Developers require some
technical understanding
Operator must initiate
failover
Highly
available
cluster
Three and a
jumpbox VM
Tightest RPO and RTO for downtime,
including downtime for upgrades.
See RPOs and RTOs.
Most expensive
Less flexible tuning than
single node
Developers require
moderate technical
understanding
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
37
Multi‑Site
Replication
Two
Two copies of the data
Data recovery is faster than single
node
Resilience to data center outages and
upgrades
Developers can initiate failover
Less flexible tuning than
single node
Developers require some
technical understanding
RPOs and RTOs
Recovery point objective (RPO) and recovery time objective (RTO) are important considerations for
availability.
The following tables describe the RPOs and RTOs for the topologies.
RPOs
This table compares the recovery point objectives for the topologies:
Type of
downtime
Single Node Leader-Follower Highly Available
Cluster
Multi‑Site
Replication
Planned
Maintenance
Zero Zero Zero Zero
Unplanned
Downtime
Zero Replica lag, or zero if in sync mode Almost zero* Replica lag
Data Recovery Time since last
backup
The time to initialize the follower VM, or
zero if in sync mode
Almost zero* Replica lag
*Database clients are notified that incomplete transactions are not saved.
RTOs
This table compares the recovery time objectives for the topologies:
Type of
downtim
e
Single Node Leader-Follower Highly
Available
Cluster
Multi‑Site Replication
Planned
Maintena
nce
Time required to
recreate the VM and
reconnect to storage
Time required depends on the type of
maintenance
Almost
zero*
Time for developer to
initiate switchover and for
switchover to complete
Unplann
ed
Downtim
e
Time required to
recreate the VM and
reconnect to storage
Time required to restore the VM or
time for operator to initiate failover
and for failover to complete
Almost
zero*
Time for developer to
initiate failover and for
failover to complete
Data
Recovery
Time to restore from
backup
Time for operator to initiate failover
and for failover to complete
Almost
zero*
Time for developer to
initiate failover and for
failover to complete
*This includes time for apps to reconnect to the MySQL service instance.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
38
Risk
When you choose a topology, risk is a factor to consider. Risk encompasses the likelihood of:
Operators being interrupted to perform disaster recovery
Encountering issues because of the complexity of the topology and technology
Single node topology has the lowest risk. Highly available clusters have the highest risk.
Topology Risk-level
Single node Low
Leader-follower Medium
Highly available cluster High
Multi‑Site Replication Medium-high
About Leader-Follower
This topic describes how the leader-follower topology works in VMware Tanzu SQL with MySQL for
VMs and contains information to help users decide whether to enable or use leader-follower.
Overview
The leader-follower topology increases the availability of a MySQL database by deploying two
MySQL VMs in two separate availability zones (AZs). When a developer creates a leader-follower
service instance, the on-demand broker deploys a leader VM in one AZ, and a follower VM in
another AZ.
A leader-follower topology enables operators to initiate a failover and send app traffic to the
follower if the leader fails. This ensures that the app bound to the MySQL database continues to
function normally.
The follower VM is only for increasing availability and is not exposed to developers to increase read
throughput. Developers who want increased read throughput can configure workload profiles. For
more information, see About Workload Types.
The following diagram shows a leader-follower service instance in two availability zones (AZs):
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
39
View a larger version of this diagram.
In the previous diagram, the leader and follower nodes are each shown in their own AZ, separated
by a dotted line.
An app sends traffic to the leader node. A two way arrow connects the leader to the follower
indicating that any data that is written to the leader is replicated to the follower. The leader and the
follower each have their own disk.
About Failover
Tanzu SQL for VMs focuses on data consistency rather than availability. Therefore, it does not
trigger failover automatically, but relies on operators to trigger a failover. When an operator triggers
a failover, the follower is promoted to the leader and app traffic is sent to the follower.
For more information, see Triggering a Leader-Follower Failover.
About Synchronous Replication
By default, any data that is written to the leader is asynchronously replicated to the follower. Tanzu
SQL for VMs also supports synchronous replication (“sync”).
In sync replication, data does not get committed to the leader node until the follower
acknowledges the commit and can replicate it. By default, the timeout for sync replication is set to
approximately 292 million years. This means that the leader always waits for the follower to confirm
receipt of the transaction.
For information about enabling sync replication, see Synchronous Replication for Leader-Follower.
About Leader-Follower Errands
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
40
Tanzu SQL for VMs automates orchestrating the standard lifecycle of creating, updating, and
deleting leader-follower service instances.
In addition, Tanzu SQL for VMs provides three leader-follower errands. An operator can use them
to control the lifecycle of a leader-follower service instance and manually intervene, when
necessary, without being an expert at MySQL.
For more information, see configure-leader-follower in
Running Errands
.
Infrastructure Requirements for Leader-Follower
Deployments
Leader-follower instances have additional infrastructure requirements to singleton instances, as
described in the following section.
Capacity Planning
When calculating IaaS usage, you must take into account that each leader-follower instance
requires two VMs. Therefore, the resources used for a leader-follower-enabled plan must be
doubled. For more information, see Resource Usage Planning for On-Demand Plans.
Availability Zones
To minimize impact of an availability zone (AZ) outage and to remove single points of failure,
VMware recommends that you provision three AZs if using leader-follower deployments. With
three AZs, the MySQL VMs are deployed to two AZs and the broker is deployed to a third.
For more information, see Availability Using Multiple AZs in
VMware Tanzu SQL with MySQL for
VMs Recommended Usage and Limitations
.
Networking Rules
In addition to the standard networking rules needed for Tanzu SQL for VMs, the operator must
ensure leader-follower-specific network rules are also set up as follows:
Leader-follower VMs bidirectionally communicate with each other over port 8008 for
orchestration.
Leader-follower VMs bidirectionally communicate with each other over port 3306 for
replication.
For information about the standard networking rules, see Required Networking Rules for On-
Demand Services.
About Highly Available Clusters
This topic describes how highly available (HA) clusters work in VMware Tanzu SQL with MySQL for
VMs and contains information to help you decide whether to use HA clusters.
Highly Available Cluster Topology
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
41
In an HA cluster topology, a cluster consists of three nodes. Each node contains the same set of
data synchronized across nodes. Data is simultaneously replicated across nodes or not written at all
if replication fails even on a single node.
With three nodes, HA clusters are highly available and resilient to failure because if a node loses
contact with the other two nodes, the other two nodes can remain in contact and continue to
accept transactions. If the number of nodes is odd, the two nodes can represent the primary
component. For more information about cluster availability, see Percona XtraDB Cluster: Quorum
and Availability of the cluster in the Percona XtraDB Cluster documentation.
In HA topology, the Tanzu SQL for VMs service uses the following:
Three MySQL servers running Percona XtraDB as the MySQL engine and Galera to keep
them in sync.
Three Switchboard proxies that are co-located on the MySQL nodes. The proxies gracefully
handle failure of nodes which enables fast failover to other nodes within the cluster. For
more information, see MySQL Proxy.
A jumpbox VM called mysql-monitor for monitoring, diagnosing and backing up MySQL
nodes. You can run the mysql-diag tool on the mysql-monitor VM to view the status of
your cluster nodes. For more information about mysql-diag, see Running mysql-diag.
The following diagram shows an HA cluster in three availability zones (AZs):
See the following image for a detailed description.
View a larger version of this diagram.
The diagram shows an app communicating with a MySQL cluster using BOSH DNS. Each of the
three MySQL nodes is located in its own AZ. Each MySQL node contains a proxy and a server, and
has its’ own disk. Arrows are shown thAT connect the MySQL nodes, indicating that they can
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
42
communicate with each other across AZs. The dotted lines indicate data replication from the
primary to the secondary databases.
Traffic from apps and clients is sent round robin to the proxies on all nodes over BOSH DNS. The
proxies direct their traffic to the server on the primary node.
If the primary server fails, a secondary node is promoted to become the new primary node. The
proxies update to ensure that traffic is sent to the new primary node.
For more information about other availability options supported by Tanzu SQL for VMs, see
Availability Options.
Infrastructure Requirements for Highly Available
Deployments
Before you decide to use HA clusters for Tanzu SQL for VMs service plans, consider the following
requirements.
Capacity Planning
When calculating IaaS usage, you must take into account that each HA instance requires three
VMs. Therefore, the resources used for an HA plan must be tripled. For more information, see
Resource Usage Planning for On-Demand Plans.
Availability Zones
To minimize impact of an availability zone (AZ) outage and to remove single points of failure,
VMware recommends that you provision three AZs if using HA deployments. With three AZs,
nodes are deployed to separate AZs.
For more information, see Availability Using Multiple AZs in
VMware Tanzu SQL with MySQL for
VMs Recommended Usage and Limitations
.
Networking Rules
In addition to the standard networking rules needed for Tanzu SQL for VMs, the operator must
ensure HA cluster specific network rules are also configured.
For information about HA cluster specific networking rules, see Required Networking Rules for
Highly Available Cluster Plans.
Highly Available Cluster Limitations
When deployed with HA topology, Tanzu SQL for VMs runs three nodes. This cluster arrangement
imposes some limitations which do not apply to single node or leader-follower MySQL database
servers. For more information about the difference between single node, leader-follower, and HA
cluster topologies, see Availability Options.
HA clusters perform validations at startup and during runtime to prevent you from using MySQL
features that are not supported. If a validation fails during startup, the server is halted and throws
an error. If a validation fails during runtime the operation is denied and throws an error.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
43
These validations are designed to ensure optimal operation for common cluster setups that do not
require experimental features and do not rely on operations not supported by HA clusters.
For more information about HA limitations, see Percona XtraDB Cluster Limitations in the Percona
XtraDB Cluster documentation.
Storage Engine Limitations
HA clusters only supports the InnoDB storage engine. The InnoDB is the default storage
engine for new tables. Pre-existing tables that are not using InnoDB are at risk because
they are not replicated within a cluster.
Large DDL statements can lock all schemas. This can be mitigated by using the Rolling
Schema Upgrade (RSU) method. For instructions on how to use the RSU method, see
Using a rolling schema upgrade (RSU) in the VMware Tanzu Support Knowledge Base.
Large transaction sizes can inhibit the performance of the cluster and apps using the cluster
because they are buffered in memory.
You cannot run a DML statement in parallel with a DDL statement, if both statements affect
the same tables. If they are expected in parallel, the DML and DDL statements are both
applied immediately to the cluster, and cause errors.
Architecture Limitations
All tables must have a primary key. You can use multi-column primary keys. This is because
HA clusters replicate using row based replication and ensure unique rows on each instance.
Explicit table locking is not supported. For more information, see EXPLICIT TABLE
LOCKING in the Percona XtraDB Cluster documentation.
By default, auto-increment variables are not sequential and each node has gaps in IDs. This
prevents auto-increment replication conflicts across the cluster. For more information, see
wsrep_auto_increment_control in the Percona XtraDB Cluster documentation.
Users cannot change the session behavior of auto-increment variables. This is because the
HA cluster controls the behavior of these variables. For example, if an app runs SET SESSION
auto_increment_increment, the cluster ignores the change.
In InnoDB, some transactions can cause deadlocks. You can minimize deadlocks in your
apps by rewriting transactions and SQL statements. For more information about deadlocks,
see Deadlocks in InnoDB and How to Minimize and Handle Deadlocks in MySQL
documentation.
Table partitioning can cause performance issues in the cluster. This is as a result of the
implicit table locks that are used when running table partition commands.
Networking Limitations
Round-trip latency between database nodes must be less than five seconds. Latency
exceeding this results in a network partition. If more than half of the nodes are partitioned,
the cluster loses quorum and becomes unusable until manually bootstrapped. For more
information about bootstrapping HA clusters, see Bootstrapping.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
44
MySQL Server Defaults for HA Components
This section describes some defaults that Tanzu SQL for VMs applies to HA components.
State Snapshot Transfer Process
When a new node is added to or rejoins a cluster, a primary node from the cluster is designated as
the state donor. The new node synchronizes with the donor through the State Snapshot Transfer
(SST) process.
Tanzu SQL for VMs uses Xtrabackup for SST, which lets the state donor node to continue
accepting reads and writes during the transfer. HA clusters default to using rsync to perform SST,
which is fast, but blocks requests during the transfer. For more information about Percona
XtraBackup SST Configuration in the Percona XtraDB Cluster documentation.
Large Data File Splitting Enabled
By default, HA clusters splits large Load Data commands into small manageable units. This deviates
from the standard behavior for MySQL. For more information, see wsrep_load_data_splitting in
the Percona XtraDB Cluster documentation.
Maximum Transaction Sizes
These are the maximum transaction sizes set for HA clusters:
The maximum write-set size that is allowed in HA clusters is 2 GB. For more information, see
wsrep_max_ws_size.
The maximum number of rows each write-set can contain defaults to no limit. For more
information, see wsrep_max_ws_rows.
MySQL Proxy
VMware Tanzu SQL with MySQL for VMs uses a proxy to send client connections to the healthy
MySQL database cluster nodes in a highly available cluster plan. Using a proxy handles failure of
nodes, enabling fast, failover to other nodes within the cluster. When a node becomes unhealthy,
the proxy closes all connections to the unhealthy node and re-routes all subsequent connections to
a healthy node.
The proxy used in Tanzu SQL for VMs is Switchboard. Switchboard was developed to replace
HAProxy as the proxy tier for the high availability cluster for MySQL databases in Tanzu SQL for
VMs.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
45
See the following image for a detailed description.
View a larger version of this diagram.
Switchboard offers the following features:
MySQL Server Access
MySQL clients communicate with nodes through this network port. These connections are
automatically passed through to the nodes.
Switchboard and API
Operators can connect to Switchboard to view the state of the nodes.
For more information about monitoring proxy health status, see Monitoring Node Health.
About Multi-Site Replication
This topic describes how the multi‑site replication topology works in VMware Tanzu SQL with
MySQL for VMs and contains information to help users decide whether to use the Multi‑Site
Replication plan.
Overview
The multi‑site replication topology in Tanzu SQL for VMs is a disaster recovery solution that enables
developers to provision a leader-follower instance across two foundations. This leader-follower
service instance is comprised of two single node instances that are configured for replication.
Operators can configure two foundations to be in the same data center or spread across multiple
data centers or geographical regions.
The foundation types are:
Primary Foundation: This foundation is deployed in your main data center. Generally, this
data center receives the majority of app traffic. Tanzu SQL for VMs assumes that the leader
is deployed on this foundation when healthy.
Secondary Foundation: This foundation is deployed in your failover data center. Generally,
this data center receives less app traffic than the primary foundation, or no traffic at all.
Tanzu SQL for VMs assumes that the follower is deployed on this foundation unless a
developer triggers a failover.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
46
For information for operators about enabling Multi‑Site Replication, see Preparing for Multi‑Site
Replication.
For information for developers about using Multi‑Site Replication, see Using Tanzu SQL for VMs for
Multi‑Site Replication.
Multi‑Site Replication Benefits
The Multi‑Site Replication plan has the following benefits:
Resilience for Service Instances: Developers can trigger a failover to maintain app uptime
during a data center outage. For more information, see Triggering Multi‑Site Replication
Failover and Switchover.
Data Center Upgrades with Zero Downtime: Operators can upgrade data centers without
taking databases offline by triggering a switchover first.
Support for Multiple Cloud Deployment Models: Operators can configure multi‑site
replication with a single cloud or hybrid cloud deployment model. Both deployment models
have the same end-user experience.
Support for Active-Passive and App-Layer Active-Active Disaster Recovery: For more
information, see About Active-Passive Topology and About App-Layer Active-Active
Topology below.
About Active-Passive Topology
In an active-passive topology, when all foundations and workloads are healthy, all app traffic is
directed to the primary foundation. The secondary foundation receives no app traffic.
VMware recommends using this topology when your secondary foundation is scaled down,
generally inactive, and does not receive significant app traffic.
For information about active-passive failover and switchover, see About Failover and Switchover
below.
The following diagram describes the active-passive topology in a healthy state:
Note: The Multi‑Site Replication plan type is configured separately from the leader-
follower service plan.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
47
View a larger version of this diagram.
In the above diagram:
The global DNS load balancer (GLB) directs traffic to the app in the primary foundation. This
app issues transactions to the leader service instance.
The follower service instance in the secondary foundation continuously replicates data from
the leader service instance in the primary foundation.
About App-Layer Active-Active Topology
In an app-layer active-active topology, when all foundations and workloads are healthy, app traffic is
directed to both the primary and secondary foundation.
VMware recommends using this topology when both your primary and secondary foundations are
scaled up and are expected to serve traffic.
For information about app-layer active-active failover and switchover, see About Failover and
Switchover below.
The following diagram describes the app-layer active-active topology in a healthy state:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
48
View a larger version of this diagram.
In the above diagram:
The GLB directs traffic to the apps in the primary and secondary foundations. The apps in
the primary and secondary foundation issue transactions to the leader service instance.
The app in the secondary foundation issues transactions to the leader as below:
1. Connects to the follower service instance
2. Issues transactions to the follower service instance
3. Forwards the transactions from the follower service instance to the leader
The follower service instance in the secondary foundation continuously replicates data from
the leader service instance in the primary foundation.
About Failover and Switchover
Tanzu SQL for VMs prioritizes data consistency over availability. Therefore, Tanzu SQL for VMs does
not trigger failover or switchover automatically and developers must manually trigger a failover or
switchover. For instructions about triggering failover or switchover, see see Triggering Multi‑Site
Replication Failover and Switchover.
Note: In the Multi‑Site Replication plan, the app-layer active-active topology does
not enable multi-primary replication. Therefore, the follower service instance is
read-only. Apps can only write to the leader.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
49
The table below describes when you should trigger a failover or switchover:
Failover if... Switchover if...
The leader MySQL process has crashed or is
unhealthy and is not automatically recovered by
BOSH.
The leader VM is destroyed or unrecoverable.
The availability zone (AZ) for the leader VM
experiences an unexpected outage.
The data center for the leader VM experiences an
unexpected outage.
Both the leader and the follower instance are
healthy.
You plan to do foundation or data center
upgrades or maintenance on your primary site.
For example, upgrading stemcells or data center
hardware.
You plan to degrade performance on the primary
site.
For information about multi‑site replication topologies in an healthy state, see About Active-Passive
Topology and About App-Layer Active-Active Topology above.
About Failover
In both the active-passive and app-layer active-active topologies, if a developer triggers a failover,
all app traffic is directed to the leader in the secondary foundation and the primary foundation
receives no app traffic.
The following diagram describes what happens when you trigger a failover for a multi‑site
replication topology:
View a larger version of this diagram.
In the above diagram:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
50
The GLB directs traffic to the app in the secondary foundation. This app issues transactions
to the leader service instance in the secondary foundation.
The leader service instance in the secondary foundation does not replicate data to another
service instance.
About Active-Passive Switchover
In an active-passive topology, if a developer triggers a switchover, all app traffic is directed to the
leader in the secondary foundation. The primary foundation receives no app traffic.
The following diagram describes what happens when you trigger a switchover in an active-passive
topology:
View a larger version of this diagram.
In the above diagram:
The GLB directs traffic to the app in the secondary foundation. This app issues transactions
to the leader service instance in the secondary foundation.
The leader service instance in the secondary foundation replicates data to the follower
service instance in the primary foundation.
About App-Layer Active-Active Switchover
If a developer triggers a switchover, app traffic is still directed to both the primary and secondary
foundation. However, the leader service instance is in secondary foundation.
The following diagram describes what happens when you trigger a switchover in an active-active
topology:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
51
View a larger version of this diagram.
In the above diagram:
The GLB directs traffic to the apps in the primary and secondary foundations. The apps in
the primary and secondary foundation issue transactions to the leader service instance in
the secondary foundation.
The app in the primary foundation issues transactions to the leader as below:
1. Connects to the follower service instance
2. Issues transactions to the follower service instance
3. Forwards the transactions from the follower service instance to the leader
The follower service instance in the primary foundation continuously replicates data from
the leader service instance in the secondary foundation
Infrastructure Requirements for Multi‑Site Replication
Before using the Multi‑Site Replication plan, consider the following requirements.
Capacity Planning
When calculating IaaS usage, you must take into account that each Multi‑Site Replication instance
requires two VMs. Therefore, the resources used for Multi‑Site Replication plan is twice the amount
of single node plan.
For more information, see Setting Limits for On-Demand Service Instances and Persistent Disk
Usage.
Networking Requirements
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
52
Operators should consider that Multi‑Site Replication instances that are deployed in data centers
that are geographically farther apart experience higher latencies.
For information about the standard networking rules, see Required Networking Rules for Multi‑Site
Replication.
Multi‑Site Replication Limitations
The Multi‑Site Replication plan has the following limitation:
Synchronous replication is not supported: For Multi‑Site Replication plans, any data that
is written to the leader is asynchronously replicated to the follower.
VMware Tanzu SQL with MySQL for VMs Recommended
Usage and Limitations
Recommended Use Cases
MySQL is a powerful open-source relational database used by apps since the mid-90s. Developers
have relied on MySQL as a first step to storing, processing and sharing data. As its user community
has grown, MySQL has become a robust system capable of handling a wide variety of use cases and
very significant workloads. Unlike other traditional databases which centralize and consolidate data,
MySQL lends itself to dedicated deployment supporting the “shared nothing” context of building
apps in the cloud.
About On-Demand Plans
Each on-demand plan instance is deployed to its own VM and is suitable for production
workloads.
The maximum-number of on-demand plan instances available to developers is set by the
operator and enforced on both a global and per-plan level quota.
Operators can update the plan settings, including the VM size and disk size after the plans
have been created. Operators cannot downsize the VMs or disk size as this can cause
data loss in pre-existing instances.
All plans deploy a single VM of the specified size with the specified disk type.
Cannot scale down the size of VMs on the plan once deployed (this protects against data
loss).
The default maximum number of connections is set at 750.
Resource Usage Planning for On-Demand Plans
For information on setting limits and calculating costs for on-demand service instances, see Setting
Limits for On-Demand Service Instances.
Availability Using Multiple AZs
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
53
VMware Tanzu SQL with MySQL for VMs supports configuring multiple availability zones (AZs).
However, assigning multiple AZs to MySQL jobs does not guarantee high availability.
You can assign on-demand plans to any of the configured AZs.
BOSH randomly deploys service instances across configured AZs. This minimizes the impact
of an AZ outage and removes single points of failure.
For all Tanzu SQL for VMs plans, select three AZs. Choosing three AZs does not increase
the number of AZs assigned to service instances to three.
Downtime During Redeploys
By default, Tanzu SQL for VMs does a rolling deploy during upgrades or when you apply
configuration changes. For single node and leader-follower plans, this results in Tanzu SQL for VMs
being inaccessible to apps for a brief period of time.
If you are using a HA cluster plan, Tanzu SQL for VMs uses rolling redeploys and your service does
not incur downtime. You can mitigate downtime by using HA cluster plans.
For more information about downtime in Tanzu SQL for VMs, see RPOs and RTOs.
Persistent Disk Usage
The persistent disk receives, temporarily stores, and encrypts MySQL backups. This results in
increased disk usage while backups are being processed. For single node and leader-follower
service plans, backups are processed on the MySQL VM. For highly available (HA) clusters, backups
are processed on the jumpbox VM.
In addition to backups, the persistent disk also stores InnoDB redo logs and binlogs. InnoDB redo
logs use a static amount of disk size. If your app is write-heavy, increase your persistent disk to
accommodate the binlogs.
For the amount of persistent disk required for your service plan, see the below sections:
Single Node and Leader-Follower
Multi-Site Replication
HA Cluster Jumpbox
HA Cluster Node
Single Node and Leader-Follower
You can configure the persistent disk size for single node or leader-follower VMs. For instructions
for configuring disk size, see Configure Service Plans.
Use the following table to determine the amount of persistent disk space you need:
InnoDB Redo Logs Total Disk Size Minimum Total Disk Size
512 MB 3N + 512 MB 5 GB
In the Total Disk Size formula from the previous table, N represents the following items in equal
value:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
54
Maximum app data. Remember to include binlogs in your size estimate for app data.
Data for copied backups.
Data for compression and encryption.
Each N combined makes 3N, as seen in the formula.
The following diagram shows how persistent disk space is used:
Multi-Site Replication
You can configure the persistent disk size for Multi-Site Replication VMs. For instructions for
configuring disk size, see Configure Service Plans.
Use the following table to determine the amount of persistent disk space you need:
InnoDB Redo Logs Total Disk Size Minimum Total Disk Size
2 GB 3N + 2 GB 5 GB
In the Total Disk Size formula from the previous table, N represents the following items in equal
value:
Maximum app data. Remember to include binlogs in your size estimate for app data.
Data for copied backups.
Data for compression and encryption.
Each N combined makes 3N, as seen in the formula.
The following diagram shows how persistent disk space is used:
HA Cluster Jumpbox
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
55
You can configure the persistent disk size for HA cluster jumpbox VMs. For instructions for
configuring disk size, see Configure Service Plans.
Use the following table to determine the amount of persistent disk space you need:
InnoDB Redo Logs Total Disk Size Minimum Total Disk Size
2 GB 2N + 2 GB 10 GB
In the Total Disk Size formula from the previous table, N represents the following items in equal
value:
Data for copied backups.
Data for compression and encryption.
Each N combined makes 2N, as seen in the formula.
The following diagram shows how persistent disk space is used:
HA Cluster Node
You can configure the persistent disk size for HA cluster node VMs. For instructions for configuring
disk size, see Configure Service Plans.
Use the following table to determine the amount of persistent disk space you need:
InnoDB Redo Logs Total Disk Size Minimum Total Disk Size
2 GB N + 2 GB 5 GB
In the Total Disk Size formula from the previous table, N represents maximum app data.
Remember to include binlogs in your size estimate for app data.
The following diagram shows how persistent disk space is used:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
56
VMware Tanzu SQL with MySQL for VMs -
Operator Guide
This topic covers the following areas:
Getting Started
Installing and Configuring
Preparing for Multi‑Site Replication
Preparing for TLS
Setting Limits for On-Demand Instances
Controlling Access to Service Plans by Org
Managing VMware Tanzu SQL with MySQL for VMs
Upgrading
Configuring Automated Backups
Manually Restoring from Backup
Accessing a Database as an Admin User
Rotating Certificates
Resolving Service Interruptions
Running Errands
Troubleshooting
Leader-Follower Procedures
Triggering a Leader-Follower Failover
Highly Available Clusters Procedures
Bootstrapping
Running mysql-diag
About the Replication Canary
VMware Tanzu SQL with MySQL for VMs - Operator Getting
Started
Installing and Configuring
Preparing for Multi‑Site Replication
Preparing for TLS
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
57
Setting Limits for On-Demand Instances
Controlling Access to Service Plans by Org
Installing and Configuring VMware Tanzu SQL with MySQL
for VMs
This topic provides instructions to operators about how to install, configure, and deploy the
VMware Tanzu SQL with MySQL for VMs tile. The Tanzu SQL for VMs service enables developers
to create and use MySQL service instances on demand.
Role-Based Access in Ops Manager
Ops Manager admins can use Role-Based Access Control (RBAC) to manage which operators can
make deployment changes, view credentials, and manage user roles in Ops Manager. Your role
permissions might not permit you to do every procedure in this topic.
For more information about roles in Ops Manager, see Roles in Ops Manager.
Prerequisites
Before you install the Tanzu SQL for VMs tile, you must:
Create an App Security Group for Tanzu SQL for VMs.
Enable the BOSH Resurrector.
Prepare for TLS. Required to enable TLS and required for Multi‑Site Replication and HA
cluster plans.
Prepare for Multi‑Site Replication: Only required if you want to replicate data across
multiple foundations or data centers.
Create an App Security Group for Tanzu SQL for VMs
To enable apps running on VMware Tanzu Application Service for VMs to communicate with the
MySQL service network, you must create an App Security Group (ASG). The ASG enables smoke
tests to run when you first install the Tanzu SQL for VMs service and apps to access the service
after it is installed.
To create an ASG for Tanzu SQL for VMs:
1. Navigate to Ops Manager Installation Dashboard > BOSH Director.
2. Click Create Networks.
3. Click your services network and record the CIDR.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
58
4. Create a JSON file named mysql-asg.json using the following template:
[
{
"protocol": "tcp",
"destination": "CIDR",
"ports": "3306"
}
]
Where CIDR is the CIDR that you recorded in the previous step.
5. Create an ASG by running:
cf create-security-group p.mysql ./mysql-asg.json
6. Bind the ASG to all running apps by running:
cf bind-running-security-group p.mysql
For more information about ASGs, see App Security Groups.
Enable the BOSH Resurrector
VMware recommends activating the BOSH Resurrector when installing Tanzu SQL for VMs. The
BOSH Resurrector increases the availability of Tanzu SQL for VMs by restarting and resuming
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
59
MySQL service.
The BOSH Resurrector does the following:
Reacts to hardware failures and network disruptions by restarting VMs on active, stable
hosts
Detects operating system failures by continuously monitoring VMs and restarts them as
required
Continuously monitors the BOSH Agent running on each service instance VM and restarts
the VM as required
For more information about the BOSH Resurrector, see BOSH Resurrector.
To enable the BOSH Resurrector:
1. Navigate to Ops Manager Installation Dashboard > BOSH Director.
2. Click Director Config.
3. Select the Enable VM Resurrector Plugin check box.
4. Click Save.
Download and Import the Tile
To download and import the Tanzu SQL for VMs tile:
1. Download the product file from VMware Tanzu Network.
2. Navigate to the Ops Manager Installation Dashboard and click Import a Product to
upload the product file.
3. Under Import a Product, click + next to the version number of Tanzu SQL for VMs. This
adds the tile to your staging area.
4. Click the newly-added Tanzu SQL for VMs tile to open its configuration panes.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
60
Configure the Tile
To configure the Tanzu SQL for VMs tile, do the following procedures.
Configure AZs and Networks
To configure an availability zone (AZ) to run the service broker and networks for the broker and
MySQL service instances:
1. Click Assign AZs and Networks.
2. Configure the fields as follows:
Field Instructions
Place
singleton
jobs in
Select the AZ that you want the MySQL broker VM to run in. The broker runs as a singleton job.
Balance
other jobs
in
Ignore; not used.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
61
Network Select a subnet for the MySQL broker. This is typically the same subnet that includes the
VMware Tanzu Application Service for VMs (TAS for VMs) component VMs.
This network is represented by the Default Network in diagram in Default Network and Service
Network.
Service
Network
Select the subnet for the on-demand service instances. This network is represented by the
Service Network in diagram in Default Network and Service Network.
If you are adding IPsec to encrypt MySQL communication, VMware recommends that you
deploy MySQL to its own network to avoid conflicts with services that are not IPsec
compatible.
3. Click Save.
Configure Service Plans
Tanzu SQL for VMs enables you to configure as many as nine service plans. Each service plan has a
corresponding section in the tile configuration, such as Plan 1, Plan 2, and so on.
By default, plans 1 through 3 are active and plans 4 through 9 are inactive. The following
procedures describe how to change these defaults.
About Creating Plans for Restoring Multi-Node Service Instances
If you offer leader-follower or highly available (HA) cluster plans, then you must configure single-
node or Multi‑Site Replication plans that can be used to restore a multi-node plan from backup.
If you offer service plans of type… Then configure a service plan of type…
leader-follower single node, with the persistent disk as large as the largest leader-follower plan
offered.
HA cluster Multi‑Site Replication, with the persistent disk as large as the largest HA cluster
plan offered.
For information about how multi-node service instances are restored, see Restore a Service
Instance in
Backing Up and Restoring VMware Tanzu SQL with MySQL for VMs
.
Procedure for Configuring Service Plans
For each plan that you want to use in your deployment:
1. Click the section for the plan. For example, Plan 1.
2. Select the plan for your desired topology.
The following tabs expand to show an example screenshot of each plan:
Warning: You cannot change the regions or networks after you Apply
Changes.
Warning: You must not set Plan 1 to Inactive. If you deactivate Plan 1, your
installation fails when you apply changes.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
62
Single Node Leader-Follower HA Cluster Multi‑Site Replication
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
63
3. Configure the fields as follows:
Field Description
Service
Plan
Access
Select one of the following options:
Enable (Default): Gives access to all orgs and displays the service plan to all
developers in the Marketplace.
Disable: Deactivates access to all orgs and hides the service plan to all developers in
the Marketplace. This deactivates creating new service instances of this plan.
Manual: Lets you manually control service access with the cf CLI. For more
information, see Controlling Access to Service Plans by Org.
Plan Name Accept the default or enter a name. This is the name that appears in the Marketplace for
developers.
Plan
Descriptio
n
Accept the default or enter a description to help developers understand plan features. VMware
recommends adding VM type details and disk size to this field.
Plan Quota Enter the maximum number of service instances that can exist at one time. If the plan quota field
is blank, the plan quota is set to the global quota by default. If you have selected the highly
available cluster plan, the Plan Quota maximum is 5.
For information about the global quota, see Setting Limits for On-Demand Service Instances.
Paid Plan Check this box to indicate that this service plan is paid.
MySQL VM
Type
Select a VM type for the MySQL nodes.
Jumpbox
VM Type
Only for highly available cluster plans. Select a VM type for the MySQL jumpbox node. This VM
is also called mysql-monitor.
MySQL
Persistent
Disk
Select a disk size. This disk stores the MySQL data.
For sizing recommendations, see Persistent Disk Usage.
Jumpbox
Persistent
Disk
Only for highly available cluster plans. Select a disk size. This disk stores backups.
For sizing recommendations, see Persistent Disk Usage.
MySQL
Availability
Zone(s)
BOSH deploys your service instances to the selected AZs. If more than one AZ is selected,
BOSH randomizes which AZ to place each VM.
Plan VM
Extensions
Specify a comma-separated list of supported VM Extensions you want to apply to service
instances created in this plan.
You can manage VM Extensions in Ops Manager or through the OM CLI. For more information,
see Create or Update a VM Extension or om create-vm-extension in GitHub.
NOTE: If you specify an extension that is not supported by Ops Manager (not present in the
BOSH cloud config), then instance creation attempts fail.
4. Click Save.
Note: If you want to replicate data across multiple foundations or data
centers, you must configure a Multi‑Site Replication plan in both foundations
using the same configurations.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
64
(Optional) Deactivate Service Plan
To deactivate a service plan:
1. If the service plan has existing service instances:
1. Click the section for the plan. For example, Plan 2.
2. Under Service Plan Access, select Disable.
3. Click Save.
4. Return to the Ops Manager Installation Dashboard and click Apply Changes to
redeploy.
5. When the deployment has redeployed, use the cf CLI or Apps Manager to delete all
existing service instances on the service plan.
6. Return to the Tanzu SQL for VMs tile configuration.
2. Click the section for the plan. For example, Plan 2.
3. Click Inactive.
4. Click Save.
Configure Global Settings
To configure global settings for all service instances:
1. Click Settings.
Warning: If you expect your developers to upgrade from one plan to another, do
not place the plans in separate AZs. For example, if you create Plan 1 in AZ1 and
Plan 2 in AZ2, developers receive an error and cannot continue if they try to
upgrade from Plan 1 to Plan 2. This prevents them from losing their data by
orphaning their disk in AZ1.
If you need to manually migrate the data from one AZ to another, see About Data
Migration in Tanzu SQL for VMs.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
65
2. Configure the fields as follows:
Field Instructions
Provide public IP
addresses to all
Service VMs
Select this check box if either of the following apply:
Your service instances need an external backup, blobstore, or syslog storage
You have configured BOSH to use an external blobstore
Maximum service
instances
Enter the global quota for all on-demand instances summed across every on-demand
plan. For information about determining global quotas, see Service Plan
Recommended Usage and Limitations.
Email address Enter an email address to send MySQL monitoring notifications to.
3. Click Save.
Configure MySQL
To set MySQL defaults and enable developers to customize their instances:
1. Click Mysql Configuration.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
66
2. Configure the fields as follows:
Field Instructions
Enable Lower
Case Table
Names
Select this check box to store all table names in lowercase. This sets the MySQL server
system variable lower_case_table_names to 1 on all Tanzu SQL for VMs instances by
default.
To permit developers to override this default, see the following check box.
For more information about lower_case_table_names , see the MySQL documentation.
Allow
Developers To
Override Lower
Case Table
Names
Select this check box to permit developers to override the configured default Enable
Lower Case Table Names value. For more information, see Optional Parameters for the
Tanzu SQL for VMs Service Instances.
Enable Local
Infile
Select this check box to activate data downloading from the local file system of the client.
VMware discourages selecting this check box. Before you activate local in-file, review the
security issues associated with LOAD DATA LOCAL. For more information, see the MySQL
documentation.
Limit binary log
disk use to 33%
of disk capacity
Select this check box if you have leader-follower or Multi‑Site Replication instances with
high-transaction rates. Limiting the size of the transaction logs reduces the risk of the
persistent disk reaching capacity and causing service interruption.
Wait Timeout Enter the amount of time in seconds that MySQL waits to close inactive connections. For
more information about wait_timeout, see the MySQL documentation.
Warning: Before you activate this feature, ensure all tables have
lowercase names. Tables with uppercase names are inaccessible after
enabling lowercase table names.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
67
Configure Backups
To learn how backups work, see About Backups.
To configure backups:
1. Click Backups.
2. Select a Backup configuration and follow the procedure for your storage solution in the
Configuring Automated Backups
topic:
3. Ceph or Amazon S3: Tanzu SQL for VMs runs an Amazon S3 client that saves backups to
an S3 bucket, a Ceph storage cluster, or another S3-compatible endpoint certified by
VMware.
For instructions about using Ceph or Amazon S3 for backups, see Back Up to Ceph or S3.
4. SCP: Tanzu SQL for VMs runs an Secure Copy Protocol (SCP) command that secure-copies
backups to a VM or physical machine operating outside of your deployment. This is the
fastest option.
SCP enables you to securely transfer files between two hosts. You can provision the backup
machine separately from your installation.
For instructions about using SCP for backups, see Back Up with SCP.
5. GCS: Tanzu SQL for VMs runs a Google Cloud Storage (GCS) SDK that saves backups to an
GCS bucket.
For instructions about using GCS for backups, see Back Up to GCS.
6. Azure: Tanzu SQL for VMs runs an Azure SDK that saves backups to an Azure storage
account.
For instructions about using Azure for backups, see Back Up to Azure Storage.
Note: You must configure backups. You cannot deactivate this feature.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
68
Configure Security
To configure the security settings for the MySQL service, do one or both:
Enable TLS for the MySQL service. See Configure TLS.
Store your service instance credentials in runtime CredHub. See Configure Secure Service
Instance Credentials.
Configure TLS
To activate TLS for the MySQL service:
1. Do the procedures in Preparing for TLS.
2. Click Security.
3. Under TLS Options, select one of the following:
4. Optional: This enables developers to configure their MySQL service VMs to use TLS.
5. Required: This enables developers to configure their MySQL service VMs to use TLS and
requires all MySQL service VMs to only accept secure connections.
6. Click Save.
7. After deploying the tile, notify your developers that they must activate TLS for their service
instances and activate TLS for their apps. See Using TLS.
Configure Secure Service Instance Credentials
You can store your service instance credentials in runtime CredHub instead of the Cloud Controller
database (CCDB). For more information about runtime CredHub, see CredHub.
To store your service instance credentials in runtime CredHub:
Warning: Selecting Required breaks any apps that are not currently
connecting over TLS.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
69
1. Ensure that you have configured the TAS for VMs tile to support securing service instance
credentials in runtime CredHub. For instructions, see Step 1: Configure the TAS for VMs
Tile.
2. Click Security.
3. Select the Enable Secure Service Instance Credentials check box.
4. Click Save.
5. After deploying the tile, notify the developers that they must unbind and rebind any
existing service instances bindings if they want to use secure service instance credentials.
Developers must:
1. Unbind the service instance from the app by running:
cf unbind-service APP SERVICE-INSTANCE
2. Rebind the service instance to the app by running:
cf bind-service PP SERVICE-INSTANCE
3. Restart the app to apply the new binding by running:
cf restart APP
4. Verify that the binding includes CredHub pointers in the VCAP_SERVICES
environment variable by running:
cf env APP
For example:
$ cf env my-app
Getting env variables for app my-app in org system / space exampl
e as admin...
OK
System-Provided:
{
"VCAP_SERVICES": {
"p.mysql": [
{
"credentials": {
"credhub-ref": "/c/548966e5-e333-4d65-8773-7b4e3bb6ca97/4a24
6b0b-83bb-46d0-b8ac-35a93374ae67/caf6e32e-7361-4869-9a57-54ab8ae6
7b3f/credentials"
},
[...]
Warning: If a developer rebinds an app to the Tanzu SQL for VMs service after
unbinding, they must also rebind any existing custom schemas to the app. When
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
70
Configure Monitoring
To activate monitoring and logging in the MySQL service:
1. Click Monitoring.
2. Configure the fields as follows:
Field Instructions
Metrics
Polling
Interval
Enter the amount of time in seconds for the frequency that the monitor polls for metrics. All
service instances emit metrics about the health and status of the MySQL server.
Enable
User
Statistics
Logging
Select this check box to collect user statistics. You can use these statistics to better understand
server activity and identify load sources. For more information about user statistics, see the
MySQl documentation.
Enable
Server
Activity
Logging
Select this check box to record who connects to the servers and what queries are processed
using the Percona Audit Log Plugin. For more information, see the Percona documentation
you rebind an app, stored code, programs, and triggers break. For more information
about binding custom schemas, see Use Custom Schemas.
Note: MySQL audit logs are not forwarded to the syslog server because they
can contain personally identifying information (PII) and secrets.
You can use the download-logs script to retrieve the logs, which each
MySQL cluster node VM stores in /var/vcap/store/mysql_audit_logs/.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
71
Enable
Read Only
Admin
User
Select this check box to create a read-only admin user, named roadmin, on each service
instance. This user can be used for auditing and monitoring without risking mutating or
changing any data. This is because roadmin cannot make changes to data.
For instructions about retrieving the credentials for roadmin, see Retrieve Admin and Read-Only
Admin Credentials for a Service Instance. The read-only admin user is always named roadmin,
however, the password varies by service instance.
3. Click Save.
Configure System Logging
To activate RFC 5424 system logging for the MySQL broker and service instance VMs:
1. Click Syslog.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
72
2. Click Yes.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
73
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
74
3. Configure the fields as follows:
Field Instructions
Address Enter the IP address or hostname of the syslog server for sending logs. For example:
logmanager.example.com.
Port Enter the port of the syslog server for sending logs. For example: 29279.
Transport
Protocol
Select the protocol you want to use to send system logs. VMware recommends using TCP.
Enable TLS If you select TCP, you can also select to send logs encrypted over TLS.
Permitted Peer Enter either:
The accepted fingerprint in SHA1 format.
The name of the remote peer. For example: *.example.com.
SSL Certificate Enter the SSL certificates for the syslog server. This ensures the logs are transported
securely.
4. Click Save.
Configure Service Instance Upgrades
This section configures the upgrade-all-service-instances errand. Tanzu SQL for VMs uses this
errand to upgrade service instances. For more information about the upgrade-all-service-
instances errand, see upgrade-all-service-instances.
To configure service instance upgrades.
Note: If your syslog server is external to your deployment, you might need
to select Provide public IP addresses to all Service VMs on the Settings
page. See Configure Global Settings.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
75
1. Click Service Instance Upgrades.
2. Configure the fields as follows:
Field Instructions
Number of
simultaneous
upgrades
Enter the maximum number of service instances that can upgrade at the same time. The
minimum value is 0 and the maximum is 1 less than the number of BOSH workers.
Increasing this value reduces the runtime of service instance upgrades.
Number of
upgrade canary
instances
Enter the number of service instances to upgrade first before upgrading the rest of the
instances. Increasing this value activates service instance upgrades to fail faster.
BOSH Upgrade
Timeout
Enter the amount of time in seconds to wait for BOSH to respond before timing out when
upgrading service instances. Increasing this value activates service instance upgrades to
fail faster.
Please type 'X' to
acknowledge
that you have
run...
If this is a fresh installation, enter `X`. Do not be concerned with the label instructions.
3. Click Save.
Review Errands (Optional)
Errands are scripts that run at specfic times to do various tasks. Tanzu SQL for VMs can run errands
to manage the broker and service instances. You do not need to change the default configurations
for errands.
Note: To determine the number of BOSH workers, navigate to BOSH
Director > Director Config and locate the value of Director Workers.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
76
Tanzu SQL for VMs uses the following types of errands:
Post-Deploy Errands: These errands run when you click Apply Changes.
Pre-Delete Errands: These errands run before you delete the Tanzu SQL for VMs tile.
Tanzu SQL for VMs also uses errands to configure leader-follower service instances. For more
information about leader-follower errands, see Errands.
You can use errands when troubleshooting the broker or service instances. For more information
about using errands for troubleshooting, see Run Service Broker Errands to Manage Brokers and
Instances.
To review errands:
1. Click Errands.
Warning: The Delete All Service Instances and Deregister Broker errand does
necessary cleanup tasks when you delete the Tanzu SQL for VMs tile or re-define
plans. Setting this errand to Off can cause problems when attempting to install the
tile again or re-define plans. VMware recommends that you do not set this errand
to Off.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
77
2. Review the settings for the following errands:
Errand Description
Post-Deploy Errands
Register On-demand MySQL
Broker
Registers a broker with the Cloud Controller and lists it in the Marketplace.
Smoke Tests Validates basic MySQL operations.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
78
Validate no IP-based
bindings in use before
upgrade-all-service-
instances
Checks if service instances have app bindings or service keys using IP
addresses or have a TLS certificate that is signed with an IP address. If either
is true, the installation fails.
Upgrade all On-demand
MySQL Service Instances
Upgrades existing instances of a service to its latest installed version.
If you want developers to individually upgrade service instances, set this
errand to OFF. For more information about individual service instance
upgrades, see About Individual Service Instance Upgrades.
Pre-Delete Errands
Delete All Service Instances
and Deregister Broker
Deletes all service instances and deregisters the broker.
Verify Stemcell Version and Apply All Changes
To verify your stemcell version and apply all changes:
1. Click Stemcell Library. For more information about using the Stemcell Library, see
Importing and Managing Stemcells.
2. Verify and, if necessary, import a new stemcell version.
3. Navigate to Ops Manager Dashboard > Review Pending Changes.
4. Click Apply Changes.
For information about the Xenial stemcells that are compatible with Tanzu SQL for VMs, see
Release Notes or VMware Tanzu Network.
Preparing for Multi-Site Replication
This topic provides instructions to operators on how to prepare foundations for multi‑site replication
using VMware Tanzu SQL with MySQL for VMs.
For information about multi-site architecture, see About Multi‑Site Replication.
Prerequisites
Before you can configure multi‑site replication for Tanzu SQL for VMs, you must confirm that you
have done the following:
1. Created two VMware Tanzu Application Service for VMs (TAS for VMs) foundations that
support Tanzu SQL for VMs v2.7 and later.
2. Deployed both foundations on a routable network. For example, the leader and follower
service instances must be able to connect to each other on their respective IP addresses.
You must verify that the CIDR ranges for the leader and follower nodes do not overlap.
3. Determined which of the following disaster recovery strategies you want to use for your
foundations: For more information, see About Active-Passive Topology. and About App-
Layer Active-Active Topology.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
79
4. Configured a global DNS load balancer to point to the two TAS for VMs foundations and
their local load balancers. For more information, see Configure Your GLB.
Enable Multi‑Site Replication
After two TAS for VMs foundations are created, you can enable multi‑site replication by doing the
following:
1. Configure networking rules. For more information on network rules, see Required
Networking Rules for Multi‑Site Replication.
2. Enable TLS for both foundations. When you paste the contents of your TLS CA certificate in
Preparing for TLS, paste both TLS CA certificates as shown in the following image:
1. Configure a Multi‑Site Replication plan on both foundations. See Configure Service Plans.
Preparing for TLS
Overview
TLS Required or Optional?
Generated or Provided CA Certificate
Workflow
If Using the Generated CA Certificate
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
80
If Providing Your Own CA Certificate
Find the CredHub Credentials in Ops Manager
Set a Custom CA Certificate
Add the CA Certificate
Enable TLS in Tanzu SQL for VMs
This topic provides an overview of how to prepare for using Transport Layer Security (TLS) with
VMware Tanzu SQL with MySQL for VMs to secure communication between apps and service
instances.
Overview
When you use TLS, you are provisioned a Tanzu SQL for VMs server with a certificate. With this
certificate, apps and clients can establish an encrypted connection with the service.
Through BOSH CredHub, Ops Manager generates a server certificate using a Certificate Authority
(CA) certificate.
If you do not want to use the CA certificate generated, you can provide your own CA certificate
and add it through the CredHub CLI. For an overview of the purpose and functionality of the
CredHub component, see CredHub.
Apps and clients use this CA certificate to check that the server certificate is trustworthy. A
trustworthy server certificate allows apps and clients to securely communicate with the Tanzu SQL
for VMs server.
VMware Tanzu Application Service for VMs (TAS for VMs) shares the CA certificate public
component in the following ways:
TAS for VMs provisions a copy of the CA certificate in the trusted store of each container’s
operating system. Apps written in Java and Spring automatically discover the CA certificate
in the trusted store.
TAS for VMs supplies the public CA certificate in an environment variable called
VCAP_SERVICES that exists in every container. Apps not written in Java and Spring can
retrieve the public component of the CA certificate from VCAP_SERVICES and use it to
establish an encrypted connection with the data service.
Warning: This procedure involves restarting all of the VMs in your deployment to
apply a CA certificate. The operation can take a long time to complete.
Note: This certificate is shared by multiple tiles. If you have already done this
procedure, you do not need to repeat it.
However, an operator must rotate the this certificate if it expires or if it becomes
compromised. For instructions about how to rotate your CA certificate follow the
steps in Rotating CA Certificates.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
81
TLS Required or Optional?
You must enable TLS if you are using Multi-Site Replication.
Also, if you use HA cluster, you need to enable TLS to facilitate restoring data from backup. This is
because to restore an HA cluster, you first create a Multi-Site Replication service instance. In turn,
before you can create a Multi-Site Replication service instance, you need TLS enabled. For
information about restoring an HA cluster, see Restore a Service Instance in
Backing Up and
Restoring VMware Tanzu SQL with MySQL for VMs
.
In production environments, VMware recommends enabling TLS as a security practice, regardless
of the database topology.
Generated or Provided CA Certificate
Ops Manager can generate a CA certificate for TLS to use.
Alternatively, you can choose to provide your own CA certificate for TLS to use.
Workflow
The workflow you follow to prepare for TLS depends on whether you use the CA certificate
generated by Ops Manager or if you bring your own CA certificate.
If Using the Generated CA Certificate
To use the CA certificate that Ops Manager generates through CredHub, follow this workflow to
enable TLS for VMware Tanzu SQL with MySQL for VMs:
1. An operator adds the CredHub-generated certificate to Ops Manager by performing the
procedures:
1. Find the CredHub Credentials in Ops Manager below
2. Add the CA Certificate below
2. An operator enables TLS in the tile configuration while installing Tanzu SQL for VMs. See
Enable TLS in Tanzu SQL for VMs below.
3. A developer modifies their app to communicate securely with the Tanzu SQL for VMs
server:
For Java and Spring apps: See Activate TLS for Java and Spring Apps.
For all other apps: See Activate TLS for Non-Spring Apps.
If Providing Your Own CA Certificate
To provide your own CA certificate instead of using the one that Ops Manager generates, follow
this workflow to enable TLS for VMware Tanzu SQL with MySQL for VMs:
Note: If you are using Ops Manager v2.5 or earlier, then Ops Manager does not
generate a certificate. You must generate one with CredHub. To use TLS with Ops
Manager v2.5 or earlier, see Preparing for TLS in MySQL for PCF v2.6.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
82
1. An operator provides a CA certificate to CredHub by performing the procedures:
1. Find the CredHub Credentials in Ops Manager below
2. Set a Custom CA Certificate below
3. Add the CA Certificate below.
2. An operator enables TLS in the tile configuration while installing Tanzu SQL for VMs. See
Enable TLS in Tanzu SQL for VMs below.
3. A developer modifies their app to communicate securely with the Tanzu SQL for VMs
server:
For Java and Spring apps: See Activate TLS for Java and Spring Apps.
For all other apps: See Activate TLS for Non-Spring Apps.
Find the CredHub Credentials in Ops Manager
To find the BOSH CredHub client name and client secret:
1. In the Ops Manager Installation Dashboard, click the BOSH Director tile.
2. Click the Credentials tab.
3. In the BOSH Director section, click the link to the BOSH Commandline Credentials.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
83
Click here to view a larger version of this image
4. Record the values for BOSH_CLIENT and BOSH_CLIENT_SECRET.
Here is an example of the credentials page:
{"credential":"BOSH_CLIENT=ops_manager
BOSH_CLIENT_SECRET=abCdE1FgHIjkL2m3n-3PqrsT4EUVwXy5
BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
BOSH_ENVIRONMENT=10.0.0.5 bosh "}
The BOSH_CLIENT is the BOSH CredHub client name and the BOSH_CLIENT_SECRET is the
BOSH CredHub client secret.
Set a Custom CA Certificate
Prerequisite: To complete this procedure, you need to have the Credhub CLI. For installation
instructions, see credhub-cli on GitHub.
Do this procedure if you are providing your own custom CA certificate instead of using the one
generated by Ops Manager or CredHub.
To add a custom CA Certificate to CredHub:
1. Record the information needed to log in to the BOSH Director VM by following the
procedure in Gather Credential and IP Address Information.
2. Log in to the Ops Manager VM by following the procedure in Log in to the Ops Manager
VM with SSH.
3. Set the API target of the CredHub CLI as your CredHub server by running:
credhub api \
https://BOSH-DIRECTOR-IP:8844 \
--ca-cert=/var/tempest/workspaces/default/root_ca_certificate
Where BOSH-DIRECTOR-IP is the IP address of the BOSH Director VM.
For example:
4. Log in to CredHub by running:
credhub login \
--client-name=CREDHUB-CLIENT-NAME \
--client-secret=CREDHUB-CLIENT-SECRET
Where
CREDHUB-CLIENT-NAME is the value you recorded for BOSH_CLIENT in Find the
CredHub Credentials in Ops Manager above.
$ credhub api \
https://10.0.0.5:8844 \
--ca-cert=/var/tempest/workspaces/default/root\_ca\_certificate
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
84
CREDHUB-CLIENT-SECRET is the value you recorded for BOSH_CLIENT_SECRET in Find
the CredHub Credentials in Ops Manager above.
For example:
5. Use the CredHub CLI to provide a CA certificate.
Create a new file called root.pem with the contents of the certificate. Then, run the
following command, specifying the path to root.pem and the private key for the certificate.
For example:
<pre class="terminal">$ credhub set \
--name="/services/tls_ca" \
--type="certificate" \
--certificate=./root.pem \
--private=ERKSOSMFF...</pre>
Add the CA Certificate
Prerequisite: To complete this procedure, you need to have the Credhub CLI. For installation
instructions, see credhub-cli on GitHub.
To add the CA Certificate to Ops Manager:
1. Record the CA certificate by running:
credhub get \
--name=/services/tls_ca \
-k ca
2. Navigate to the Ops Manager Installation Dashboard > BOSH Director > Security.
3. Append the contents of the CA certificate you recorded in the previous step into Trusted
Certificates.
4. Click Save.
Enable TLS in Tanzu SQL for VMs
To enable TLS in the Tanzu SQL for VMs tile:
1. Enable TLS by doing one of the following:
If you are configuring TLS for an existing installation: Follow the procedure in
Configure Security.
$ credhub login \
--client-name=credhub \
--client-secret=abcdefghijklm123456789
Note: Your deployment can have multiple CA certificates. VMware
recommends a dedicated CA certificate for services.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
85
If you are configuring TLS for a new installation: Follow the procedures in
Installing and Configuring Tanzu SQL for VMs, including enabling TLS in the
Configure Security section.
2. Navigate to Ops Manager Installation Dashboard > Review Pending Changes.
3. Ensure that the CA certificate is deployed to all VMs by selecting:
VMware Tanzu Application Service for VMs
VMware Tanzu SQL with MySQL for VMs
The Upgrade All On-Demand Service Instances errand
4. Click Apply Changes. This restarts all the VMs in your deployment and applies your CA
certificate.
Setting Limits for On-Demand Service Instances
Create Global-level Quotas
Create Plan-level Quotas
Create and Set Org-level Quotas
Create and Set Space-level Quotas
View Current Org and Space-level Quotas
Monitor Quota Use and Service Instance Count
Calculate Resource Costs for On-Demand Plans
Calculate Maximum Resource Cost Per On-Demand Plan
Calculate Maximum Resource Cost for All On-Demand Plans
Calculate Actual Resource Cost of all On-Demand Plans
On-demand provisioning is intended to accelerate app development by eliminating the need for
development teams to request and wait for operators to create a service instance. However, to
control costs, operations teams and administrators must ensure responsible use of resources.
There are several ways to control the provisioning of on-demand service instances by setting
various quotas at these levels:
Global
Plan
Org
Space
After you set quotas, you can:
View Current Org and Space-level Quotas
Monitor Quota Use and Service Instance Count
Calculate Resource Costs for On-Demand Plans
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
86
Create Global-level Quotas
Each on-demand service has a separate service broker. A global quota at the service level sets the
maximum number of service instances that can be created by a given service broker. If a service
has more than one plan, then the number of service instances for all plans combined cannot
exceed the global quota for the service.
The operator sets a global quota for each service tile independently. For example, if you have two
service tiles, you must set a separate global service quota for each of them.
When the global quota is reached for a service, no more instances of that service can be created
unless the quota is increased, or some instances of that service are deleted.
Create Plan-level Quotas
A service may offer one or more plans. You can set a separate quota per plan so that instances of
that plan cannot exceed the plan quota. For a service with multiple plans, the total number of
instances created for all plans combined cannot exceed the global quota for the service. If the plan
quota field is blank, the plan quota is set to the global quota by default.
When the plan quota is reached, no more instances of that plan can be created unless the plan
quota is increased or some instances of that plan are deleted.
Create and Set Org-level Quotas
An org-level quota applies to all on-demand services and sets the maximum number of service
instances an organization can create within their foundation. For example, if you set your org-level
quota to 100, developers can create up to 100 service instances in that org using any combination
of on-demand services.
When this quota is met, no more service instances of any kind can be created in the org unless the
quota is increased or some service instances are deleted.
To create and set an org-level quota, do the following:
1. Run this command to create a quota for service instances at the org level:
Where:
QUOTA-NAME—A name for this quota
TOTAL-MEMORY—Maximum memory used by all service instances combined
INSTANCE-MEMORY—Maximum memory used by any single service instance
ROUTES—Maximum number of routes allowed for all service instances combined
SERVICE-INSTANCES—Maximum number of service instances allowed for the org
For example:
cf create-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES
-s SERVICE-INSTANCES --allow-paid-service-plans
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
87
2. Associate the quota you created above with a specific org by running the following
command:
For example:
For more information on managing org-level quotas, see Creating and Modifying Quota Plans.
Create and Set Space-level Quotas
A space-level service quota applies to all on-demand services and sets the maximum number of
service instances that can be created within a given space in a foundation. For example, if you set
your space-level quota to 100, developers can create up to 100 service instances in that space
using any combination of on-demand services.
When this quota is met, no more service instances of any kind can be created in the space unless
the quota is updated or some service instances are deleted.
To create and set a space-level quota, do the following:
1. Run the following command to create the quota:
Where:
QUOTA-NAME—A name for this quota
TOTAL-MEMORY—Maximum memory used by all service instances combined
INSTANCE-MEMORY—Maximum memory used by any single service instance
ROUTES—Maximum number of routes allowed for all service instances combined
SERVICE-INSTANCES—Maximum number of service instances allowed for the org
For example:
2. Associate the quota you created above with a specific space by running the following
command:
For example:
cf create-quota myquota -m 1024mb -i 16gb -r 30 -s 50 --allow-paid-serv
ice-plans
cf set-quota ORG-NAME QUOTA-NAME
cf set-quota dev_org myquota
cf create-space-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r
ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans
cf create-space-quota myspacequota -m 1024mb -i 16gb -r 30 -s 50 --allo
w-paid-service-plans
cf set-space-quota SPACE-NAME QUOTA-NAME
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
88
For more information on managing space-level quotas, see Creating and Modifying Quota Plans.
View Current Org and Space-level Quotas
To view org quotas, run the following command.
cf org ORG-NAME
To view space quotas, run the following command:
cf space SPACE-NAME
For more information on managing org and space-level quotas, see the Creating and Modifying
Quota Plans.
Monitor Quota Use and Service Instance Count
Service-level and plan-level quota use, and total number of service instances, are available through
the on-demand broker metrics emitted to Loggregator. These metrics are listed below:
Metric Name Description
on-demand-broker/SERVICE-NAME/quota_remaining Quota remaining for all instances across all
plans
on-demand-broker/SERVICE-NAME/PLAN-NAME/
quota_remaining
Quota remaining for a specific plan
on-demand-broker/SERVICE-NAME/total_instances Total instances created across all plans
on-demand-broker/SERVICE-NAME/PLAN-NAME/
total_instances
Total instances created for a specific plan
You can also view service instance usage information in Apps Manager. For more information, see
Reporting Instance Usage with Apps Manager.
Calculate Resource Costs for On-Demand Plans
On-demand plans use dedicated VMs, disks, and various other resources from an IaaS, such as
AWS. To calculate maximum resource cost for plans individually or combined, you multiply the
quota by the cost of the resources selected in the plan configuration(s). The specific costs depend
on your IaaS.
To view configurations for your Tanzu SQL for VMs on-demand plan, do the following:
1. Navigate to Ops Manager Installation Dashboard > VMware Tanzu SQL with MySQL for
VMs > Settings.
cf set-space-quota myspace myspacequota
Note: Quota metrics are not emitted if no quota has been set.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
89
Single Node and Leader Follower Galera
2. Click the section for the plan you want to view. For example, Plan 1.
The image below shows an example that includes the VM type and persistent disk selected for the
server VMs, as well as the quota for this plan.
Click the tabs below to show an example of each plan:
Calculate Maximum Resource Cost Per On-Demand Plan
To calculate the maximum cost of VMs and persistent disk for each plan, do the following
calculation:
plan quota x cost of selected resources
For example, if you selected the options in the above image, you have selected a VM type micro
and a persistent disk type 20 GB, and the plan quota is 15. The VM and persistent disk types have
an associated cost for the IaaS you are using. Therefore, to calculate the maximum cost of
resources for this plan, multiply the cost of the resources selected by the plan quota:
(15 x cost of micro VM type) + (15 x cost of 20 GB persistent disk) = max cost per plan
Calculate Maximum Resource Cost for All On-Demand Plans
To calculate the maximum cost for all plans combined, add together the maximum costs for each
plan. This assumes that the sum of your individual plan quotas is less than the global quota.
Here is an example:
Note: Although operators can limit on-demand instances with plan quotas and a
global quota, as described in the above topics, IaaS resource usage still varies based
on the number of on-demand instances provisioned.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
90
(plan1 quota x plan1 resource cost) + ( plan2 quota x plan2 resource cost) = max cost for all
plans
Calculate Actual Resource Cost of all On-Demand Plans
To calculate the current actual resource cost across all your on-demand plans:
1. Find the number of instances currently provisioned for each active plan by looking at the
total_instance metric for that plan.
2. Multiply the total_instance count for each plan by that plan’s resource costs. Record the
costs for each plan.
3. Add up the costs noted in Step 2 to get your total current resource costs.
For example:
(plan1 total_instances x plan1 resource cost) + (plan2 total_instances x plan2 resource cost) =
current cost for all plans
Global
Plan
Org
Space
After you set quotas, you can:
View Current Org and Space-level Quotas
Monitor Quota Use and Service Instance Count
Calculate Resource Costs for On-Demand Plans
Create Global-level Quotas
Each on-demand service has a separate service broker. A global quota at the service level sets the
maximum number of service instances that can be created by a given service broker. If a service
has more than one plan, then the number of service instances for all plans combined cannot
exceed the global quota for the service.
The operator sets a global quota for each service tile independently. For example, if you have two
service tiles, you must set a separate global service quota for each of them.
When the global quota is reached for a service, no more instances of that service can be created
unless the quota is increased, or in some instances of that service are deleted.
Create Plan-level Quotas
A service might offer one or more plans. You can set a separate quota per plan so that instances of
that plan cannot exceed the plan quota. For a service with multiple plans, the total number of
instances created for all plans combined cannot exceed the global quota for the service. If the plan
quota field is blank, the plan quota is set to the global quota by default.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
91
When the plan quota is reached, no more instances of that plan can be created unless the plan
quota is increased or some instances of that plan are deleted.
Create and Set Org-level Quotas
An org-level quota applies to all on-demand services and sets the maximum number of service
instances an organization can create within their foundation. For example, if you set your org-level
quota to 100, developers can create up to 100 service instances in that org using any combination
of on-demand services.
When this quota is met, no more service instances of any kind can be created in the org unless the
quota is increased or in some service instances are deleted.
To create and set an org-level quota, do the following:
1. Run the following command to create a quota for service instances at the org level:
cf create-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERV
ICE-INSTANCES --allow-paid-service-plans
Where:
QUOTA-NAME—A name for this quota.
TOTAL-MEMORY—Maximum memory used by all service instances combined.
INSTANCE-MEMORY—Maximum memory used by any single service instance.
ROUTES—Maximum number of routes allowed for all service instances combined.
SERVICE-INSTANCES—Maximum number of service instances allowed for the org.
For example:
2. Associate the quota you previously created with a specific org by running the following
command:
cf set-quota ORG-NAME QUOTA-NAME
For example:
For more information on managing org-level quotas, see Creating and Modifying Quota Plans.
Create and Set Space-level Quotas
A space-level service quota applies to all on-demand services and sets the maximum number of
service instances that can be created within a given space in a foundation. For example, if you set
your space-level quota to 100, developers can create up to 100 service instances in that space
using any combination of on-demand services.
cf create-quota myquota -m 1024mb -i 16gb -r 30 -s 50 --allow-paid-serv
ice-plans
cf set-quota dev_org myquota
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
92
When this quota is met, no more service instances of any kind can be created in the space unless
the quota is updated or in some service instances are deleted.
To create and set a space-level quota, do the following:
1. Run the following command to create the quota:
cf create-space-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -
s SERVICE-INSTANCES --allow-paid-service-plans
Where:
QUOTA-NAME—A name for this quota.
TOTAL-MEMORY—Maximum memory used by all service instances combined.
INSTANCE-MEMORY—Maximum memory used by any single service instance.
ROUTES—Maximum number of routes allowed for all service instances combined.
SERVICE-INSTANCES—Maximum number of service instances allowed for the org.
For example:
2. Associate the quota you previously created with a specific space by running the following
command:
cf set-space-quota SPACE-NAME QUOTA-NAME
For example:
For more information on managing space-level quotas, see Creating and Modifying Quota Plans.
View Current Org and Space-level Quotas
To view org quotas, run the following command.
cf org ORG-NAME
To view space quotas, run the following command:
cf space SPACE-NAME
For more information on managing org and space-level quotas, see the Creating and Modifying
Quota Plans.
Monitor Quota Use and Service Instance Count
Service-level and plan-level quota use, and total number of service instances, are available through
the on-demand broker metrics emitted to Loggregator.
cf create-space-quota myspacequota -m 1024mb -i 16gb -r 30 -s 50 --allo
w-paid-service-plans
cf set-space-quota myspace myspacequota
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
93
Single Node and Leader Follower Galera
These are the metrics:
Metric Name Description
on-demand-broker/SERVICE-NAME/quota_remaining Quota remaining for all instances across all
plans
on-demand-broker/SERVICE-NAME/PLAN-NAME/
quota_remaining
Quota remaining for a specific plan
on-demand-broker/SERVICE-NAME/total_instances Total instances created across all plans
on-demand-broker/SERVICE-NAME/PLAN-NAME/
total_instances
Total instances created for a specific plan
You can also view service instance usage information in Apps Manager. For more information, see
Reporting Instance Usage with Apps Manager.
Calculate Resource Costs for On-Demand Plans
On-demand plans use dedicated VMs, disks, and various other resources from an IaaS, such as
AWS. To calculate maximum resource cost for plans individually or combined, you multiply the
quota by the cost of the resources selected in the plan configuration(s). The specific costs depend
on your IaaS.
To view configurations for your Tanzu SQL for VMs on-demand plan, do the following:
1. Navigate to Ops Manager Installation Dashboard > VMware Tanzu SQL with MySQL for
VMs > Settings.
2. Click the section for the plan you want to view. For example, Plan 1.
The following image shows an example that includes the VM type and persistent disk selected for
the server VMs, and the quota for this plan.
Click the following tabs to show an example of each plan:
Note: Quota metrics are not emitted if no quota has been set.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
94
Calculate Maximum Resource Cost Per On-Demand Plan
To calculate the maximum cost of VMs and persistent disk for each plan, do the following
calculation:
plan quota x cost of selected resources
For example, if you selected the options in the previous image, you have selected a VM type micro
and a persistent disk type 20 GB, and the plan quota is 15. The VM and persistent disk types have
an associated cost for the IaaS you are using. Therefore, to calculate the maximum cost of
resources for this plan, multiply the cost of the resources selected by the plan quota:
(15 x cost of micro VM type) + (15 x cost of 20 GB persistent disk) = max cost per plan
Calculate Maximum Resource Cost for All On-Demand Plans
To calculate the maximum cost for all plans combined, add together the maximum costs for each
plan. This assumes that the sum of your individual plan quotas is less than the global quota.
Here is an example:
(plan1 quota x plan1 resource cost) + ( plan2 quota x plan2 resource cost) = max cost for all
plans
Calculate Actual Resource Cost of all On-Demand Plans
To calculate the current actual resource cost across all your on-demand plans:
Note: Although operators can limit on-demand instances with plan quotas and a
global quota, as described in the above topics, IaaS resource usage still varies based
on the number of on-demand instances provisioned.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
95
1. Find the number of instances currently provisioned for each active plan by looking at the
total_instance metric for that plan.
2. Multiply the total_instance count for each plan by that plan’s resource costs. Record the
costs for each plan.
3. Add up the costs noted in Step 2 to get your total current resource costs.
For example:
(plan1 total_instances x plan1 resource cost) + (plan2 total_instances x plan2 resource cost) =
current cost for all plans
Controlling Access to Service Plans by Org
This topic gives instructions for controlling access to service plans by org. You can also set limits for
the number of service intances globally and per plan.
For more information, see Setting Limits for On-Demand Instances.
Control Access to Service Plan by Orgs
You are able to control which CF orgs are able to access specific service plans in VMware Tanzu
SQL with MySQL for VMs. By default, active service plans are visible to all orgs. Controlling which
orgs have access to a specific service plan enables you to ensure that the resource-intensive
service plans are only available to the orgs that explicitly need them.
To configure Tanzu SQL for VMs to control service-plan access, do the following:
1. Set the Service Plan Access field to Manual on any active service plan.
For more information, see Configure Active Service Plans.
2. Click Save.
3. Return to the Ops Manager Installation Dashboard and click Review Pending Changes.
4. Click Apply Changes.
5. For each org that you want to use the service plan, do the following:
1. Log in to the Cloud Foundry Command Line Interface (cf CLI) as an admin user:
2. Activate service access to the org:
Where:
PLAN: The name of the specific plan to enable, set to manual in step 1.
ORGANIZATION: The name of the org that needs access to PLAN.
For example,
cf login
cf enable-service-access p.mysql -p PLAN -o ORGANIZATION
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
96
The org can now use the plan.
For information on modifying and viewing service-plan access, see Managing Access to Service
Plans.
VMware Tanzu SQL with MySQL for VMs - Managing Tanzu
MySQL for VMs
Managing VMware Tanzu SQL with MySQL for VMs
Upgrading
Configuring Automated Backups
Manually Restoring from Backup
Accessing a Database as an Admin User
Rotating Certificates
Resolving Service Interruptions
Running Errands
Troubleshooting
Leader-Follower Procedures
Triggering a Leader-Follower Failover
Highly Available Clusters Procedures
Bootstrapping
Running mysql-diag
About the Replication Canary
Upgrading VMware Tanzu SQL with MySQL for VMs
This topic explains how to upgrade the VMware Tanzu SQL with MySQL for VMs service and
existing service instances. It also explains the service interruptions that can result from service
changes and upgrades and from failures at the process, VM, and IaaS level.
For product versions and upgrade paths, see Upgrade Planner.
Upgrade Tanzu SQL for VMs
To upgrade the Tanzu SQL for VMs service, follow the Ops Manager process that you use to install
the service for the first time. Your configuration settings migrate to the new version automatically.
To upgrade Tanzu SQL for VMs:
1. Review the Release Notes for the version you are upgrading to.
$ cf enable-service-access p.mysql -o prodteam -p db-large
Enabling access to plan db-large of service p.mysql for org prodt
eam as admin...
OK
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
97
2. Download the Ubuntu Xenial stemcell from VMware Tanzu Network, and import it into the
Ops Manager Stemcell Library. For instructions, see Verify Stemcell Version and Apply All
Changes.
3. Download the desired version of the product from VMware Tanzu Network.
4. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload
the product file.
5. Under the Import a Product button, click + next to Tanzu SQL for VMs. This adds the tile
to your staging area.
6. Click the newly-added Tanzu SQL for VMs tile to review its configuration panes. Click Save
on any panes where you make changes.
7. If the following conditions all apply, select the Force path style access to bucket
checkbox:
You are upgrading from v2.8.x or v2.9.0 to v2.9.1 or later.
You use S3-compatible storage to store backups. See Back Up to Amazon S3 or
Ceph in
Configuring Automated Backups
.
The S3-compatible bucket, such as Minio, requires a path-style URL instead of a
virtual hosted-style URL.
The Force path style access to bucket checkbox is on the Backups pane. For more
information, see Back Up to Amazon S3 or Ceph in
Configuring Automated Backups
and
v2.9.1 in
Release Notes
.
8. (Optional) If you want developers to individually upgrade service instances, navigate to the
Errands pane and select Off for Upgrade all On-demand MySQL Service Instances.
By default, the upgrade-all-service-instances errand runs after each upgrade. For more
information, see About Individual Service Instance Upgrades.
9. Navigate to Ops Manager Dashboard > Review Pending Changes. For more information
about this Ops Manager page, see Reviewing Pending Product Changes.
10. For the Tanzu SQL for VMs tile, enable the Register On-demand MySQL Broker errand if
the errand is not already enabled.
11. Click Apply Changes.
Upgrading the Tanzu SQL for VMs service and service instances can temporarily interrupt the
service. For more information, see Service Interruptions.
About Individual Service Instance Upgrades
Note: To decrease the runtime for service instance upgrades, configure the
upgrade-all-service-instances errand in the tile. For instructions about
configuring this errand, see Configure Service Instance Upgrades.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
98
After you upgrade the Tanzu SQL for VMs tile, existing service instances must be upgraded to use
the latest version of the tile. Developers cannot create new bindings to service instances that have
not been upgraded.
To decrease runtime for service instance upgrades, developers can individually upgrade on-demand
service instances using the Cloud Foundry Command Line Interface (cf CLI). Developers can
upgrade individual service instances by following the procedure in Upgrade an Individual Service
Instance.
Developers can only upgrade individual service instances if you disable the upgrade-all-service-
instances errand when upgrading the tile. By default, Tanzu SQL for VMs runs this errand when
you upgrade the tile. However, this operation can take a long time. You must also ensure that the
register-broker errand is run during upgrades. For more information about the register broker
errand, see register-broker.
Service Interruptions
Service changes, upgrades, and failures at the process, VM, and IaaS level can cause outages in the
Tanzu SQL for VMs service.
Read this section if:
You are planning an upgrade.
You are experiencing a service interruption and are wondering why.
You are planning to update or change a service instance and want to know if it might cause
a service interruption.
Stemcell or Service Update
An operator updates a stemcell version or their version of Tanzu SQL for VMs.
Impact: Apps lose access to the MySQL service while Ops Manager updates the service
instance they are bound to. The service resumes within 10–15 minutes.
Required Actions: None. If the update deploys successfully, apps reconnect automatically.
Plan Change
A developer changes their service instance to provide a different service plan, using cf update-
service or Apps Manager.
Impact: Apps lose access to the MySQL service while Ops Manager updates the service
instance they are bound to. The service resumes within 10–15 minutes.
Required Actions: None. If the plan change deploys successfully, the apps reconnect
automatically.
Service Broker Deployments
Note: For developers to be able to upgrade individual service instances, you must
use VMware Tanzu Application Service for VMs v2.7 or later.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
99
Automated backups are not taken during service broker deployments.
When the service broker is unavailable, such as during upgrades and re-deployments, automated
backups fail. Automated backups resume according to schedule when the service broker is online
again.
For general information about backups, see Backing Up and Restoring VMware Tanzu SQL with
MySQL for VMs and Configuring Automated Backups.
Configuring Automated Backups
This topic describes how to configure automated, physical backups for VMware Tanzu SQL with
MySQL for VMs.
Developers can create physical backups using the Cloud Foundry Command Line Interface (cf CLI)
or logical backups using mysqldump. For more information about physical backups, see Backing Up
and Restoring VMware Tanzu SQL with MySQL for VMs. For more information about logical
backups, see Backing Up and Restoring with mysqldump.
You can restore a physical backup by following the procedures in Restore a Service Instance.
About Configuring Automated Backups
You can configure Tanzu SQL for VMs to automatically back up databases to external storage.
Tanzu SQL for VMs backs up the entire data directory for each service instance.
Tanzu SQL for VMs backs up your database on a schedule. You configure this schedule with a cron
expression.
To configure backups, follow the procedure for your external storage solution:
Back Up Using SCP
Back Up to Amazon S3 or Ceph
Back Up to GCS
Back Up to Azure Storage
Back Up Using SCP
Secure copy protocol (SCP) enables operators to use any storage solution on the destination VM.
This is the fastest method for backing up your database.
When you configure backups with SCP, Tanzu SQL for VMs runs an SCP command that uses SFTP
to securely copy backups to a VM or physical machine operating outside of your deployment. The
operator provisions the backup machine separately from their installation.
Note: Configuring a cron expression overrides the default schedule for your service
instance.
Developers can override the default for their service instance. For more
information, see Backup Schedule.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
100
To back up your database using SCP:
Create a Public and Private Key Pair
Configure Backups in Ops Manager
Create a Public and Private Key‑Pair
Tanzu SQL for VMs accesses a remote host as a user with a private key for authentication. VMware
recommends that this user and key-pair is only used for Tanzu SQL for VMs.
1. Determine the remote host that you use to store backups for Tanzu SQL for VMs. Ensure
that the MySQL service instances can access the remote host.
2. (Recommended) Create a new user for Tanzu SQL for VMs on the destination VM.
3. (Recommended) Create a new public and private key-pair for authenticating as the above
user on the destination VM.
Configure Backups in Ops Manager
Use Ops Manager to configure Tanzu SQL for VMs to back up using SCP.
1. In Ops Manager, open the Tanzu SQL for VMs tile Backups pane.
2. Select SCP.
Note: VMware recommends using a VM outside your deployment for the
destination of SCP backups. As a result, you might need to enable public IPs
for the MySQL VMs.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
101
3. Configure the fields as follows:
Field Instructions
Userna
me
Enter the user that you created in Create a Public and Private Key‑Pair.
Private
Key
Enter the private key that you created in Create a Public and Private Key‑Pair.
Store the public key that is used for SSH and SCP access on the destination VM.
Hostna
me
Enter the IP address or DNS entry that is used to access the destination VM.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
102
Destinat
ion
Director
y
Enter the directory that Tanzu SQL for VMs uploads backups to.
SCP
Port
Enter the SCP port number for SSH. This port usually is 22.
Cron
Schedul
e
Enter a cron expression using standard syntax. The cron expression sets the schedule for taking
backups for each service instance. This overrides the default schedule for your service instance. Test
your cron expression using a website such as Crontab Guru.
Fingerp
rint
(Optional) Enter the fingerprint for the destination VM public key. The fingerprint detects any
changes to the destination VM.
Back Up to Amazon S3 or Ceph
When you configure backups for Amazon S3 or Ceph, Tanzu SQL for VMs runs an Amazon S3 client
that saves the backups to one of the following:
an Amazon S3 bucket
a Ceph storage cluster
another S3-compatible endpoint certified by VMware
For information about Amazon S3 buckets, see the Amazon documentation.
For information about Ceph storage clusters, see the Ceph documentation.
To back up your database to Amazon S3 or Ceph:
Create a Custom Policy and Access Key
Configure Backups in Ops Manager
Create a Custom Policy and Access Key
Tanzu SQL for VMs accesses your S3 bucket through a user account. VMware recommends that
this account be only used by Tanzu SQL for VMs. You must apply a minimal policy that enables the
user account upload backups to your S3 bucket. Then give the policy the permission to list and
upload to buckets.
The procedure in this section assumes that you are using an Amazon S3 bucket. If you are using a
Ceph or another S3-compatible bucket to create the policy and access key, follow the
documentation for your storage solution. For more information about Ceph S3 bucket policies, see
the Ceph documentation.
To create a policy and access key in AWS:
1. Create a policy for your Tanzu SQL for VMs user account.
In AWS, create a new custom policy by following this procedure in the AWS
documentation.
Note: Developers can override the default for their service instance. For more
information, see Backup Schedule.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
103
Paste in the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "MySQLBackupPolicy",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::MY_BUCKET_NAME/*",
"arn:aws:s3:::MY_BUCKET_NAME"
]
}
]
}
2. Record the Access Key ID and Secret Access Key user credentials for a new user account
by following this procedure in the AWS documentation. Ensure you select Programmatic
access and Attach existing policies to user directly. You must attach the policy you
created in the previous step.
Configure Backups in Ops Manager
Use Ops Manager to connect Tanzu SQL for VMs to your S3 account.
Prerequisite: Before beginning this procedure, you must have an S3 bucket in which to store the
backups.
1. In Ops Manager, open the Tanzu SQL for VMs tile Backups pane.
2. Select Ceph or Amazon S3.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
104
Endpoint
URL
Enter the S3-compatible endpoint URL for uploading backups.
The URL must start with http:// or https://.
The default is https://s3.amazonaws.com.
If you are using a public S3 endpoint, see the S3 Endpoint procedure in Step 3: Director Config
Page.
Region Enter the region where your bucket is located.
Bucket
Name
Enter the name of your bucket.
Do not include an s3:// prefix, a trailing /, or underscores. VMware recommends using the
naming convention DEPLOYMENT-backups. For example, sandbox-backups.
Force path
style access
to bucket
The default behavior in Tanzu SQL for VMs 2.9 and later uses a virtual-style URL.
Select this check box if you use:
Amazon S3 and your bucket name is not compatible with virtual hosted-style URLs.
An S3-compatible endpoint such as Minio that might require path-style URLs.
For general information about the deprecation of S3 path-style URLs, see AWS blog posts:
Amazon S3 Path Deprecation Plan – The Rest of the Story and the subsequent Update to
Amazon S3 Path Deprecation Plan.
This checkbox is not available in Tanzu SQL for VMs v2.9.0.
Bucket Path (Optional) Enter the path in the bucket to store backups.
You can use this to keep the backups from this foundation separate from those of other
foundations that might also backup to this bucket. For example, Foundation-1. This field is
only available as of v2.9.4.
Cron
Schedule
Enter a cron expression using standard syntax. The cron expression sets the schedule for taking
backups for each service instance. This overrides the default schedule for your service instance.
Test your cron expression using a website such as Crontab Guru.
Back Up to GCS
When you configure backups for a Google Cloud Storage (GCS) bucket, Tanzu SQL for VMs runs a
GCS SDK that saves backups to a GCS bucket.
For information about GCS buckets, see the GCS documentation.
To back up your database to Google Cloud Storage (GCS):
Create a Service Account and Private Key
Configure Backups in Ops Manager
Create a Service Account and Private Key
Note: If you are using a blobstore that uses a specific set of domains in its
server certificate, add a new wildcard domain or use path-style URLs if
supported by the blobstore.
Note: Developers can override the default for their service instance. For
more information, see Backup Schedule.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
106
Tanzu SQL for VMs accesses your GCS bucket through a service account. VMware recommends
that this account is only used by Tanzu SQL for VMs. You must apply a minimal policy that enables
the service account to upload backups to your GCS bucket.
The service account needs the following permissions:
List and upload to buckets
(Optional) Create buckets if they do not already exist
To create a service account and private key in GCS:
1. Create a new service account by following this procedure in the GCS documentation.
When you create the service account:
1. Enter a unique name for the service account name.
2. Add the Storage Admin role.
3. Create and download a private key JSON file.
Configure Backups in Ops Manager
Use Ops Manager to connect Tanzu SQL for VMs to your GCS account.
1. In Ops Manager, open the Tanzu SQL for VMs tile Backups pane.
2. Select GCS.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
107
3. Configure the fields as follows:
Field Instructions
Project
ID
Enter the Project ID for the Google Cloud project that you are using.
Bucket
name
Enter the bucket name that Tanzu SQL for VMs uploads backups to.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
108
Bucket
Path
(Optional) Enter the path in the bucket to store backups.
You can use this to keep the backups from this foundation separate from those of other
foundations that might also backup to this bucket. For example, Foundation-1. This field is only
available as of v2.9.4.
Service
Account
JSON
Enter the contents of the service account JSON file that you downloaded when creating a service
account in Create a Service Account and Private Key.
Cron
Schedul
e
Enter a cron expression using standard syntax. The cron expression sets the schedule for taking
backups for each service instance. This overrides the default schedule for your service instance.
Test your cron expression using a website such as Crontab Guru.
Back Up to Azure Storage
When you configure backups for Azure Storage, Tanzu SQL for VMs runs an Azure SDK that saves
backups to an Azure storage account.
For information about Azure Storage, see the Azure documentation.
To back up your database to Azure Storage:
Create a Storage Account and Access Key
Configure Backups in Ops Manager
Create a Storage Account and Access Key
Tanzu SQL for VMs accesses your Azure Storage account through a storage access key. VMware
recommends that this account be only used by Tanzu SQL for VMs. You must apply a minimal
policy that enables the storage account upload backups to your Azure Storage.
The storage account needs the following permissions:
List and upload to buckets
(Optional) Create buckets if they do not already exist
To create a storage account and access key:
1. Create a new storage account by following this procedure in the Azure documentation.
2. View your access key by following this procedure in the Azure documentation
Configure Backups in Ops Manager
To back up your database to your Azure Storage account:
1. In Ops Manager, open the Tanzu SQL for VMs tile Backups pane.
2. Select Azure.
Note: Developers can override the default for their service instance. For more
information, see Backup Schedule.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
109
Azure
Storage
Access
Key
Enter one of the storage access keys that you viewed in Create a Storage Account and Access
Key.
Containe
r Name
Enter the container name that Tanzu SQL for VMs uploads backups to.
Blob
Store
Base URL
To use an on-premise blob storage, enter the hostname of the blob storage. By default, backups
are sent to the public Azure blob storage.
Bucket
Path
(Optional) Enter the path in the bucket to store backups.
You can use this to keep the backups from this foundation separate from those of other
foundations that might also backup to this bucket. For example, Foundation-1. This field is only
available as of v2.9.4.
Cron
Schedule
Enter a cron expression using standard syntax. The cron expression sets the schedule for taking
backups for each service instance. This overrides the default schedule for your service instance.
Test your cron expression using a website such as Crontab Guru.
Manually Restoring from Backup
If you are restoring a service instance to the same foundation and you still have the original service
instance, then follow the instructions in Backing Up and Restoring VMware Tanzu SQL with MySQL
for VMs.
This topic describes how to manually restore a VMware Tanzu SQL with MySQL for VMs service
instance from a backup in the following cases:
You have lost or deleted the service instance that backup came from.
You are restoring to a different foundation.
Overview
Restoring Tanzu SQL for VMs from backup is a manual process primarily intended for disaster
recovery or migrating data to a different foundation. Restoring a Tanzu SQL for VMs service
instance replaces all of its data and running state.
To restore a Tanzu SQL for VMs instance from a backup:
1. Identify and Download the Backup Artifact
2. Retrieve Backup Encryption Key
3. Create and Prepare a New Service Instance for Restore
Note: Developers can override the default for their service instance. For more
information, see Backup Schedule.
Note: VMware recommends that you always configure a single instance plan to
streamline the restore process for leader-follower and HA cluster plans.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
111
4. Restore the Service Instance
5. Restage the Service Instance
Identify and Download the Backup Artifact
The procedure you need to follow to find and download the backup artifact depends on whether or
not the service instance was deleted.
Follow the procedure for your situation:
If Restoring a Backup from a Deleted Service Instance
If Restoring to a Different Foundation
If Restoring a Backup from a Deleted Service Instance
If you are restoring a backup from a lost or deleted instance, you cannot follow the instructions in
Backing Up and Restoring VMware Tanzu SQL with MySQL for VMs because you do not have the
GUID for the service instance nor the timestamp for the backup.
The instructions on this section decribe how to find these things and then download the backup
artifact.
1. Find the GUID of the service instance by searching the broker logs. Try searching the
broker log for entries about deleted deployments, such as:
or
In the log entries example, the GUID of the service instance begins with 6c1db.
For information about broker logs, see Access Broker and Instance Logs and VMs.
2. Log in to your backup storage system. Your backup storage system is whatever you
configured in Configuring Automated Backups. For example, it might be an S3 bucket or a
file system on a backup host (SCP).
3. Identify and download the backup artifact, the tarfile. The name of the tarfile contains the
GUID and an epoch timestamp. For example, 6c1db434-29ef-47c4-9f22-
59fe53676b07_1598049440.tar.
4. Record the backup ID, which is the name of the TAR file without the .tar extension. In the
example, the backup ID is 6c1db434-29ef-47c4-9f22-59fe53676b07_1598049440.
5. Untar the backup artifact. Keep the backup file with the tar.gpg extension.
[on-demand-service-broker] 2020/08/21 23:48:04.821405 Request DELETE /v
2/service_instances/6c1db434-29ef-47c4-9f22-59fe53676b07 Completed 202
in 545.989368ms | Start Time: 2020/08/21 23:48:04.275382
[on-demand-service-broker] [8db9f496-e83f-4aa6-82f0-528ddbff4c0a] 2020/
08/21 23:53:09.978376 BOSH task ID 348 status: done delete deployment f
or instance 6c1db434-29ef-47c4-9f22-59fe53676b07: Description: delete d
eployment service-instance_6c1db434-29ef-47c4-9f22-59fe53676b07 Result:
/deployments/service-instance_6c1db434-29ef-47c4-9f22-59fe53676b07
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
112
The file with the .txt extension contains metadata to help identify the backup, but this text
file is not needed for the restore procedure.
If Restoring to a Different Foundation
If you have a backup of a service instance taken from one foundation, and you want to restore the
backup to a different foundation, you cannot follow the instructions in Backing Up and Restoring
VMware Tanzu SQL with MySQL for VMs.
Instead, follow these instructions to identify and download the backup artifact. These instructions
are more straightforward than If Restoring a Backup from a Deleted Service Instance because you
have access to the backed up service instance and therefore its GUID and a timestamp for the
backup.
1. Find the GUID of the service instance by running:
cf service SERVICE-INSTANCE-NAME --guid
Where SERVICE-INSTANCE-NAME is the service instance from which the backup was taken.
For example:
2. List the backups available and determine the one you want to download:
cf adbr list-backups SERVICE-INSTANCE-NAME
Where SERVICE-INSTANCE-NAME is the service instance from which the backup was taken.
For example:
3. If the service instance is running, download the backup tarfile. Your backup storage system
is whatever you configured in Configuring Automated Backups. For example, it might be an
S3 bucket or a file system on a backup host (SCP).
4. Untar the backup artifact. Keep the backup file with the tar.gpg extension.
The file with the .txt extension contains metadata to help identify the backup, but this text
file is not needed for the restore procedure.
Retrieve Backup Encryption Key
Each backup artifact has its own encryption key, stored in runtime CredHub.
To retrieve the backup encryption key:
$ cf service my-instance --guid
6c1db434-29ef-47c4-9f22-59fe53676b07
$ cf adbr list-backups my-instance
Getting backups of service instance my-instance in org…
Backup ID Time of Backup
6c1db434-29ef-47c4-9f22-59fe53676b07_1598049440 Fri Aug 21 22:37:20 U
TC 2020
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
113
1. To find the GUID for the MySQL service broker VM, run:
bosh deployments
2. To SSH onto the broker VM, run:
bosh -d pivotal-mysql-GUID ssh
Where is GUID is the GUID you recorded in step 1.
For example:
3. Fetch the credentials to authenticate with the TAS for VMs CredHub:
1. Navigate to the Ops Manager Installation Dashboard.
2. In the TAS for VMs tile, click the Credentials tab.
3. Record the credentials for Credhub Admin Client Client Credentials.
4. Set the path for the CredHub CLI and authenticate with runtime CredHub:
export PATH=/var/vcap/packages/credhub-cli/bin:$PATH
credhub api https://credhub.service.cf.internal:8844 --ca-cert /var/vcap/jobs/a
dbr-api/config/credhub_ca.pem
credhub login --client-name CREDHUB-CLIENT --client-secret CREDHUB-CLIENT-SECRE
T
Where:
CREDHUB-CLIENT is the identity of the credential
CREDHUB-CLIENT-SECRET is the password of the credential
For example:
5. To obtain the encryption key for the backup, run:
credhub get -n /tanzu-mysql/backups/BACKUP-ID
Where is BACKUP-ID is the name of the backup artifact you identified.
For example:
$ bosh -d pivotal-mysql-f7f3ce3767943537c36a ssh
$ export PATH=/var/vcap/packages/credhub-cli/bin:$PATH
$ credhub api https://credhub.service.cf.internal:8844 --ca-cert /var/v
cap/jobs/adbr-api/config/credhub_ca.pem
$ credhub login --client-name credhub_admin_client --client-secret o2i3
0fj2fjvjoi3j
$ credhub get -n /tanzu-mysql/backups/6c1db434-29ef-47c4-9f22-59fe53676
b07_1598049440
id: b918cda9-0c8b-4011-bbba-f78bdb5ceea4
name: /tanzu-mysql/backups/6c1db434-29ef-47c4-9f22-59fe53676b07_1598049
440
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
114
6. Record the value of the password. This is the backup encryption key that need when you
restore the backup.
In this example, it is NWjbvbB3pjOC. Use this key when you restore the backup.
Create and Prepare a New Service Instance for Restore
You can only restore single node and leader-follower backup artifacts to a single node service
instance. Ensure that the persistent disk in the single node plan is at least as large as the persistent
disk of your largest leader-follower. For information about persistent disk sizing recommendations,
see Persistent Disk Usage.
To prepare a new service instance for restore:
1. Create a new MySQL service instance by running:
cf create-service p.mysql NEW-PLAN NEW-INSTANCE-NAME
Where:
NEW-PLAN is the name of the service plan for your new service instance. The plan you
choose depends on the service instance topology that you are restoring. If the
topology that you are restoring is:
Single node or leader-follower: Select a single node plan.
Multi‑Site Replication or HA cluster: Select a Multi‑Site Replication plan.
NEW-INSTANCE-NAME is the name of the new service instance.
For more information, see Create a Service Instance.
2. Monitor the status of the service instance creation by running:
watch cf service NEW-INSTANCE-NAME
Where NEW-INSTANCE-NAME is the name of the new service instance.
3. Locate and record the GUID associated with your new service instance by running:
cf service NEW-INSTANCE-NAME --guid
4. From the Ops Manager VM, find and record the BOSH instance GUID for your service
instance by running:
bosh -e BOSH-ENVIRONMENT -d service-instance_GUID instances
Where GUID is the service instance GUID you previously recorded.
For example:
type: password
value: NWjbvbB3pjOC
version_created_at: "2020-08-21T22:37:20Z"
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
115
The BOSH instance GUID is the value after mysql/.
5. Copy the downloaded backup to the new service instance by running:
bosh -e BOSH-ENVIRONMENT -d service-instance_GUID \
scp mysql-backup-STRING.tar.gpg \
mysql/BOSH-INSTANCE-GUID:DESTINATION-PATH
Where:
GUID is the service instance GUID.
mysql-backup-STRING.tar.gpg is the backup file you downloaded in Identify and
Download the Backup Artifact.
BOSH-INSTANCE-GUID is the BOSH instance GUID you recorded in the previous step.
DESTINATION-PATH is where the backup file saves on the BOSH VM.
For example:
Restore the Service Instance
You can restore a single node, leader-follower, HA cluster service, or Multi‑Site Replication
instance using the restore utility. The restore utility is packaged with the VMware Tanzu SQL with
MySQL for VMs tile.
The restore utility does the following:
Deletes any existing data
Decrypts and unzips the backup artifact
Restores the backup artifact into the MySQL data directory
$ bosh -e my-env -d service-instance_12345678-90ab-cdef-1234-567890abcd
ef instances
Deployment 'service-instance_12345678-90ab-cdef-1234-567890abcdef'
Instance Process State AZ
IPs
mysql/d7ff8d46-c3e8-449f-aea7-5a05b0a1929c running us-central1
-a 10.0.8.10
1 instances
$ bosh -e my-env -d service-instance_12345678-90ab-cdef-1234-567890abcd
ef \
scp mysql-backup-1595910545-b42f275a-c355-49c3-9e33-bda6fcf2ffd0.ta
r.gpg \
mysql/d7ff8d46-c3e8-449f-aea7-5a05b0a1929c:/tmp/
Warning: Restoring a service instance is destructive. VMware recommends that
you only restore to a new and unused service instance.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
116
To restore a service instance:
1. Use the BOSH CLI to SSH in to the newly created MySQL service instance by following the
procedure in BOSH SSH.
2. After securely logging in to MySQL, become root by running:
sudo su
3. Restore the backup artifact into the data directory by running:
mysql-restore --encryption-key ENCRYPTION-KEY --restore-file RESTORE-FILE-PATH
Where:
ENCRYPTION-KEY is the backup encryption key you recorded in Retrieve Backup
Encryption Key.
RESTORE-FILE-PATH is the full path on the BOSH VM where the backup artifact
exists.
For example:
Restage the Service Instance
After you restore your single node, leader-follower, HA cluster service instance, or Multi‑Site
Replication you must restage your new service instance. For Multi‑Site Replication plans, you must
also re-establish replication between the leader and follower service instances.
To restage your service instance:
1. If you are restoring a leader-follower service instance, update the plan by running:
cf update-service NEW-INSTANCE-NAME -p LEADER-FOLLOWER-PLAN
2. If you are restoring a HA cluster service instance, update the plan by running:
cf update-service NEW-INSTANCE-NAME -p HA-CLUSTER-PLAN
3. If you are restoring a Multi‑Site Replication service instance, you must re-establish
replication:
1. Create a follower Multi‑Site Replication service instance by following the procedure
in Create Multi‑Site Replication Service Instances. Ensure that you only create a
follower service instance in the secondary foundation.
2. Configure replication between the leader service instance you restored and the
follower instance by following the procedure in Configure Multi‑Site Replication.
4. Determine if the app is currently bound to a MySQL service instance by running:
$ mysql-restore --encryption-key NWjbvbB3pjOC --restore-file /tmp/mysql
-backup-1595910545-b42f275a-c355-49c3-9e33-bda6fcf2ffd0.tar.gpg
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
117
cf services
5. If the previous step shows that the app is currently bound to a MySQL instance, unbind it
by running:
cf unbind-service APP-NAME OLD-INSTANCE-NAME
6. Update your app to bind to the new service instance by running:
cf bind-service APP-NAME NEW-INSTANCE-NAME
7. Restage your app to make the changes take effect by running:
cf restage APP-NAME
Your app must be running and able to access the restored data.
Accessing a Database as an Admin User
This topic describes how to access a database as an admin user with either CredHub credentials or
BOSH SSH.
Overview
You can access a database as an admin user to do actions that cannot be done as a normal binding
user.
You can do the following actions as an admin user:
Add users
Create new schemas
View system schemas
You can choose to access your database service instance as an admin, in one of the following ways:
Using BOSH SSH: If your BOSH agent is healthy, you can BOSH SSH into the MySQL VM.
This option can be faster. See Connect to MySQL with BOSH SSH below.
Using CredHub Credentials: If your BOSH agent is unhealthy, you can use this option. See
Connect to MySQL with CredHub Credentials below.
Connect to MySQL with BOSH SSH
To connect to MySQL with BOSH SSH:
Warning: If a developer rebinds an app to the Tanzu SQL for VMs service after
unbinding, they must also rebind any existing custom schemas to the app. When
you rebind an app, stored code, programs, and triggers break. For more information
about binding custom schemas, see Use Custom Schemas.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
118
1. BOSH SSH into your node by following the procedure in BOSH SSH in the Ops Manager
documentation.
2. Connect to your MySQL VM by running:
mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf
Connect to MySQL with CredHub Credentials
To retrieve the admin credentials for a service instance from BOSH CredHub:
1. Use the cf CLI to determine the GUID associated with the service instance for which you
want to retrieve credentials by running:
cf service SERVICE-INSTANCE-NAME --guid
For example:
If you do not know the name of the service instance, you can list service instances in the
space with cf services.
2. Follow the steps in Gather Credential and IP Address Information and Log In to the Ops
Manager VM with SSH of
Advanced Troubleshooting with the BOSH CLI
to SSH into the
Ops Manager VM.
3. From the Ops Manager VM, log in to your BOSH Director with the BOSH CLI. See
Authenticate with the BOSH Director VM in
Advanced Troubleshooting with the BOSH CLI
.
4. Find the values for BOSH_CLIENT and BOSH_CLIENT_SECRET:
1. In the Ops Manager Installation Dashboard, click the BOSH Director tile.
2. Click the Credentials tab.
3. In the BOSH Director section, click the link to the BOSH Commandline
Credentials .
4. Record the values for BOSH_CLIENT and BOSH_CLIENT_SECRET.
5. Set the API target of the CredHub CLI to your BOSH CredHub server by running:
credhub api https://BOSH-DIRECTOR-IP:8844 \
--ca-cert=/var/tempest/workspaces/default/root_ca_certificate
Where BOSH-DIRECTOR-IP is the IP address of the BOSH Director VM.
For example:
6. Log in to CredHub by running:
$ cf service my-service-instance --guid
12345678-90ab-cdef-1234-567890abcdef
$ credhub api https://10.0.0.5:8844 \
--ca-cert=/var/tempest/workspaces/default/root_ca_certificate
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
119
credhub login \
--client-name=BOSH-CLIENT \
--client-secret=BOSH-CLIENT-SECRET
For example:
7. Use the CredHub CLI to retrieve the credentials by doing one of following :
Retrieve the password for the admin user by running:
credhub get -n /p-bosh/service-instance_GUID/admin_password
In the output, the password appears under value. Record the password.
For example:
Retrieve the password for the read-only admin user by running:
credhub get -n /p-bosh/service-instance_GUID/read_only_admin_password
In the output, the password appears under value. Record the password.
8. Record the IP of your service instance. See Connect Using an IP Address.
9. Connect to your database by doing one of the following:
Connect using a management tool. See Using Management Tools for Tanzu SQL for
VMs.
Connect directly from your workstation using the MySQL client by running:
When prompted for a password, enter the password you recorded.
Rotating Certificates
This topic describes how to check expiration dates and rotate certificates used by VMware Tanzu
SQL with MySQL for VMs.
$ credhub login \
--client-name=credhub \
--client-secret=abcdefghijklm123456789
$ credhub get \
-n /p-bosh/service-instance_70d30bb6-7f30-441a-a87c-05a5e4afff2
6/admin_password
id: d6e5bd10-3b60-4a1a-9e01-c76da688b847
name: /p-bosh/service-instance_70d30bb6-7f30-441a-a87c-05a5e4af
ff26/admin_password
type: password
value: UMF2DXsqNPPlCNWMdVMcNv7RC3Wi10
version_created_at: 2018-04-02T23:16:09Z
mysql -h IP-ADDRESS -u admin -P 3306 -p
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
120
Rotate Services TLS Certificate Authority
To rotate the Services TLS CA and its leaf certificates, use one of the following procedures:
Ops Manager v2.10: See Rotating the Services TLS CA and Its Leaf Certificates.
Ops Manager v2.9: See Rotating the Services TLS CA and Its Leaf Certificates.
Ops Manager v2.8: See Rotating the Services TLS CA and Its Leaf Certificates.
Ops Manager v2.7: See Rotating the Services TLS CA and Its Leaf Certificates.
Ops Manager v2.9 and later is compatible with CredHub Maestro. Tanzu SQL for VMs v2.8 and later
is compatible with CredHub Maestro.
Certificates Used by Tanzu SQL for VMs
If you are using Ops Manager v2.9 or later, you can rotate all MySQL certificates in the following
table using CredHub Maestro. For Ops Manager v2.9 and earlier, you can rotate the Services TLS
CA using a manual procedure.
For more information about procedures to use to rotate certificates, see Rotate Services TLS
Certificate Authority.
The following table lists the certificates used by Tanzu SQL for VMs:
Certificate Rotated by Tanzu SQL for
VMs?
/services/tls_ca No
/opsmgr/pivotal-mysql-GUID/adbr_api_cert No
/p-bosh/pivotal-mysql-GUID/agent_ca_2_9_x No
/p-bosh/pivotal-mysql-GUID/agent_client_ssl_2_9_x No
/p-bosh/pivotal-mysql-GUID/agent_server_ssl_2_9_x No
/p-bosh/pivotal-mysql-GUID/services_tls_accessor_cert No
/p-bosh/service-instance_GUID/adbr_agent_cert No
/p-bosh/service-instance_GUID/agent_ca No
/p-bosh/service-instance_GUID/agent_client_tls No
/p-bosh/service-instance_GUID/agent_server_tls No
/p-bosh/service-instance_GUID/mysql_server_tls No
/p-bosh/service-instance_GUID/pxc_internal_ca No
/p-bosh/service-instance_GUID/pxc_tls_ca No
/p-bosh/service-instance_GUID/pxc_tls_server No
/p-bosh/service-instance_GUID/restore_ca No
/p-bosh/service-instance_GUID/restore_client_tls No
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
121
/p-bosh/service-instance_GUID/restore_server_tls No
/p-bosh/service-instance_GUID/streaming_backup_ca Yes
/p-bosh/service-instance_GUID/streaming_backup_server_cert Yes
In the previous table, GUID is the GUID for the service instance. To find the GUID for your service
instance, follow the procedure in Find Information about Your Service Instance.
If you are using a PXC-type database, Tanzu SQL for VMs rotates the Galera certificate by
renaming it.
Resolving Service Interruptions
This topic explains events in the lifecycle of a VMware Tanzu SQL with MySQL for VMs service
instance that might cause temporary service interruptions.
Stemcell or Service Update
An operator can update a stemcell version or their version of Tanzu SQL for VMs.
Impact: Apps lose access to the MySQL service while Ops Manager updates the service
instance they are bound to. The service resumes within 10-15 minutes.
Required Actions: None. If the update deploys successfully, the apps automatically
reconnect.
Plan Change
A developer can change their service instance to provide a different service plan, using cf update-
service or Apps Manager.
Impact: Apps lose access to the MySQL service while Ops Manager updates the service
instance they are bound to. The service resumes within 10-15 minutes.
Required Actions: None. If the plan change deploys successfully, apps reconnect
automatically.
VM Process Failure
A process, like the MySQL server, fails on the service instance VM.
Impact:
BOSH (monit) brings the process back automatically.
Depending on the process and what it was doing, the service can experience 60-
120 seconds of downtime.
Until the process resumes, apps might be unable to use MySQL, metrics or logging
can stop, and other features might be interrupted.
Required Actions: None. If the process resumes cleanly and without manual intervention,
apps reconnect automatically.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
122
VM Failure
A Tanzu SQL for VMs VM fails and goes offline due to either a virtualization problem or a host
hardware problem.
Impact:
If the BOSH Resurrector is enabled (recommended), BOSH can detect the failure,
recreate the VM, and reattach the same persistent disk and IP address.
Downtime largely depends on how quickly the Resurrector notices, usually 1-2
minutes, and how long it takes the IaaS to create a replacement VM.
If the Resurrector is not enabled, some IaaSes, for example, vSphere, have similar
resurrection or HA features.
Apps cannot connect to MySQL until the VM is recreated and the My SQL server
process is resumed.
Based on prior experience with BOSH Resurrector, typical downtime is 8-10
minutes.
Required Actions:
If the VM is part of a leader-follower pair, when the VM comes back, it is read-only.
Therefore, run the configure-leader-follower errand to ensure the leader VM is
writable. For more information, see configure-leader-follower in
Running Errands
.
If the VM is not part of a leader-follower pair, when the VM comes back, no further
action is required for the app developer to continue operations.
AZ Failure
An availability zone (AZ) goes offline entirely or loses connectivity to other AZs (net split). This
causes service interruption in multi-AZ deployments where Diego has placed multiple instances of a
MySQL-using app in different AZs.
Impact:
Some app instances can still be able to connect and continue operating.
App instances in the other AZs are not able to connect.
Downtime: Unknown
Required Actions: Recovery of the app - database connection must be automatic.
Depending on the app, manual intervention might be required to check data consistency.
Region Failure
Example: An entire region fails, bringing VMware Tanzu Application Service for VMs
components offline.
Impact:
The entire installation needs to be brought back up manually.
Downtime: Unknown
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
123
Required Actions: Each service instance might need to be restored individually depending
on the restored state of the platform.
Running Errands
This topic describes each of the service broker errands and how to run an errand using the BOSH
CLI.
Overview
Errands can manage service brokers and run mass operations on service instances created by
brokers. To run an errand, see Run an Errand.
For all supported VMware Tanzu SQL with MySQL for VMs errands, see the full errand list:
find-deprecated-bindings
smoke-tests
configure-leader-follower
make-leader
make-read-only
upgrade-all-service-instances
register-broker
delete-all-service-instances-and-deregister-broker
recreate-all-service-instances
Run an Errand
To run an errand:
1. View the BOSH deployment name for your MySQL service broker by running:
bosh deployments
2. Run:
bosh -d pivotal-mysql-GUID run-errand ERRAND-NAME
Where:
pivotal-mysql-GUID is the BOSH deployment name for your MySQL service broker.
ERRAND-NAME is the name of the errand you want to run.
For example:
$ bosh -d pivotal-mysql-e3ddd36247fe5b923caf run-errand find-deprecated
-bindings
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
124
Available Errands
The following sections describe each service broker errand that you can run. To run an errand, see
Run an Errand.
find-deprecated-bindings
The find-deprecated-bindings errand does the following:
Lists app bindings and services keys that are deprecated in Tanzu SQL for VMs v2.9. The
bindings are deprecated because they do not require TLS.
Exits whether or not a deprecated binding is found.
The find-deprecated-bindings errand has the following example output:
VMware recommends that operators configure bindings to require TLS. For more information, see
Configure TLS.
smoke-tests
The smoke-tests errand does the following:
Validates that the service broker has been installed and configured correctly.
Creates and deletes a new space and service instance that conducts tests.
If this errand runs successfully, Tanzu SQL for VMs has installed successfully.
Stdout +---------------------------+-------------------------------------
-+------------------------+--------------------------+--------------------+--
-----------------+-----------------------------+
| SERVICE | SERVICE GUI
| ORG | SPACE | APP OR SERVICE KEY |
TYPE | REASON |
+---------------------------+-------------------------------------
-+------------------------+--------------------------+--------------------+--
-----------------+-----------------------------+
| tlsDB | a999db0b-176e-4ac8-8342-d72b338d1f0c
| MYSQL-ORG-upgrade-test | MYSQL-SPACE-upgrade-test | user-cli | Se
rviceKeyBinding | no tls |
| tlsDB | a999db0b-176e-4ac8-8342-d72b338d1f0c
| MYSQL-ORG-upgrade-test | MYSQL-SPACE-upgrade-test | user-cli | Se
rviceKeyBinding | no dns: hostname="10.0.8.6" |
| upgrade-outdated-instance | 34f26746-fb46-4f14-87bc-e1ddce26f340
| MYSQL-ORG-upgrade-test | MYSQL-SPACE-upgrade-test | cs-accept | Ap
pBinding | no dns: hostname="10.0.8.5" |
| tlsDB | a999db0b-176e-4ac8-8342-d72b338d1f0c
| MYSQL-ORG-upgrade-test | MYSQL-SPACE-upgrade-test | cs-accept-tls | Ap
pBinding | no dns: hostname="10.0.8.6" |
+---------------------------+-------------------------------------
-+------------------------+--------------------------+--------------------+--
-----------------+-----------------------------+
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
125
configure-leader-follower
The configure-leader-follower errand does the following:
Configures replication on the follower and ensures the leader is writable.
Runs after every create or update of a leader-follower instance.
Fails and alerts operators with BOSH errand output if the service instance is in a unhealthy
state.
This errand is used to trigger a leader-follower failover. You can use this errand to create custom
failover scripts. For more information, see Triggering a Leader-Follower Failover.
make-leader
The make-leader errand does the following:
Promotes a follower VM to a leader.
Removes replication configuration from the VM, waits for all transactions to be applied to
the VM, and sets the VM as writable.
Fails if the original leader is still writeable. This protects against data divergence.
This errand is used to trigger a leader-follower failover. You can use this errand to create custom
failover scripts. For more information, see Triggering a Leader-Follower Failover.
make-read-only
The make-read-only errand does the following:
Demotes a leader VM to a follower.
Sets the VM to read-only and guarantees that apps no longer write to this VM.
Relays all in-flight transactions on the former leader VM to the follower VM if the follower
VM is accessible.
This errand is used to trigger a leader-follower failover. You can use this errand to create custom
failover scripts. For more information, see Triggering a Leader-Follower Failover.
upgrade-all-service-instances
The upgrade-all-service-instances errand does the following:
Collects all of the service instances that the on-demand broker has registered.
Issues an upgrade command and deploys the a new manifest to the on-demand broker for
each service instance.
Adds to a retry list any instances that have ongoing BOSH tasks at the time of upgrade.
Retries any instances in the retry list until all instances are upgraded.
When you make changes to the plan configuration, the errand upgrades all the Tanzu SQL for VMs
service instances to the latest version of the plan.
If any instance fails to upgrade, the errand fails immediately. This prevents systemic problems from
spreading to the rest of your service instances.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
126
register-broker
The register-broker errand does the following:
Registers the service broker with Cloud Controller.
Enables service access for any plans that are enabled on the tile.
Disables service access for any plans that are disabled on the tile.
Does nothing for any plans that are set to manual on the tile.
You should run this errand whenever the broker is re-deployed with new catalog metadata to
update the Marketplace.
Plans with disabled service access are only visible to admin Cloud Foundry users. Non-admin Cloud
Foundry users, including Org Managers and Space Managers, cannot see these plans.
delete-all-service-instances-and-deregister-broker
The delete-all-service-instances-and-deregister-broker errand does the following:
Deactivates service access to the service offering for all orgs and spaces. The errand
deactivates service access to ensure that new instances cannot be provisioned during the
lifetime of the errand.
Unbinds all apps from the service instances.
Runs any pre-delete errands for each instance.
Deletes the BOSH deployment of each service instance.
Checks for deletion failure of each instance, which results in the errand failing immediately.
Determines whether any instances have been created while the errand was running. If new
instances are detected, the errand returns an error. In this case, VMware recommends
running the errand again.
Deregisters the broker from the Cloud Controller.
Ops Manager runs this errand only when deleting Tanzu SQL for VMs. Running this errand removes
all service instances and their data.
recreate-all-service-instances
The recreate-all-service-instances errand recreates all service instance VMs managed by an on-
demand broker.
You might want use this errand in the following cases:
Rotating the Ops Manager root certificate authority (CA). For more information about
rotating CAs, see Rotating CAs and Leaf Certificates.
Fully restoring the platform during disaster recovery or migration.
Warning: This errand destroys all on-demand service instances and deregisters the
broker from the Cloud Controller. Use it with extreme caution.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
127
Troubleshooting VMware Tanzu SQL with MySQL for VMs
Learn about the basic instructions for troubleshooting on-demand VMware Tanzu SQL with MySQL
for VMs.
This topic provides operators with basic instructions for troubleshooting on-demand VMware Tanzu
SQL with MySQL for VMs. For information about temporary Tanzu SQL for VMs service
interruptions, see Service Interruptions.
Troubleshoot Errors
This section provides information on how to troubleshoot specific errors or error messages.
Common Services Errors
The following errors occur in multiple services:
Failed Installation
Cannot Create or Delete Service Instances
Broker Request Timeouts
Instance Does Not Exist
Cannot Bind to or Unbind from Service Instances
Cannot Connect to a Service Instance
Upgrade All Service Instances Errand Fails
Missing Logs and Metrics
Failed Installation
Symptom Tanzu SQL for VMs fails to install.
Cause Reasons for a failed installation include:
Certificate issues: The on-demand broker (ODB) requires valid certificates.
Deploy fails. This could be due to a variety of reasons.
Networking problems:
Cloud Foundry cannot reach the Tanzu SQL for VMs broker
Cloud Foundry cannot reach the service instances
The service network cannot access the BOSH director
The Register broker errand fails.
The smoke test errand fails.
Resource sizing issues: These occur when the resource sizes selected for a given plan
are less than Tanzu SQL for VMs requires to function.
Other service-specific issues.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
128
Solution
To troubleshoot:
Certificate issues: Ensure that your certificates are valid and generate new ones if
necessary. To generate new certificates, contact Support.
Deploy fails: View the logs using Ops Manager to determine why the deploy is failing.
Networking problems: For how to troubleshoot, see Networking problems.
Register broker errand fails: For how to troubleshoot, see Register broker errand.
Resource sizing issues: Check your resource configuration in Ops Manager and ensure
that the configuration matches that recommended by the service.
Cannot Create or Delete Service Instances
Symptom If developers report errors such as:
Cause Reasons include:
Problems with the deployment manifest
Authentication errors
Network errors
Quota errors
Solution
To troubleshoot:
1. If the BOSH error shows a problem with the deployment manifest, open the manifest
in a text editor to inspect it.
2. To continue troubleshooting, Log in to BOSH and target the Tanzu SQL for VMs
instance using the instructions on parsing a Cloud Foundry error message.
3. Retrieve the BOSH task ID from the error message and run the following command:
bosh task TASK-ID
4. If you need more information, access the broker logs and use the broker-request-id
from the previous error message to search the logs for more information. Search for:
Authentication errors
Network errors
Quota errors
Broker Request Timeouts
Instance provisioning failed: There was a problem completing
your request. Please contact your operations team providing
the following information: service: redis-acceptance, servic
e-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broke
r-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id:
442, operation: create
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
129
Symptom If developers report errors such as:
Cause Cloud Foundry might not be connected to the service broker, or there might be a large number
of queued tasks.
Solution
To troubleshoot:
1. Confirm that Cloud Foundry (CF) is connected to the service broker.
2. Check the BOSH queue size:
1. Log in to BOSH as an admin.
2. Run bosh tasks
If there are a large number of queued tasks, the system might be under too much
load. BOSH is configured with two workers and one status worker, which might not
be sufficient resources for the level of load.
3. If the task queue is long, advise the app developers to try again once the system is
under less of a load.
Instance Does Not Exist
Symptom If developers report errors such as:
Cause The instance might have been deleted.
Solution
To troubleshoot:
1. Confirm that the Tanzu SQL for VMs instance exists in BOSH and obtain the GUID CF
by running:
cf service MY-INSTANCE --guid
2. Using the GUID that you obtained previously, run:
bosh -d service-instance_GUID vms
If the BOSH deployment is not found, it has been deleted from BOSH. Contact Support for
further assistance.
Cannot Bind to or Unbind from Service Instances
Server error, status code: 504, error code: 10001, message:
The request to the service broker timed out: https://BROKER-
URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13
f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b
Server error, status code: 502, error code: 10001, message:
Service broker error: instance does not exist
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
130
Symptom If developers report errors such as:
Cause This might be due to authentication or network errors.
Solution
To find out the exact issue with the binding process:
1. Access the service broker logs.
2. Search the logs for the broker-request-id string listed in the error message.
3. Check for:
Authentication errors
Network errors
4. Contact Support for further assistance if you are unable to resolve the problem.
Cannot Connect to a Service Instance
Symptom Developers report that their app cannot use service instances that they have successfully
created and bound.
Cause The error might originate from the service or be network related.
Solution
To solve this issue, ask the user to send application logs that show the connection error. If the
error originates from the service, then follow Tanzu SQL for VMs-specific instructions. If the
issue appears to be network-related, then:
1. Check that application security groups are configured correctly. Access can be
configured for the service network that the tile is deployed to.
2. Ensure that the network the TAS for VMs tile is deployed to has network access to the
service network. You can find the network definition for this service network in the
BOSH Director tile.
3. In Ops Manager go into the service tile and see the service network that is configured
in the **Networks** tab.
4. In Ops Manager go into the TAS for VMs tile and see the network it is assigned to.
Make sure that these networks can access each other.
Upgrade All Service Instances Errand Fails
Symptom The upgrade-all-service-instances errand fails.
Server error, status code: 502, error code: 10001, message:
Service broker error: There was a problem completing your re
quest. Please contact your operations team providing the fol
lowing information: service: example-service, service-instan
ce-guid: 8d69de6c-88c6-4283-b8bc-1c46103714e2, broker-reques
t-id: 15f4f87e-200a-4b1a-b76c-1c4b6597c2e1, operation: bind
Note: Service instances can also become temporarily inaccessible during upgrades
and VM or network failures. See Service Interruptions for more information.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
131
Cause There might be a problem with a particular instance.
Solution
To troubleshoot:
1. Look at the errand output in the Tanzu Operations Manager log.
2. If an instance has failed to upgrade, debug and fix it before running the errand again to
prevent any failure issues from spreading to other on-demand instances.
3. After the Tanzu Operations Manager log no longer lists the deployment as failing,
run the errand again to upgrade the rest of the instances.
Missing Logs and Metrics
Symptom No logs are being emitted by the on-demand broker.
Cause Syslog might not be configured correctly, or you might have network access issues.
Solution
To troubleshoot:
1. Ensure you have configured syslog for the tile.
2. Check that your syslog forwarding address is correct in Ops Manager.
3. Ensure that you have network connectivity between the networks that the tile is using
and the syslog destination. If the destination is external, you need to use the public ip
VM extension feature available in your Ops Manager tile configuration settings.
4. Verify that Loggregator is emitting metrics:
1. Install the cf log-cache plug-in. For instructions, see the Log Cache CLI
Plug-in GitHub repository.
2. Find the GUID for your service instance by running:
cf service SERVICE-INSTANCE --guid
3. Find logs from your service instance by running:
cf log-stream | grep "SERVICE-GUID"
4. If no metrics appear within five minutes, verify that the broker network has
access to the Loggregator system on all required ports.
5. If you are unable to resolve the issue, contact Support.
Leader-Follower Service Instance Errors
This section provides solutions for the following errands:
Unable to Determine Leader and Follower
Both Leader and Follower Instances Are Writable
Both Leader and Follower Instances Are Read-Only
Unable to Determine Leader and Follower
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
132
Symptom This problem happens when the configure-leader-follower errand fails because it cannot
determine the VM roles.
The configure-leader-follower errand exits with 1 and the errand logs contain the following:
Cause Something has happened to the instances, such as a failure or manual intervention. As a result,
there is not enough information available to determine the correct state and topology without
operator intervention to resolve the issue.
$ Unable to determine leader and follower based on transacti
on history.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
133
Solution
Use the inspect errand to determine which instance can be the leader. Then, using the
orchestration errands and backup/restore, you can put the service instance into a safe
topology, and then rerun the configure-leader-follower errand. This is shown in the
following example.
This example shows one outcome that the inspect errand can return:
1. Use the inspect errand to retrieve relevant information about the two VMs:
In the previous scenario, the first instance is missing data but does not have
replication configured. The second instance has data, and also has replication
configured. The following instructions resolve this by copying data to the first
instance, and resuming replication.
2. Take a backup of the second instance using the Create a Tanzu SQL for VMs Logical
Backup steps.
3. Restore the backup artifact to the first instance using the Restore from a Tanzu SQL for
VMs Logical Backup steps.
At this point, the instances have equivalent data.
4. Run the configure-leader-follower errand to reconfigure replication:
$ bosh -e my-env -d my-dep run-errand inspect
[...]
Instance mysql/4ecad54b-0704-47eb-8eef-eb228ca
b9724
Exit Code 0
Stdout -
Stderr 2017/12/11 18:25:54 Started executing c
ommand: inspect
2017/12/11 18:25:54 Started GET http
s://127.0.0.1:8443/status
2017/12/11 18:25:54
Has Data: false
Read Only: true
GTID Executed: 1d774323-de9e-11e7-be01-
42010a001014:1-25
Replication Configured: false
Instance mysql/e0b94ade-0114-4d49-a929-ce1616d
8beda
Exit Code 0
Stdout -
Stderr 2017/12/11 18:25:54 Started executing c
ommand: inspect
2017/12/11 18:25:54
Started GET https://127.0.0.1:8443/stat
us
2017/12/11 18:25:54
Has Data: true
Read Only: true
GTID Executed: 1d774323-de9e-11e7-be01-
42010a001014:1-25
Replication Configured: true
2 errand(s)
Succeeded
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
134
For example:
Both Leader and Follower Instances Are Writable
Symptom This problem happens when the configure-leader-follower errand fails because both VMs
are writable and the VMs might hold differing data.
The configure–leader-follower errand exits with 1 and the errand logs contain the following:
Cause Tanzu SQL for VMs tries to ensure that there is only one writable instance of the leader-follower
pair at any given time. However, in certain situations, such as network partitions, or manual
intervention outside of the provided bosh errands, it is possible for both instances to be
writable.
The service instances remain in this state until an operator resolves the issue to ensure that the
correct instance is promoted and reduce the potential for data divergence.
bosh -e ENVIRONMENT -d DEPLOYMENT \
run-errand configure-leader-follower \
--instance=mysql/GUID-OF-LEADER
$ bosh -e my-env -d my-dep \
run-errand configure-leader-follower \
--instance=mysql/4ecad54b-0704-47eb-8eef-eb2
28cab9724
$ Both mysql instances are writable. Please ensure no diverg
ent data and set one instance to read-only mode.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
135
Solution
1. Use the inspect errand to retrieve the GTID Executed set for each VM:
If the GTID Executed sets for both instances are the same, continue to Step 2. If they
are different, continue to Step 4.
2. Look at the value of GTID Executed for both instances.
If the range after the GUID is equivalent, either instance can be made read-
only, as described in Step 3.
If one instance has a range that is a subset of the other, the instance with the
subset must be made read-only, as described in Step 3.
3. Based on the information you gathered in the previous step, run the make-read-only
errand to make the appropriate instance read-only:
$ bosh -e my-env -d my-dep run-errand inspect
[...]
Instance mysql/4ecad54b-0704-47eb-8eef-eb228
cab9724
Exit Code 0
Stdout -
Stderr 2017/12/11 18:25:54 Started executi
ng command: inspect
2017/12/11 18:25:54 Started GET http
s:127.0.0.1:8443/status
2017/12/11 18:25:54
Has Data: true
Read Only: false
GTID Executed: 1d774323-de9e-11e7-be0
1-42010a001014:1-23
Replication Configured: false
Instance mysql/e0b94ade-0114-4d49-a929-ce161
6d8beda
Exit Code 0
Stdout -
Stderr 2017/12/11 18:25:54 Started executi
ng command: inspect
2017/12/11 18:25:54 Started GET http
s:127.0.0.1:8443/status
2017/12/11 18:25:54
Has Data: true
Read Only: false
GTID Executed: 1d774323-de9e-11e7-be0
1-42010a001014:1-25
Replication Configured: false
2 errand(s)
Succeeded
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
136
bosh -e ENVIRONMENT -d DEPLOYMENT \
run-errand make-read-only \
--instance=mysql/MYSQL-SUBSET-INSTANCE
For example:
4. If the GTID Executed sets are neither equivalent nor subsets, data has diverged and
you must determine what data has diverged as part of the procedure:
1. Use the make-read-only errand to set both instances to read-only to
prevent further data divergence.
bosh -e ENVIRONMENT -d DEPLOYMENT \
run-errand make-read-only \
--instance=mysql/MYSQL-INSTANCE
For example:
2. Take a backup of both instances using the Create a Tanzu SQL for VMs
Logical Backup steps.
3. Manually inspect the data on each instance to determine the discrepancies
and put the data on the instance that is further ahead—this instance has the
higher GTID Executed set, and is the new leader.
4. Migrate all appropriate data to the new leader instance.
5. After putting all data on the leader, ssh onto the follower:
bosh -e ENVIRONMENT -d DEPLOYMENT ssh mysql/GUID-OF-F
OLLOWER
For example:
6. Become root with the command sudo su.
7. Stop the mysql process with the command monit stop mysql.
8. Delete the data directory of the follower with the command rm -rf
/var/vcap/store/mysql.
9. Start the mysql process with the command monit start mysql.
$ bosh -e my-env -d my-dep \
run-errand make-read-only \
--instance=mysql/e0b94ade-0114-4d49-a929
-ce1616d8beda
[...]
succeeded
$ bosh -e my-env -d my-dep \
run-errand make-read-only \
--instance=mysql/e0b94ade-011
4-4d49-a929-ce1616d8beda
[...]
succeeded
$ bosh -e my-env -d my-dep ssh mysql/e0b94ade-0
114-4d49-a929-ce1616d8beda
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
137
10. Use the configure-leader-follower errand to copy the leader data to the
follower and resume replication:
bosh -e ENVIRONMENT -d DEPLOYMENT \
run-errand configure-leader-follower
\
--instance=mysql/GUID-OF-LEADER
For example:
Both Leader and Follower Instances Are Read-Only
Symptom Developers report that apps cannot write to the database. In a leader-follower topology, the
leader VM is writable and the follower VM is read-only. However, if both VMs are read-only,
apps cannot write to the database.
Cause This problem happens if the leader VM fails and the BOSH Resurrector is activated. When the
leader is resurrected, it is set as read-only.
$ bosh -e my-env -d my-dep \
run-errand configure-leader-fol
lower \
--instance=mysql/4ecad54b-0704-
47eb-8eef-eb228cab9724
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
138
Solution
1. Use the inspect errand to confirm that both VMs are in a read-only state:
2. Examine the output and locate the information about the leader-follower Tanzu SQL
for VMs VMs:
3. If Read Only is set to true for both VMs, make the leader writable using the following
command:
bosh -e ENVIRONMENT -d DEPLOYMENT run-errand inspect
Instance mysql/4eexample54b-0704-47eb-8eef-eb2ex
ample724
Exit Code 0
Stdout -
Stderr 2017/12/11 18:25:54 Started executing c
ommand: inspect
2017/12/11 18:25:54 Started GET https:99
9.0.0.1:8443/status
2017/12/11 18:25:54
Has Data: true
Read Only: true
GTID Executed: 1d779999-de9e-11e7-be01-42
010a009999:1-23
Replication Configured: true
Instance mysql/e0exampleade-0114-4d49-a929-cexam
ple8beda
Exit Code 0
Stdout -
Stderr 2017/12/11 18:25:54 Started executing c
ommand: inspect
2017/12/11 18:25:54 Started GET https:99
9.0.0.1:8443/status
2017/12/11 18:25:54
Has Data: true
Read Only: true
GTID Executed: 1d779999-de9e-11e7-be01-42
010a009999:1-25
Replication Configured: false
2 errand(s)
Succeeded
bosh -e ENVIRONMENT -d DEPLOYMENT \
run-errand configure-leader-follower \
--instance=mysql/GUID-OF-LEADER
For example, if the second instance is the lea
der:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
139
Inoperable App and Database Errors
This section provides a solution for the following errors:
Persistent Disk is Full
Cannot Access Database Table
Persistent Disk is Full
Symptom Developers report that read, write, and cf CLI operations do not work. Developers cannot
upgrade to a larger Tanzu SQL for VMs service plan to free up disk space.
If your persistent disk is full, apps become inoperable. In this state, read, write, and Cloud
Foundry Command-Line Interface (cf CLI) operations do not work.
Cause This problem happens if your persistent disk is full. When you use the BOSH CLI to target your
deployment, you see that instances are at 100% persistent disk usage.
Available disk space can be increased by deleting log files. After deleting logs, you can then
upgrade to a larger Tanzu SQL for VMs service plan.
You can also turn off binary logging before developers do large data uploads or if their
databases have a high transaction volume.
Solution
To resolve this issue, do one of the following:
If your persistent disk is already full, delete binary logs. See MySQL for PCF hangs
when server VM persistent disk is full in the Knowledge Base.
If the majority of your persistent disk are binary logs but it is not currently full, turn
off binary logging. See Binary Logs Filling up the Persistent Disk in the Knowledge
Base.
Cannot Access Database Table
$ bosh -e my-env -d my-dep \
run-errand configure-leader-follower \
--instance=mysql/e0exampleade-0114-4d49-a9
29-cexample8beda
Warning: Deleting binary logs is a destructive procedure and can
result in MySQL data loss. Only do this procedure with the
assistance of Support.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
140
Symptom When you query an existing table, you see an error similar to the following:
Cause
This error occurs if you created an uppercase table name and then activated lowercase table
names.
You activate lowercase table names either by:
Setting the optional enable_lower_case_table_names parameter to true with the cf
CLI. For more information about the parameter, see Lowercase Table Names.
Selecting Enable Lower Case Table Names in the Mysql Configuration pane of the
tile. For more information about this configuration, see Configure MySQL.
Solution
To resolve this issue:
1. Deactivate lowercase table names by doing one of the following:
Set the optional enable_lower_case_table_names parameter to false with
the cf CLI. For instructions, see Set Optional Parameters.
Activate lowercase table names in the tile:
1. Deselect Enable Lower Case Table Names in the Mysql
Configuration pane of the tile.
2. Navigate to the Ops Manager Installation Dashboard, click Review
Pending Changes, and then click Apply Changes.
2. (Optional) If you want to activate lowercase table names again, rename your table to
lowercase and then activate lowercase table names.
Highly Available Cluster Errors
This section provides solutions for the following errands:
Unresponsive Node in a Highly Available Cluster
Many Replication Errors in Logs for Highly Available Clusters
Unresponsive Node in a Highly Available Cluster
Symptom A client connected to a Tanzu SQL for VMs cluster node reports the following error:
Some clients might instead return the following:
Cause If the client is connected to a Tanzu SQL for VMs cluster node and that node loses connection
to the rest of the cluster, the node stops accepting writes. If the connection to this node is
made through the proxy, the proxy automatically re-routes further connections to a different
node.
ERROR 1146 (42S02): Table 'mysql.foobar' doesn't exist
WSREP has not yet prepared this node for application use
unknown error
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
141
Solution
A node can become unresponsive for a number of reasons. For solutions, see the following:
Network Latency: If network latency causes a node to become unresponsive, the
node drops but eventually rejoins. The node automatically rejoins only if one node has
left the cluster. Consult your IaaS network settings to reduce your network latency.
MySQL Process Failure: If the MySQL process crashes, monit and BOSH restores the
process. If the process is not restored, run the download-logs tool and consult the
error logs it generates. For more information, see the download-logs section.
Firewall Rule Change: If your firewall rules change, it might prevent a node from
communicating with the rest of the cluster. This causes the node to become
unresponsive. In this case, the logs show the node leaving the cluster but do not show
network latency errors.
To confirm that the node is unresponsive because of a firewall rule change, SSH from
a responsive node to the unresponsive node. If you cannot connect, the node is
unresponsive due to a firewall rule change. Change your firewall rules to activate the
unresponsive node to rejoin the cluster.
VM Failure: If you cannot SSH into a node and you are not detecting either network
latency or firewall issues, your node might be down due to VM failure. To confirm that
the node is unresponsive and recreate the VM, see Recreate a Corrupted VM in a
Highly Available.
Node Unable to Rejoin: If a detached existing node fails to join the cluster, its
sequence_number might be higher than those of the nodes with quorum. A higher
sequence_number on the detached node indicates that it has recent changes to the
data that the primary component lacks. You can verify this by looking at the node’s
error log at /var/vcap/sys/log/pxc-mysql/mysql.err.log.
To restore the cluster, complete one of the following:
Network Latency: If network latency causes a node to become
unresponsive, the node drops but eventually rejoins. The node automatically
rejoins only if one node has left the cluster. Consult your IaaS network
settings to reduce your network latency.
MySQL Process Failure: If the MySQL process crashes, monit and BOSH
restores the process. If the process is not restored, run the download-logs
tool and consult the error logs it generates. For more information, see the
download-logs section below.
If bootstrapping does not restore the cluster, you can manually force the
node to rejoin the cluster. This removes all of the unsynchronized data from
the detached server node and creates a new copy of the cluster data on the
node. For more information, see Force a Node to Rejoin a Highly Available
Cluster Manually.
Many Replication Errors in Logs for Highly Available Clusters
Warning: Forcing a node to rejoin the cluster is a
destructive procedure. Only do this procedure with the
assistance of Support.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
142
Symptom You see many replication errors in the MySQL logs, like the following:
Cause This problem happens when there are errors in SQL statements.
Solution
For solutions for replication errors in MySQL log files, see the following table:
Additional Error Solution
ALTER TABLE errors Fix the ALTER TABLE error.
This error can occur when an app issues an invalid data definition
statement. Other nodes log this problem as a replication error
because they fail to replicate the ALTER TABLE.
Increased persistent disk
usage or running out of
working memory
Decode the GRA_*.log files and look for errors. See How to
decode Galera GRA log files in the Support knowledge base. The
GRA log files contain failing DDL statements.
If you see replication errors, but no ALTER TABLE or persistent disk or memory issues, you can
ignore the replication errors.
Failed Backups
If an automated backup or a backup initiated from the ApplicationDataBackupRestore (adbr) plug-in
fails, verify that the 2345 port from the TAS for VMs to the ODB component is open.
Automated Backups or adbr Plug-in Backups Fail
160318 9:25:16 [Warning] WSREP: RBR event 1 Query apply
warning: 1, 16992456
160318 9:25:16 [Warning] WSREP: Ignoring error for TO is
olated action: source: abcd1234-abcd-1234-abcd-1234abcd1234
version: 3 local: 0 state: APPLYING flags: 65 conn_id: 24680
4 trx_id: -1 seqnos (l: 865022, g: 16992456, s: 16992455, d:
16992455, ts: 2530660989030983)
160318 9:25:16 [ERROR] Slave SQL: Error 'Duplicate colum
n name 'number'' on query. Default database: 'cf_0123456_123
4_abcd_1234_abcd1234abcd'. Query: 'ALTER TABLE ...'
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
143
Symptom The following are true:
The backup fails.
The adbr-api logs for the broker show:
The gorouter logs on the TAS for VMs deployment show:
Where SYSTEM-DOMAIN is your system domain.
Cause Port 2345 that allows communication between the TAS for VMs and ODB components is
closed.
Solution
Open port 2345 from the TAS for VMs component to the ODB component. See Required
Networking Rules for Tanzu SQL for VMs in
On-Demand Networking
.
Troubleshoot Components
This section provides guidance on checking for and fixing issues in on-demand service components.
BOSH Problems
Large BOSH Queue
On-demand service brokers add tasks to the BOSH request queue, which can back up and cause
delay under heavy loads. An app developer who requests a new Tanzu SQL for VMs instance sees
create in progress in the Cloud Foundry Command Line Interface (cf CLI) until BOSH processes
the queued request.
Ops Manager currently deploys two BOSH workers to process its queue. Future versions of Ops
Manager will let users configure the number of BOSH workers.
On-demand service brokers add tasks to the BOSH request queue, which can back up and cause
delay under heavy loads. An app developer who requests a new Tanzu SQL for VMs instance sees
create in progress in the Cloud Foundry Command Line Interface (cf CLI) until BOSH processes
the queued request.
Ops Manager currently deploys two BOSH workers to process its' queue. Users of future versions
of Ops Manager can configure the number of BOSH workers.
Configuration
Service Instances in Failing State
Configuration
backup failed with response: 502 Bad Gateway: Register
ed endpoint failed to handle the request.
adbr-api.SYSTEM-DOMAIN - [2021-01-20T19:30:00.91108027
1Z] "POST /service_instances/acb85c98-151e-4f13-9f0f-d
e057ef18d67/backup HTTP/1.1" 502 ... x_cf_routererro
r:"endpoint_failure" ...
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
144
Service Instances in Failing State
The VM or Disk type that you configured in the plan page of the tile in Ops Manager might not be
large enough for the Tanzu SQL for VMs service instance to start. See tile-specific guidance on
resource requirements.
Authentication
UAA Changes
If you have rotated any UAA user credentials then you may see authentication issues in the service
broker logs.
To resolve this, redeploy the Tanzu SQL for VMs tile in Ops Manager. This provides the broker with
the latest configuration.
If you have rotated any UAA user credentials then you might see authentication issues in the
service broker logs.
To resolve this, redeploy the Tanzu SQL for VMs tile in Ops Manager. This provides the broker with
the latest configuration.
Networking
Common issues with networking include:
Issue Solution
Latency when connecting to the Tanzu SQL for VMs
service instance to create or delete a binding.
Try again or improve network performance.
Firewall rules are blocking connections from the Tanzu
SQL for VMs service broker to the service instance.
Open the Tanzu SQL for VMs tile in Ops Manager and
check the two networks configured in the Networks pane.
Ensure that these networks allow access to each other.
Firewall rules are blocking connections from the service
network to the BOSH director network.
Ensure that service instances can access the Director so
that the BOSH agents can report in.
Apps cannot access the service network. Configure Cloud Foundry application security groups to
allow runtime access to the service network.
Problems accessing BOSH’s UAA or the BOSH director. Follow network troubleshooting and check that the BOSH
director is online
Note: You must ensure that any changes to UAA credentials are reflected in the
Ops Manager credentials tab of the VMware Tanzu Application Service for VMs
tile.
Note: You must ensure that any changes to UAA credentials are reflected in the
Ops Manager credentials tab of the VMware Tanzu Application Service for VMs
tile.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
145
Validate Service Broker Connectivity to Service Instances
To validate connectivity, do the following:
1. View the BOSH deployment name for your service broker by running:
bosh deployments
2. SSH into the Tanzu SQL for VMs service broker by running:
bosh -d DEPLOYMENT-NAME ssh
3. If no BOSH task-id appears in the error message, look in the broker log using the broker-
request-id from the task.
Networking
Common issues with networking include:
Issue Solution
Latency when connecting to the Tanzu SQL for VMs
service instance to create or delete a binding.
Try again or improve network performance.
Firewall rules are blocking connections from the Tanzu
SQL for VMs service broker to the service instance.
Open the Tanzu SQL for VMs tile in Ops Manager and
check the two networks configured in the Networks pane.
Ensure that these networks allow access to each other.
Firewall rules are blocking connections from the service
network to the BOSH director network.
Ensure that service instances can access the Director so
that the BOSH agents can report in.
Apps cannot access the service network. Configure Cloud Foundry application security groups to
allow runtime access to the service network.
Problems accessing BOSH’s UAA or the BOSH director. Follow network troubleshooting and check that the BOSH
director is online
Validate Service Broker Connectivity to Service Instances
To determine whether there is an issue with the Tanzu SQL for VMs deployment:
1. Inspect the VMs by running:
bosh -d service-instance_GUID vms --vitals
2. For additional information, run:
bosh -d service-instance_GUID instances --ps --vitals
If the VM is failing, follow the service-specific information. Any unadvised corrective actions (such
as running BOSH restart on a VM) can cause issues in the service instance.
A failing process or failing VM might come back automatically after a temporary service outage. See
VM Process Failure and VM Failure.
To validate connectivity, do the following:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
146
1. View the BOSH deployment name for your service broker by running:
bosh deployments
2. SSH into the Tanzu SQL for VMs service broker by running:
bosh -d DEPLOYMENT-NAME ssh
3. If no BOSH task-id appears in the error message, look in the broker log using the broker-
request-id from the task.
Validate App Access to Service Instance
Use cf ssh to access to the app container, then try connecting to the Tanzu SQL for VMs service
instance using the binding included in the VCAP_SERVICES environment variable.
Quotas
Plan Quota Issues
If developers report errors such as:
1. Check your current plan quota.
2. Increase the plan quota.
3. Log in to Ops Manager.
4. Reconfigure the quota on the plan page.
5. Deploy the tile.
6. Find who is using the plan quota and take the appropriate action.
Global Quota Issues
If developers report errors such as:
1. Check your current global quota.
2. Increase the global quota.
3. Log in to Ops Manager.
4. Reconfigure the quota on the on-demand settings page.
5. Deploy the tile.
6. Find out who is using the quota and take the appropriate action.
Message: Service broker error: The quota for this service plan has been excee
ded.
Please contact your Operator for help.
Message: Service broker error: The quota for this service has been exceeded.
Please contact your Operator for help.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
147
Failing Jobs and Unhealthy Instances
To determine whether there is an issue with the Tanzu SQL for VMs deployment:
1. Inspect the VMs by running:
bosh -d service-instance_GUID vms --vitals
2. For additional information, run:
bosh -d service-instance_GUID instances --ps --vitals
If the VM is failing, follow the service-specific information. Any unadvised corrective actions (such
as running BOSH restart on a VM) can cause issues in the service instance.
A failing process or failing VM might come back automatically after a temporary service outage. See
VM Process Failure and VM Failure.
AZ or Region Failure
Failures at the IaaS level, such as Availability Zone (AZ) or region failures, can interrupt service and
require manual restoration. See AZ Failure and Region Failure.
Techniques for Troubleshooting
Instructions on interacting with the on-demand service broker and on-demand service instance
BOSH deployments, and on performing general maintenance and housekeeping tasks
Parse a Cloud Foundry (CF) Error Message
Failed operations (create, update, bind, unbind, delete) result in an error message. You can retrieve
the error message later by running the cf CLI command cf service INSTANCE-NAME.
$ cf service myservice
Service instance: myservice
Service: super-db
Bound apps:
Tags:
Plan: dedicated-vm
Description: Dedicated Instance
Documentation url:
Dashboard:
Last Operation
Status: create failed
Message: Instance provisioning failed: There was a problem completing your re
quest.
Please contact your operations team providing the following information:
service: redis-acceptance,
service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089,
broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac,
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
148
Use the information in the Message field to debug further. Provide this information to Support when
filing a ticket.
The task-id field maps to the BOSH task ID. For more information on a failed BOSH task, use the
bosh task TASK-ID.
The broker-request-guid maps to the portion of the On-Demand Broker log containing the failed
step. Access the broker log through your syslog aggregator, or access BOSH logs for the broker by
typing bosh logs broker 0. If you have more than one broker instance, repeat this process for
each instance.
Access Broker and Instance Logs and VMs
Before following the procedures below, log in to the cf CLI and the BOSH CLI.
Access Broker Logs and VMs
You can access logs using Ops Manager by clicking on the Logs tab in the tile and downloading the
broker logs.
To access logs using the BOSH CLI, do the following:
1. Identify the on-demand broker (ODB) deployment by running the following command:
bosh deployments
2. View VMs in the deployment by running the following command:
bosh -d DEPLOYMENT-NAME instances
3. SSH onto the VM by running the following command:
bosh -d DEPLOYMENT-NAME ssh
4. Download the broker logs by running the following command:
bosh -d DEPLOYMENT-NAME logs
The archive generated by BOSH includes the following logs:
Log Name Description
broker.stdo
ut.log
Requests to the on-demand broker and the actions the broker performs while orchestrating the request
(e.g. generating a manifest and calling BOSH). Start here when troubleshooting.
bpm.log Control script logs for starting and stopping the on-demand broker.
post-
start.stderr.l
og
Errors that occur during post-start verification.
task-id: 442,
operation: create
Started: 2017-03-13T10:16:55Z
Updated: 2017-03-13T10:17:58Z
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
149
post-
start.stdout.
log
Post-start verification.
drain.stderr.
log
Errors that occur while running the drain script.
Access Service Instance Logs and VMs
1. To target an individual service instance deployment, retrieve the GUID of your service
instance with the following cf CLI command:
cf service MY-SERVICE --guid
2. To view VMs in the deployment, run the following command:
bosh -d service-instance_GUID instances
3. To SSH into a VM, run the following command:
bosh -d service-instance_GUID ssh
4. To download the instance logs, run the following command:
bosh -d service-instance_GUID logs
Run Service Broker Errands to Manage Brokers and Instances
From the BOSH CLI, you can run service broker errands that manage the service brokers and
perform mass operations on the service instances that the brokers created. These service broker
errands include:
register-broker registers a broker with the Cloud Controller and lists it in the Marketplace.
deregister-broker deregisters a broker with the Cloud Controller and removes it from the
Marketplace.
upgrade-all-service-instances upgrades existing instances of a service to its latest
installed version.
delete-all-service-instances deletes all instances of service.
orphan-deployments detects "orphan" instances that are running on BOSH but not
registered with the Cloud Controller.
To run an errand, run the following command:
bosh -d DEPLOYMENT-NAME run-errand ERRAND-NAME
For example:
bosh -d my-deployment run-errand deregister-broker
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
150
Register Broker
The register-broker errand does the following:
Registers the service broker with Cloud Controller.
Enables service access for any plans that are enabled on the tile.
Disables service access for any plans that are disabled on the tile.
Does nothing for any plans that are set to manual on the tile.
You should run this errand whenever the broker is re-deployed with new catalog metadata to
update the Marketplace.
Plans with disabled service access are only visible to admin Cloud Foundry users. Non-admin Cloud
Foundry users, including Org Managers and Space Managers, cannot see these plans.
Deregister Broker
This errand deregisters a broker from Cloud Foundry.
The errand does the following:
Deletes the service broker from Cloud Controller
Fails if there are any service instances, with or without bindings
Use the Delete All Service Instances errand to delete any existing service instances.
To run the errand, run the following command:
bosh -d DEPLOYMENT-NAME run-errand deregister-broker
Upgrade All Service Instances
The upgrade-all-service-instances errand does the following:
Collects all of the service instances that the on-demand broker has registered.
Issues an upgrade command and deploys the a new manifest to the on-demand broker for
each service instance.
Adds to a retry list any instances that have ongoing BOSH tasks at the time of upgrade.
Retries any instances in the retry list until all instances are upgraded.
When you make changes to the plan configuration, the errand upgrades all the Tanzu SQL for VMs
service instances to the latest version of the plan.
If any instance fails to upgrade, the errand fails immediately. This prevents systemic problems from
spreading to the rest of your service instances.
Delete All Service Instances
This errand uses the Cloud Controller API to delete all instances of your broker’s service offering in
every Cloud Foundry org and space. It only deletes instances the Cloud Controller knows about. It
does not delete orphan BOSH deployments.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
151
The delete-all-service-instances errand does the following:
1. Unbinds all apps from the service instances.
2. Deletes all service instances sequentially. Each service instance deletion includes:
1. Running any pre-delete errands
2. Deleting the BOSH deployment of the service instance
3. Removing any ODB-managed secrets from BOSH CredHub
4. Checking for instance deletion failure, which results in the errand failing immediately
3. Determines whether any instances have been created while the errand was running. If new
instances are detected, the errand returns an error. In this case, VMware recommends
running the errand again.
To run the errand, run the following command:
bosh -d service-instance_GUID delete-deployment
Detect Orphaned Service Instances
A service instance is defined as "orphaned" when the BOSH deployment for the instance is still
running, but the service is no longer registered in Cloud Foundry.
The orphan-deployments errand collates a list of service deployments that have no matching service
instances in Cloud Foundry and return the list to the operator. It is then up to the operator to
remove the orphaned BOSH deployments.
To run the errand, run the following command:
bosh -d DEPLOYMENT-NAME run-errand orphan-deployments
If orphan deployments exist---The errand script does the following:
Exit with exit code 10
Output a list of deployment names under a [stdout] header
Provide a detailed error message under a [stderr] header
For example:
Note: Orphan BOSH deployments do not correspond to a known service instance.
While rare, orphan deployments can occur. Use the orphan-deployments errand to
identify them.
Warning: Use extreme caution when running this errand. You should only use it
when you want to totally destroy all of the on-demand service instances in an
environment.
[stdout]
[{"deployment\_name":"service-instance\_80e3c5a7-80be-49f0-8512-44840f3c4d1
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
152
These details will also be available through the BOSH /tasks/ API endpoint for use in scripting:
If no orphan deployments exist---The errand script does the following:
Exit with exit code 0
Stdout will be an empty list of deployments
Stderr will be None
If the errand encounters an error during running---The errand script does the following:
Exit with exit 1
Stdout will be empty
Any error messages will be under stderr
To clean up orphaned instances, run the following command on each instance:
b"}]
[stderr]
Orphan BOSH deployments detected with no corresponding service instance in Cl
oud Foundry. Before deleting any deployment it is recommended to verify the s
ervice instance no longer exists in Cloud Foundry and any data is safe to del
ete.
Errand 'orphan-deployments' completed with error (exit code 10)
$ curl 'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?t
ype=result' | jq .
{
"exit_code": 10,
"stdout": "[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44
840f3c4d1b"}]\n",
"stderr": "Orphan BOSH deployments detected with no corresponding service i
nstance in Cloud Foundry. Before deleting any deployment it is recommended to
verify the service instance no longer exists in Cloud Foundry and any data is
safe to delete.\n",
"logs": {
"blobstore_id": "d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d"
}
}
[stdout]
[]
[stderr]
None
Errand 'orphan-deployments' completed successfully (exit code 0)
WARNING: Running this command may leave IaaS resources in an unusable state.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
153
bosh delete-deployment service-instance_SERVICE-INSTANCE-GUID
View Resource Saturation and Scaling
To view usage statistics for any service, do the following:
1. Run the following command:
bosh -d DEPLOYMENT-NAME vms --vitals
2. To view process-level information, run:
bosh -d DEPLOYMENT-NAME instances --ps
Identify Apps using a Service Instance
To identify which apps are using a specific service instance from the name of the BOSH
deployment:
1. Take the deployment name and strip the service-instance_ leaving you with the GUID.
2. Log in to CF as an admin.
3. Obtain a list of all service bindings by running the following:
cf curl /v2/service_instances/GUID/service_bindings
4. The output from the above curl gives you a list of resources, with each item referencing a
service binding, which contains the APP-URL. To find the name, org, and space for the app,
run the following:
1. cf curl APP-URL and record the app name under entity.name.
2. cf curl SPACE-URL to obtain the space, using the entity.space_url from the above
curl. Record the space name under entity.name.
3. cf curl ORGANIZATION-URL to obtain the org, using the entity.organization_url
from the above curl. Record the organization name under entity.name.
Monitor Quota Saturation and Service Instance Count
Quota saturation and total number of service instances are available through ODB metrics emitted
to Loggregator. The metric names are shown below:
Metric Name Description
on-demand-broker/SERVICE-NAME-MARKETPLACE/quota_remaining global quota remaining for all instances across
all plans
Note: When running cf curl ensure that you query all pages, because the
responses are limited to a certain number of bindings per page. The default is 50.
To find the next page curl the value under next_url.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
154
Metric Name Description
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-
NAME/quota_remaining
quota remaining for a particular plan
on-demand-broker/SERVICE-NAME-MARKETPLACE/total_instances total instances created across all plans
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-
NAME/total_instances
total instances created for a given plan
Techniques for Troubleshooting Highly Available Clusters
If your cluster is experiencing downtime or in a degraded state, VMware recommends gathering
information to diagnose the type of failure the cluster is experiencing with the following workflow:
See AZ Failure and Region Failure.
Techniques for Troubleshooting
Instructions on interacting with the on-demand service broker and on-demand service instance
BOSH deployments, and on performing general maintenance and housekeeping tasks
Parse a Cloud Foundry (CF) Error Message
Failed operations (create, update, bind, unbind, delete) result in an error message. You can retrieve
the error message later by running the cf CLI command cf service INSTANCE-NAME.
$ cf service myservice
Service instance: myservice
Service: super-db
Bound apps:
Tags:
Plan: dedicated-vm
Description: Dedicated Instance
Documentation url:
Dashboard:
Last Operation
Status: create failed
Message: Instance provisioning failed: There was a problem completing your request.
Please contact your operations team providing the following information:
service: redis-acceptance,
service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089,
broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac,
task-id: 442,
operation: create
Started: 2017-03-13T10:16:55Z
Updated: 2017-03-13T10:17:58Z
Use the information in the Message field to debug further. Provide this information to Support when
filing a ticket.
Note: Quota metrics are not emitted if no quota has been set.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
155
The task-id field maps to the BOSH task ID. For more information on a failed BOSH task, use the
bosh task TASK-ID.
The broker-request-guid maps to the portion of the On-Demand Broker log containing the failed
step. Access the broker log through your syslog aggregator, or access BOSH logs for the broker by
typing bosh logs broker 0. If you have more than one broker instance, repeat this process for
each instance.
Access Broker and Instance Logs and VMs
Before following these procedures, log in to the cf CLI and the BOSH CLI.
Access Broker Logs and VMs
You can access logs using Tanzu Operations Manager by clicking on the Logs tab in the tile and
downloading the broker logs.
To access logs using the BOSH CLI, do the following:
1. Identify the on-demand broker (ODB) deployment by running the following command:
bosh deployments
2. View VMs in the deployment by running the following command:
bosh -d DEPLOYMENT-NAME instances
3. SSH onto the VM by running the following command:
bosh -d DEPLOYMENT-NAME ssh
4. Download the broker logs by running the following command:
bosh -d DEPLOYMENT-NAME logs
The archive generated by BOSH includes the following logs:
Log Name Description
broker.stdo
ut.log
Requests to the on-demand broker and the actions the broker performs while orchestrating the request
(e.g. generating a manifest and calling BOSH). Start here when troubleshooting.
bpm.log Control script logs for starting and stopping the on-demand broker.
post-
start.stderr.l
og
Errors that occur during post-start verification.
post-
start.stdout.
log
Post-start verification.
drain.stderr.
log
Errors that occur while running the drain script.
Access Service Instance Logs and VMs
1. To target an individual service instance deployment, retrieve the GUID of your service
instance with the following cf CLI command:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
156
cf service MY-SERVICE --guid
2. To view VMs in the deployment, run the following command:
bosh -d service-instance_GUID instances
3. To SSH into a VM, run the following command:
bosh -d service-instance_GUID ssh
4. To download the instance logs, run the following command:
bosh -d service-instance_GUID logs
Run Service Broker Errands to Manage Brokers and Instances
From the BOSH CLI, you can run service broker errands that manage the service brokers and
perform mass operations on the service instances that the brokers created. These service broker
errands include:
register-broker registers a broker with the Cloud Controller and lists it in the Marketplace.
deregister-broker deregisters a broker with the Cloud Controller and removes it from the
Marketplace.
upgrade-all-service-instances upgrades existing instances of a service to its latest
installed version.
delete-all-service-instances deletes all instances of service.
orphan-deployments detects “orphan” instances that are running on BOSH but not
registered with the Cloud Controller.
To run an errand, run the following command:
bosh -d DEPLOYMENT-NAME run-errand ERRAND-NAME
For example:
Register Broker
The register-broker errand does the following:
Registers the service broker with Cloud Controller.
Activates service access for any plans that are activated on the tile.
Deactivates service access for any plans that are deactivated on the tile.
Does nothing for any plans that are set to manual on the tile.
You can run this errand whenever the broker is re-deployed with new catalog metadata to update
the Marketplace.
Plans with deactivated service access are only visible to admin Cloud Foundry users. Non-admin
Cloud Foundry users, including Org Managers and Space Managers, cannot see these plans.
Deregister Broker
bosh -d my-deployment run-errand deregister-broker
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
157
This errand deregisters a broker from Cloud Foundry.
The errand does the following:
Deletes the service broker from Cloud Controller
Fails if there are any service instances, with or without bindings
Use the Delete All Service Instances errand to delete any existing service instances.
To run the errand, run the following command:
bosh -d DEPLOYMENT-NAME run-errand deregister-broker
Upgrade All Service Instances
The upgrade-all-service-instances errand does the following:
Collects all of the service instances that the on-demand broker has registered.
Issues an upgrade command and deploys the a new manifest to the on-demand broker for
each service instance.
Adds to a retry list any instances that have ongoing BOSH tasks at the time of upgrade.
Retries any instances in the retry list until all instances are upgraded.
When you make changes to the plan configuration, the errand upgrades all the Tanzu SQL for VMs
service instances to the latest version of the plan.
If any instance fails to upgrade, the errand fails immediately. This prevents systemic problems from
spreading to the rest of your service instances.
Delete All Service Instances
This errand uses the Cloud Controller API to delete all instances of your broker’s service offering in
every Cloud Foundry org and space. It only deletes instances the Cloud Controller knows about. It
does not delete orphan BOSH deployments.
The delete-all-service-instances errand does the following:
1. Unbinds all apps from the service instances.
2. Deletes all service instances sequentially. Each service instance deletion includes:
1. Running any pre-delete errands
2. Deleting the BOSH deployment of the service instance
3. Removing any ODB-managed secrets from BOSH CredHub
4. Checking for instance deletion failure, which results in the errand failing immediately
3. Determines whether any instances have been created while the errand was running. If new
instances are detected, the errand returns an error. In this case, VMware recommends
Note: Orphan BOSH deployments do not correspond to a known service instance.
While rare, orphan deployments can occur. Use the orphan-deployments errand to
identify them.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
158
running the errand again.
To run the errand, run the following command:
bosh -d service-instance_GUID delete-deployment
Detect Orphaned Service Instances
A service instance is defined as “orphaned” when the BOSH deployment for the instance is still
running, but the service is no longer registered in Cloud Foundry.
The orphan-deployments errand collates a list of service deployments that have no matching service
instances in Cloud Foundry and return the list to the operator. It is then up to the operator to
remove the orphaned BOSH deployments.
To run the errand, run the following command:
bosh -d DEPLOYMENT-NAME run-errand orphan-deployments
If orphan deployments exist—The errand script does the following:
Exit with exit code 10
Output a list of deployment names under a [stdout] header
Provide a detailed error message under a [stderr] header
For example:
These details are also available through the BOSH /tasks/ API endpoint for use in scripting:
Warning: Use extreme caution when running this errand. You can only use it when
you want to totally destroy all of the on-demand service instances in an
environment.
[stdout]
[{"deployment\_name":"service-instance\_80e3c5a7-80be-49f0-8512-44840f3c4d1
b"}]
[stderr]
Orphan BOSH deployments detected with no corresponding service instance in Cl
oud Foundry. Before deleting any deployment it is recommended to verify the s
ervice instance no longer exists in Cloud Foundry and any data is safe to del
ete.
Errand 'orphan-deployments' completed with error (exit code 10)
$ curl 'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?t
ype=result' | jq .
{
"exit_code": 10,
"stdout": "[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44
840f3c4d1b"}]\n",
"stderr": "Orphan BOSH deployments detected with no corresponding service i
nstance in Cloud Foundry. Before deleting any deployment it is recommended to
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
159
If no orphan deployments exist—The errand script does the following:
Exit with exit code 0
Stdout is an empty list of deployments
Stderr is None
If the errand encounters an error during running—The errand script does the following:
Exit with exit 1
Stdout is empty
Any error messages are under stderr
To clean up orphaned instances, run the following command on each instance:
bosh delete-deployment service-instance_SERVICE-INSTANCE-GUID
Reinstall a Tile
To reinstall the Tanzu SQL for VMs tile, see Reinstalling MySQL for Pivotal Cloud Foundry version 2
and higher in the Support Knowledge Base.
View Resource Saturation and Scaling
To view usage statistics for any service, do the following:
1. Run the following command:
bosh -d DEPLOYMENT-NAME vms --vitals
2. To view process-level information, run:
bosh -d DEPLOYMENT-NAME instances --ps
Identify Apps using a Service Instance
verify the service instance no longer exists in Cloud Foundry and any data is
safe to delete.\n",
"logs": {
"blobstore_id": "d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d"
}
}
[stdout]
[]
[stderr]
None
Errand 'orphan-deployments' completed successfully (exit code 0)
WARNING: Running this command might leave IaaS resources in an unusable state.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
160
To identify which apps are using a specific service instance from the name of the BOSH
deployment:
1. Take the deployment name and strip the service-instance_ leaving you with the GUID.
2. Log in to CF as an admin.
3. Obtain a list of all service bindings by running the following: cf curl
/v2/service_instances/GUID/service_bindings
4. The output from the curl gives you a list of resources, with each item referencing a service
binding, which contains the APP-URL. To find the name, org, and space for the app, run the
following:
1. cf curl APP-URL and record the app name under entity.name.
2. cf curl SPACE-URL to obtain the space, using the entity.space_url from the curl.
Record the space name under entity.name.
3. cf curl ORGANIZATION-URL to obtain the org, using the entity.organization_url
from the curl. Record the organization name under entity.name.
Monitor Quota Saturation and Service Instance Count
Quota saturation and total number of service instances are available through ODB metrics emitted
to Loggregator. The metric names are shown in the following table::
Metric Name Description
on-demand-broker/SERVICE-NAME-MARKETPLACE/quota_remaining global quota remaining for all instances across
all plans
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-
NAME/quota_remaining
quota remaining for a particular plan
on-demand-broker/SERVICE-NAME-MARKETPLACE/total_instances total instances created across all plans
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-
NAME/total_instances
total instances created for a given plan
Techniques for Troubleshooting Highly Available Clusters
If your cluster is experiencing downtime or in a degraded state, VMware recommends gathering
information to diagnose the type of failure the cluster is experiencing with the following workflow:
1. Consult solutions for common errors. See Highly Available Cluster Troubleshooting Errors.
2. Use mysql-diag to view a summary of the network, disk, and replication state of each
cluster node. Depending on the output from mysql-diag, you might recover your cluster
Note: When running cf curl ensure that you query all pages, because the
responses are limited to a certain number of bindings per page. The default is 50.
To find the next page curl the value under next_url.
Note: Quota metrics are not emitted if no quota has been set.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
161
with the following troubleshooting techniques:
To force a node to rejoin the cluster, see Force a Node to Rejoin a Highly Available
Cluster Manually.
To re-create a corrupted VM, see Re-create a Corrupted VM in a Highly Available
Cluster.
To check if replication is working, see Check Replication in a Highly Available
Cluster.
For more information about mysql-diag, see Running mysql-diag.
3. Run download-logs against each node in your Tanzu SQL for VMs cluster, proxies, and
jumpbox VM. You must run download-logs before attempting recovery because any failures
in the recovery procedure can result in logs being lost or made inaccessible.
For more information, see the download-logs section.
4. If you are uncertain about the recovery steps to take, submit a ticket through Support.
When you submit a ticket provide the following information:
mysql-diag output: A summary of the network, disk, and replication state. The
Running mysql-diag topic explains how to run mysql-diag.
download-logs logs: Logs from your Tanzu SQL for VMs cluster, proxies, and
jumpbox VM. The download-logs section explains how to run download-logs.
Deployment environment: The environment that Tanzu SQL for VMs is running in
such as VMware Tanzu Application Service for VMs or a service tile.
Version numbers: The versions of the installed Ops Manager, TAS for VMs, and
Tanzu SQL for VMs.
Force a Node to Rejoin a Highly Available Cluster Manually
If a detached node fails to rejoin the cluster after a configured grace period, you can manually force
the node to rejoin the cluster. This procedure removes all the data on the node, forces the node to
join the cluster, and creates a new copy of the cluster data on the node.
Note: VMware recommends that you use the -X flag to get the complete
set of available logs. However, if your cluster processes a high volume of
transactions, the complete set might be too large and you can omit this flag
to fetch the essential set of logs.
Warning: Do not attempt to resolve cluster issues by reconfiguring the cluster,
such as changing the number of nodes or networks. Only follow the diagnosis steps
in this document. If you are unsure how to proceed, contact Support.
Warning: If you manually force a node to rejoin the cluster, data stored on the local
node is lost. Do not force nodes to rejoin the cluster if you want to preserve
unsynchronized data. Only do this procedure with the assistance of Support.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
162
Before following this procedure, try to bootstrap the cluster. For more information, see
Bootstrapping.
To manually force a node to rejoin the cluster, do the following:
1. SSH into the node by following the procedure in BOSH SSH.
2. Become root by running:
3. Shut down the mysqld process on the node by running:
4. Remove the unsynchronized data on the node by running:
5. Prepare the node before restarting by running:
6. Restart the mysqld process by running:
Re-create a Corrupted VM in a Highly Available Cluster
To re-create a corrupted VM:
1. To log in to the BOSH Director VM by doing the following procedures:
1. Gather the information needed to log in to the BOSH Director VM by doing the
procedure in Gather Credential and IP Address Information.
2. Log in to the Ops Manager VM by doing the procedure in Log in to the Ops
Manager VM with SSH.
3. Log in to the BOSH Director VM by doing the procedure in Authenticate with the
BOSH Director VM.
2. Identify and re-create the unresponsive node with bosh cloudcheck, by doing the
procedure in BOSH Cloudcheck and run Recreate VM using last known apply spec.
Check Replication Status in a Highly Available Cluster
If you see stale data in your cluster, you can check whether replication is functioning normally.
sudo su
monit stop galera-init
rm -rf /var/vcap/store/pxc-mysql
/var/vcap/jobs/pxc-mysql/bin/pre-start
monit start galera-init
Warning: Recreating a node clears its' logs. Ensure the node is completely
down before recreating it.
Warning: Only re-create one node. Do not re-create the entire cluster. If
more than one node is down, contact Support.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
163
To check the replication status, do the following:
1. To log in to the BOSH Director VM, do the following:
1. Gather the information needed to log in to the BOSH Director VM by doing the
procedure in Gather Credential and IP Address Information.
2. Log in to the Ops Manager VM by doing the procedure in Log in to the Ops
Manager VM with SSH.
2. Create a dummy database in the first node by running:
mysql -h FIRST-NODE-IP-ADDRESS \ -u YOUR-IDENTITY \ -p -e "create database veri
fy_healthy;"
Where:
FIRST-NODE-IP-ADDRESS is the IP address of the first node you recorded in step 1.
YOUR-IDENTITY is the value of identity that you recorded in step 1.
3. Create a dummy table in the dummy database by running:
4. Insert data into the dummy table by running:
5. Query the table and verify that the three rows of dummy data exist on the first node by
running:
* `FIRST-NODE-IP-ADDRESS` is the IP address of the first node you recorded in step 1. *
`YOUR-IDENTITY` is the value of `identity` that you recorded in step 1.
6. Create a dummy table in the dummy database by running:
mysql -h FIRST-NODE-IP-ADDRESS \ -u your-identity \ -p -D verify_healthy \ -e
"create table dummy_table (id int not null primary key auto_increment, info tex
t) \ engine='innodb';"
7. Insert data into the dummy table by running:
mysql -h FIRST-NODE-IP-ADDRESS \
-u your-identity \
-p -D verify_healthy \
-e "create table dummy_table (id int not null primary key auto_increm
ent, info text) \
engine='innodb';"
mysql -h FIRST-NODE-IP-ADDRESS \
-u YOUR-IDENTITY \
-p -D verify_healthy \
-e "insert into dummy_table(info) values ('dummy data'),('more dummy
data'),('even more dummy data');"
mysql -h FIRST-NODE-IP-ADDRESS \
-u YOUR-IDENTITY \
-p -D verify_healthy \
-e "select * from dummy_table;"
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
164
mysql -h FIRST-NODE-IP-ADDRESS \ -u YOUR-IDENTITY \ -p -D verify_healthy \ -e
"insert into dummy_table(info) values ('dummy data'),('more dummy data'),('even
more dummy data');"
8. Query the table and verify that the three rows of dummy data exist on the first node by
running:
mysql -h FIRST-NODE-IP-ADDRESS \ -u YOUR-IDENTITY \ -p -D verify_healthy \ -e
"select * from dummy_table;"
When prompted for a password, provide the password value recorded in step 1. The
previous command returns output similar to the following:
9. Verify that the other nodes contain the same dummy data by doing the following for each
of the remaining MySQL server IP addresses:
1. Query the dummy table by running :
When prompted for a password, provide the password value recorded in step 1.
2. Verify that the node contains the same three rows of dummy data as the other
nodes by running:
When prompted for a password, provide the password value recorded in step
3. Verify that the above command returns output similar to the following:
+----+----------------------+
| id | info |
+----+----------------------+
| 4 | dummy data |
| 7 | more dummy data |
| 10 | even more dummy data |
+----+----------------------+
mysql -h NEXT-NODE-IP-ADDRESS \
-u YOUR-IDENTITY \
-p -D verify\_healthy \
-e "select * from dummy_table;"
mysql -h NEXT-NODE-IP-ADDRESS \
-u YOUR-IDENTITY \
-p -D verify\_healthy \
-e "select * from dummy\_table;"
+----+----------------------+
| id | info |
+----+----------------------+
| 4 | dummy data |
| 7 | more dummy data |
| 10 | even more dummy data |
+----+----------------------+
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
165
10. If each MySQL server instance does not return the same result, before proceeding
further or making any changes to your deployment, contact Support.
If each MySQL server instance returns the same result, then you can safely proceed to
scaling down your cluster to a single node.
+----+----------------------+
11. Verify that the other nodes contain the same dummy data by doing the following for each
of the remaining MySQL server IP addresses:
1. Query the dummy table by running :
mysql -h NEXT-NODE-IP-ADDRESS \ -u YOUR-IDENTITY \ -p -D verify\_healthy
\ -e "select * from dummy_table;"
When prompted for a password, provide the password value recorded in step 1.
2. Verify that the node contains the same three rows of dummy data as the other
nodes by running:
mysql -h NEXT-NODE-IP-ADDRESS \
-u YOUR-IDENTITY \
-p -D verify\\_healthy \
-e "select \* from dummy\\_table;"
When prompted for a password, provide the password value recorded in step 1.
3. Verify that the previous command returns output similar to the following:
+----+----------------------+
| id | info |
+----+----------------------+
| 4 | dummy data |
| 7 | more dummy data |
| 10 | even more dummy data |
+----+----------------------+
12. If each MySQL server instance does not return the same result, before proceeding
further or making any changes to your deployment, contact Support. If each MySQL
server instance returns the same result, then you can safely proceed to scaling down
your cluster to a single node.
Tools for Troubleshooting
The troubleshooting techniques described here use the following tools.
download-logs
download-logs is a script that you can run from your Ops Manager VM to aggregate logs from your
Tanzu SQL for VMs cluster nodes, proxies, and, with highly available clusters, the jumpbox VM.
To use the download-logs script:
1. Download and unzip the download-logs script from VMware Tanzu Network.
2. From the Ops Manager Installation Dashboard, navigate to BOSH Director > Credentials.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
166
3. Click Link to Credential for the Bosh Commandline Credentials.
4. From the plaintext file that opens, record the values for the following:
BOSH_CLIENT
BOSH_CLIENT_SECRET
BOSH_CA_CERT
BOSH_ENVIRONMENT
5. From the BOSH CLI, view the name of the BOSH deployment for Tanzu SQL for VMs by
running:
bosh deployments
Record the name of the BOSH deployment.
6. SSH into your Ops Manager VM by doing the procedures in Gather Credential and IP
Address Information and SSH into Ops Manager.
7. File transfer or copy-paste the download-logs script to a working directory on the Ops
Manager VM.
8. Set local environment variables to the same BOSH variable values that you recorded earlier,
including BOSH_DEPLOYMENT for the deployment name.
For example:
9. Run the download-logs script by running:
./download-logs -o .
The script saves a compressed file of logs combined from all Tanzu SQL for VMs VMs. The
filename has the form TIMESTAMP-mysql-logs.tar.gz.gpg.
Tools for Troubleshooting
The troubleshooting techniques described above use the following tools.
download-logs
download-logs is a script that you can run from your Ops Manager VM to aggregate logs from your
Tanzu SQL for VMs cluster nodes, proxies, and, with highly available clusters, the jumpbox VM.
To use the download-logs script:
1. Download and unzip the download-logs script from VMware Tanzu Network.
2. From the Ops Manager Installation Dashboard, navigate to BOSH Director > Credentials.
3. Click Link to Credential for the Bosh Commandline Credentials.
$ BOSH_CLIENT=ops_manager \
BOSH_CLIENT_SECRET=a123bc-E_4Ke3fb-gImbl3xw4a7meW0rY
BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate \
BOSH_ENVIRONMENT=10.0.0.5 \
BOSH_DEPLOYMENT=pivotal-mysql-14c4
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
167
4. From the plaintext file that opens, record the values for the following:
BOSH_CLIENT
BOSH_CLIENT_SECRET
BOSH_CA_CERT
BOSH_ENVIRONMENT
5. From the BOSH CLI, view the name of the BOSH deployment for Tanzu SQL for VMs by
running:
bosh deployments
Record the name of the BOSH deployment.
6. SSH into your Ops Manager VM by doing the procedures in Gather Credential and IP
Address Information and Log in to the Ops Manager VM with SSH.
7. File transfer or copy-paste the download-logs script to a working directory on the Ops
Manager VM.
8. Set local environment variables to the same BOSH variable values that you recorded earlier,
including BOSH_DEPLOYMENT for the deployment name.
For example:
9. Run the download-logs script by running:
./download-logs -o .
The script saves a compressed file of logs combined from all Tanzu SQL for VMs VMs. The
filename has the form TIMESTAMP-mysql-logs.tar.gz.gpg.
mysql-diag
The mysql-diag tools outputs the current status of a highly available (HA) Tanzu SQL for VMs
cluster and suggests recovery actions if the cluster fails.
For more information, see Running mysql-diag.
Knowledge Base (Community)
Find the answer to your question and browse product discussions and solutions by searching the
VMware Tanzu Knowledge Base.
File a Support Ticket
$ BOSH\_CLIENT=ops\_manager \
BOSH\_CLIENT\_SECRET=a123bc-E_4Ke3fb-gImbl3xw4a7meW0rY
BOSH\_CA\_CERT=/var/tempest/workspaces/default/root\_ca\_certificate
\
BOSH\_ENVIRONMENT=10.0.0.5 \
BOSH\_DEPLOYMENT=pivotal-mysql-14c4
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
168
You can file a ticket with Support. Be sure to provide the error message from cf service YOUR-
SERVICE-INSTANCE.
To expedite troubleshooting, provide your service broker logs and your service instance logs. If
your cf service YOUR-SERVICE-INSTANCE output includes a task-id, provide the BOSH task output.
About Data Migration in VMware Tanzu SQL with MySQL for
VMs
This topic describes how to trigger a failover of apps from the leader to the follower.
Leader-Follower Procedures
Triggering a Leader-Follower Failover
Triggering a Leader-Follower Failover
This topic describes how to trigger a failover of apps from the leader to the follower.
Overview
You might want to trigger a failover in the following scenarios:
You want to take the leader VM down to do planned maintenance.
The performance of the leader VM degrades.
The leader VM fails unexpectedly.
The AZ where the leader VM is located goes offline unexpectedly.
You can use the following metrics to determine if you need to trigger a failover:
/p.mysql/available: This metric monitors whether the MySQL server is currently available.
For more information, see Server Availability.
/p.mysql/follower/seconds_behind_master: This metric monitors how far behind the
follower is in applying writes from the leader. For more information, see Leader-Follower
Metrics.
/p.mysql/follower/seconds_since_leader_heartbeat: This metric monitors the number of
seconds that elapse between the leader heartbeat and the replication of the heartbeat in
the follower. For more information, see Leader-Follower Metrics.
For information about errands used to trigger failover, see configure-leader-follower, make-leader,
and make-read-only.
To trigger a failover:
1. Retrieve Information
2. Promote the Follower
3. Clean Up Former Leader VM (Optional)
4. Configure the New Follower
5. Unbind and Rebind the App
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
169
Retrieve Information
To retrieve the information necessary for stopping the leader and promoting the follower:
1. Log in to your deployment by running:
cf login API-URL
When prompted, enter your credentials.
2. Target the org and space where the leader-follower service instance is located by running:
cf target -o DESTINATION-ORG -s DESTINATION-SPACE
3. Record the GUID of the service instance by running:
cf service SERVICE-INSTANCE-NAME --guid
Where SERVICE-INSTANCE-NAME is the name of the leader-follower service instance.
For example:
If you do not know the name of the service instance, you can list service instances in the
space with cf services.
4. Follow the procedure at Gather Credential and IP Address Information and Log in to the
Ops Manager VM with SSH to SSH into the Ops Manager VM.
5. From the Ops Manager VM, log in to your BOSH Director with the BOSH CLI. For more
information on logging in with the BOSH CLI, see Log in to the BOSH Director.
6. Use the BOSH CLI to run the inspect errand by running:
bosh -d service-instance_GUID run-errand inspect
Where GUID is the GUID of the leader-follower service instance you recorded.
For example:
7. See the output about the leader-follower MySQL VMs and identify the instance marked
Role: leader.
For example output:
$ cf service my-lf-instance --guid
82ddc607-710a-404e-b1b8-a7e3ea7ec063
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
run-errand inspect
Instance mysql/ca0ed8b5-7590-4cde-bba8-7ca2935f2bd0
Exit Code 0
Stdout 2018/04/03 18:08:46 Started executing command: inspect
2018/04/03 18:08:46
IP Address: 10.0.8.11
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
170
8. Record the index of the instance marked Role: leader. In this example output, the index of
the leader VM is ca0ed8b5-7590-4cde-bba8-7ca2935f2bd0.
9. Record the index of the other instance, which is the follower VM. In this example output,
the index of the follower VM is 37e4b6bc-2ed6-4bd2-84d1-e59a91f5e7f8.
10. If you still have access to the AZ where the leader VM is located, determine if the leader
VM is in the AZ you want to take offline by running:
bosh -d service-instance_GUID run-errand instances
For example:
Role: leader
Read Only: false
Replication Configured: false
Replication Mode: async
Has Data: true
GTID Executed: 82ddc607-710a-404e-b1b8-a7e3ea7ec063:1-18
2018/04/03 18:08:46 Successfully executed command: inspect
Stderr -
Instance mysql/37e4b6bc-2ed6-4bd2-84d1-e59a91f5e7f8
Exit Code 0
Stdout 2018/04/03 18:08:46 Started executing command: inspect
2018/04/03 18:08:46
IP Address: 10.0.8.10
Role: follower
Read Only: true
Replication Configured: true
Replication Mode: async
Has Data: true
GTID Executed: 82ddc607-710a-404e-b1b8-a7e3ea7ec063:1-18
2018/04/03 18:08:46 Successfully executed command: inspect
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
instances
Deployment 'service-instance_f378ec82-61a4-4e66-8ed9-889c7cf5342f'
Instance Process State AZ
IPs
mysql/ca0ed8b5-7590-4cde-bba8-7ca2935f2bd0 failing us-central1-
f 10.0.8.11
mysql/37e4b6bc-2ed6-4bd2-84d1-e59a91f5e7f8 running us-central1-
a 10.0.8.10
2 instances
Note: The leader VM might not display its status as failing if you are
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
171
Examine the output to determine if the leader VM is in the AZ you want to take offline.
Promote the Follower
To stop the leader VM and promote the follower VM to the new leader:
1. Stop any data from being written to the leader VM by setting it to read-only by running:
bosh -d service-instance_GUID \
run-errand make-read-only \
--instance=mysql/INDEX
Where:
GUID: This is the GUID of the leader-follower service instance retrieved above.
INDEX: This is the index of the leader VM retrieved above.
For example:
2. If you still have access to the AZ where the leader VM is located, stop the leader VM by
running:
bosh -d service-instance_GUID stop mysql/INDEX
Use the index of the leader VM retrieved above.
For example:
3. Set the follower VM as writable by running:
bosh -d service-instance_GUID run-errand make-leader --instance=mysql/INDEX
Use the index of the follower VM retrieved above.
For example:
If this command returns an error, re-run it until the follower VM has completed applying the
transactions.
At this point, a single instance is working but leader-follower replication has not yet been restored.
To fail your app over to a single instance instead of restoring leader-follower, skip to Unbind and
performing planned maintenance.
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
run-errand make-read-only \
--instance=mysql/ca0ed8b5-7590-4cde-bba8-7ca2935f2bd0
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
stop mysql/ca0ed8b5-7590-4cde-bba8-7ca2935f2bd0
$ bosh -d service-instance\_82dc607-710a-404e-b1b8-a7e3ea7ec063 \
run-errand make-leader \
--instance=mysql/37e4b6bc-2ed6-4bd2-84d1-e59a91f5e7f8
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
172
Rebind the App.
If you are triggering a failover in response to the AZ of the leader VM going offline, you can fail your
app over to a single instance by following the procedure in Unbind and Rebind the App below.
However, to restore leader-follower, you must regain access to the AZ where your leader VM is
located before following the procedure in Clean Up Former Leader VM (Optional) and Configure
the New Follower below.
Clean Up Former Leader VM (Optional)
If you are triggering a failover in response to a failing leader VM, to clean up the former leader VM:
1. Deactivate resurrection, specifying the same deployment as previously shown, by running:
bosh update-resurrection off
2. Retrieve the CID of the failing former leader VM by running:
bosh -d service-instance_GUID instances \
--details \
--failing \
--column=”VM CID” \
--json
For example:
3. Retrieve the disk CID of the failing former leader VM by running:
bosh -d service-instance_GUID instances \
--details \
--failing \
--column=”Disk CIDs” \
--json
For example:
4. Delete the failing former leader VM by running:
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 instan
ces \
--details \
--failing \
--column=”VM CID” \
--json
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 instan
ces \
--details \
--failing \
--column=”Disk CIDs” \
--json
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
173
bosh -d service-instance_GUID delete-vm vm-CID
Where:
GUID: This is the GUID of the leader-follower service instance retrieved above.
CID: This is the CID of the failing former leader VM retrieved above.
For example:
5. Orphan the disk of the failing former leader VM by running:
bosh -d service-instance_GUID orphan-disk DISK-CID
Where:
GUID: This is the GUID of the leader-follower service instance retrieved above.
DISK-CID: This is the disk CID of the failing former leader VM retrieved above.
For example:
Orphaning a disk rather than deleting it preserves the disk for possible recovery. After
performing recovery operations, you can reattach the disk to a VM. BOSH deletes
orphaned disks after five days by default.
Configure the New Follower
To restart the former leader VM and configure it as the new follower:
1. Create the former leader VM again by running:
bosh -d service-instance_GUID \
recreate \
mysql/INDEX
Where:
GUID: This is the GUID of the leader-follower service instance retrieved above.
INDEX: This is the index of the former leader VM that you are re-creating
For example:
2. Set the former leader VM as a follower, using the same values as above, by running:
GUID: This is the GUID of the leader-follower service instance retrieved above.
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
delete-vm i-1db9ede6
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
orphan-disk b-1db9ede6
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
recreate \
mysql/ca0ed8b5-7590-4cde-bba8-7ca2935f2bd01.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
174
INDEX: This is the index of the former leader VM that you are re-creating
For example:
3. Set the former leader VM as a follower, using the same values as previously shown, by
running:
bosh -d service-instance_GUID \
run-errand configure-leader-follower \
--instance=mysql/INDEX
For example:
4. Use the BOSH CLI to run the inspect errand, using the same value as previously shown, by
running:
bosh -d service-instance_GUID \
run-errand inspect
For example:
If the output displays one instance marked Role: leader and another instance marked
Role: follower, then leader-follower replication and high availability are resumed. The
deployment should be in its original, working state. You can turn resurrection back on if you
want too.
Unbind and Rebind the App
To fail their apps over to the new leader VM, your developers must bind and rebind their apps to
the leader-follower service instance:
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
recreate \
mysql/ca0ed8b5-7590-4cde-bba8-7ca2935f2bd01.
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
run-errand configure-leader-follower \
--instance=mysql/ca0ed8b5-7590-4cde-bba8-7ca2935f2bd0
$ bosh -d service-instance\_82ddc607-710a-404e-b1b8-a7e3ea7ec063 \
run-errand inspect
Note: If you have BOSH DNS enabled in Ops Manager, you do not need to unbind
and re-bind your app to a leader-follower service instance to failover the app. The
operator activates BOSH DNS in BOSH Director > BOSH DNS Config.
Warning: If a developer rebinds an app to the Tanzu SQL for VMs service after
unbinding, they must also rebind any existing custom schemas to the app. When
you rebind an app, stored code, programs, and triggers break. For more information
about binding custom schemas, see Use Custom Schemas.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
175
To unbind and rebind your app:
1. Unbind the app from the leader-follower service instance by running:
cf unbind-service APP-NAME SERVICE-INSTANCE-NAME
Where:
APP-NAME: This is the name of the app bound to the leader-follower service instance.
SERVICE-INSTANCE-NAME: This is the name of the leader-follower service instance.
2. Rebind the app to the leader-follower service instance by running:
cf bind-service APP-NAME SERVICE-INSTANCE-NAME
3. Restage the app by running:
cf restage APP-NAME
HA Tanzu MySQL for VMs Clusters Procedures
These topics describe procedures for HA clusters.
Highly Available Clusters Procedures
Bootstrapping
Running mysql-diag
About the Replication Canary
Bootstrapping
When to Bootstrap
Run the Bootstrap Errand
Determine Type of Cluster Failure
Scenario 1: Virtual Machines Running, Cluster Disrupted
Scenario 2: Virtual Machines Terminated or Lost
Bootstrap Manually
Shut Down MySQL
Choose Node to Bootstrap
Bootstrap the First Node
Restart Remaining Nodes
Verify that the Nodes Have Joined the Cluster
This topic describes how to bootstrap your MySQL cluster in the event of a cluster failure.
You can bootstrap your cluster by using one of two methods:
Run the bootstrap errand. See Run the Bootstrap Errand below.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
176
Bootstrap manually. In the case of a power failure where the mysql instances did not fail
gracefully, you need to bootstrap manually. See Bootstrap Manually below.
When to Bootstrap
You must bootstrap a cluster that loses quorum. A cluster loses quorum when fewer than half the
nodes can communicate with each other for longer than the configured grace period. If a cluster
does not lose quorum, individual unhealthy nodes automatically rejoin the cluster after resolving the
error, restarting the node, or restoring connectivity.
To check whether your cluster has lost quorum, look for the following symptoms:
All nodes appear “Unhealthy” on the proxy dashboard. The proxy dashboard is viewable at
proxy-BOSH-JOB-INDEX.p-mysql.YOUR-SYSTEM-DOMAIN.
All responsive nodes report the value of wsrep_cluster_status as non-Primary:
All unresponsive nodes respond with ERROR 1047 when using most statement types in the
MySQL client:
For more information about checking the state of your cluster, see Check Cluster State.
mysql> SHOW STATUS LIKE 'wsrep_cluster_status';
+----------------------+-------------+
| Variable_name | Value |
+----------------------+-------------+
| wsrep_cluster_status | non-Primary |
+----------------------+-------------+
mysql> select * from mysql.user;
ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for appli
cation use
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
177
Run the Bootstrap Errand
VMware Tanzu SQL with MySQL for VMs includes a BOSH errand that automates the manual
bootstrapping procedure in the Bootstrap Manually section below.
It finds the node with the highest transaction sequence number and asks it to start up by itself (i.e.
in bootstrap mode) and then asks the remaining nodes to join the cluster.
In most cases, running the errand recovers your cluster. But, certain scenarios require additional
steps.
Determine Type of Cluster Failure
To determine which set of instructions to follow:
1. List your MySQL instances by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT instances
Where:
YOUR-ENV is the environment where you deployed the cluster.
YOUR-DEPLOYMENT is the deployment cluster name.
For example:
2. Find and record the Process State for your MySQL instances. In the following example
output, the MySQL instances are in the failing process state.
$ bosh -e prod -d mysql instances
Instance Pr
ocess State AZ IPs
backup-restore/c635410e-917d-46aa-b054-86d222b6d1c0 ru
nning us-central1-b 10.0.4.47
bootstrap/a31af4ff-e1df-4ff1-a781-abc3c6320ed4 -
us-central1-b -
broker-registrar/1a93e53d-af7c-4308-85d4-3b2b80d504e4 -
us-central1-b 10.0.4.58
cf-mysql-broker/137d52b8-a1b0-41f3-847f-c44f51f87728 ru
nning us-central1-c 10.0.4.57
cf-mysql-broker/28b463b1-cc12-42bf-b34b-82ca7c417c41 ru
nning us-central1-b 10.0.4.56
deregister-and-purge-instances/4cb93432-4d90-4f1d-8152-d0c238fa5aab -
us-central1-b -
monitoring/f7117dcb-1c22-495e-a99e-cf2add90dea9 ru
nning us-central1-b 10.0.4.48
mysql/220fe72a-9026-4e2e-9fe3-1f5c0b6bf09b fa
iling us-central1-b 10.0.4.44
mysql/28a210ac-cb98-4ab4-9672-9f4c661c57b8 fa
iling us-central1-f 10.0.4.46
mysql/c1639373-26a2-44ce-85db-c9fe5a42964b fa
iling us-central1-c 10.0.4.45
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
178
3. Choose your scenario:
If your MySQL instances are in the failing state, continue to Scenario 1.
If your MySQL instances are in the - state, continue to Scenario 2.
Scenario 1: VMs Running, Cluster Disrupted
In this scenario, the VMs are running, but the cluster has been disrupted.
To bootstrap in this scenario:
1. Run the bootstrap errand on the VM where the bootstrap errand is co-located, by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT run-errand --instance=mysql/0 bootstrap
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
The command returns many lines of output, eventually followed by:
2. If the errand fails, run the bootstrap errand command again after a few minutes. The
bootstrap errand might not work the first time.
3. If the errand fails after several tries, bootstrap your cluster manually. See Bootstrap
Manually below.
Scenario 2: VMs Terminated or Lost
proxy/87c5683d-12f5-426c-b925-62521529f64a ru
nning us-central1-b 10.0.4.60
proxy/b0115ccd-7973-42d3-b6de-edb5ae53c63e ru
nning us-central1-c 10.0.4.61
rejoin-unsafe/8ce9370a-e86b-4638-bf76-e103f858413f -
us-central1-b -
smoke-tests/e026aaef-efd9-4644-8d14-0811cb1ba733 -
us-central1-b 10.0.4.59
Note: The errand runs for a long time, during which no output is returned.
Bootstrap errand completed
[stderr]
+ echo 'Started bootstrap errand ...'
+ JOB\_DIR=/var/vcap/jobs/bootstrap
+ CONFIG\_PATH=/var/vcap/jobs/bootstrap/config/config.yml
+ /var/vcap/packages/bootstrap/bin/cf-mysql-bootstrap -configPath=/var/
vcap/jobs/bootstrap/config/config.yml
+ echo 'Bootstrap errand completed'
+ exit 0
Errand 'bootstrap' completed successfully (exit code 0)
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
179
In severe circumstances, it is possible to lose all your VMs. You must re-create them before you
can begin recovering the cluster.
When MySQL instances are in the - state, the VMs are lost. The procedures in this scenario bring
the instances from a - state to a failing state. Then you run the bootstrap errand similar to
Scenario 1 above and restore configuration.
To recover terminated or lost VMs, follow the procedures in the sections below:
1. Re-create the Missing VMs: Bring MySQL instances from a - state to a failing state.
2. Run the Bootstrap Errand: Because your instances are now in the failing state, continue
similarly to Scenario 1 above.
3. Restore the BOSH Configuration: Go back to unignoring all instances and redeploy. This is a
critical and mandatory step.
Re-create the Missing VMs
The procedure in this section uses BOSH to re-create the VMs, install software on them, and try to
start the jobs.
The procedure below allows you to:
Redeploy your cluster while expecting the jobs to fail.
Instruct BOSH to ignore the state of each instance in your cluster. This allows BOSH to
deploy the software to each instance even if the instance is failing.
To re-create your missing VMs:
1. If BOSH Resurrector is enabled, disable it by running:
bosh -e YOUR-ENV update-resurrection off
Where YOUR-ENV is the name of your environment.
2. Download the current manifest by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT manifest > /tmp/manifest.yml
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
3. Redeploy deployment by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT deploy /tmp/manifest.yml
Where:
Warning: If you do not set each of your ignored instances to unignore, your
instances are not updated in future deploys. You must follow the procedure in the
final section of
Scenario 2
, Restore the BOSH Configuration.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
180
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
4. View the instance GUID of the VM that attempted to start by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT instances
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
Record the instance GUID, which is the string after mysql/ in your BOSH instances output.
5. Instruct BOSH to ignore each MySQL VM by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT ignore mysql/INSTANCE-GUID
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
INSTANCE-GUID is the GUID of your instance you recorded in the above step.
6. Repeat steps 4 and 5 until all instances have attempted to start.
7. If you disabled the BOSH Resurrector in step 2, re-enable it by running:
bosh -e YOUR-ENV update-resurrection on
Where YOUR-ENV is the name of your environment.
8. Confirm that your MySQL instances have gone from the - state to the failing state by
running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT instances
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
Run the Bootstrap Errand
After you re-create the VMs, all instances now have a failing process state and have the MySQL
code. You must run the bootstrap errand to recover the cluster.
To bootstrap:
Note: Expect one of the MySQL VMs to fail. Deploying causes BOSH to
create new VMs and install the software. Forming a cluster is in a
subsequent step.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
181
1. Run the bootstrap errand by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT run-errand --instance=mysql/0 bootstrap
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
The command returns many lines of output, eventually with the following successful output:
2. If the errand fails, run the bootstrap errand command again after a few minutes. The
bootstrap errand might not work immediately.
3. See that the errand completes successfully in the shell output and continue to Restore the
BOSH Configuration below.
Restore the BOSH Configuration
To restore your BOSH configuration to its previous state, this procedure unignores each instance
that was previously ignored:
1. For each ignored instance, run:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT unignore mysql/INSTANCE-GUID
Where:
YOUR-ENV is the name of your environment.
Note: The errand runs for a long time, during which no output is returned.
Bootstrap errand completed
[stderr]
echo 'Started bootstrap errand ...'
JOB\_DIR=/var/vcap/jobs/bootstrap
CONFIG\_PATH=/var/vcap/jobs/bootstrap/config/config.yml
/var/vcap/packages/bootstrap/bin/cf-mysql-bootstrap -configPath=/var/
vcap/jobs/bootstrap/config/config.yml
echo 'Bootstrap errand completed'
exit 0
Errand 'bootstrap' completed successfully (exit code 0)
Note: After you complete the bootstrap errand, you might still see instances
in the failing state. Continue to the next section anyway.
Warning: If you do not set each of your ignored instances to unignore, your
instances are never updated in future deploys.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
182
YOUR-DEPLOYMENT is the name of your deployment.
INSTANCE-GUID is the GUID of your instance.
2. Redeploy your deployment by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT deploy
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
3. Verify that all mysql instances are in a running state by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT instances
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
Bootstrap Manually
If the bootstrap errand is not able to automatically recover the cluster, you need to do the steps
manually.
Follow the procedures in the sections below to manually bootstrap your cluster.
Shut Down MySQL
Follow these steps for each node in the cluster to stop the galera-init process for each node. For
each node, record if the monit stop command was successful:
1. List the nodes in the cluster by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT instances
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
2. For each node in the cluster, do steps 3–5. You cannot bootstrap the cluster unless you
have shut down the mysqld process on all nodes in the cluster.
3. SSH into the node by running:
Warning: The following procedures are prone to user-error and can result in lost
data if followed incorrectly. Follow the procedure in Run the Bootstrap Errand
above first, and only resort to the manual process if the errand fails to repair the
cluster.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
183
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT ssh mysql/GUID
Where: GUID is the GUID of a node.
For example:
4. To shut down the mysqld process on the node, run:
monit stop galera-init
5. Record if the monit command succeeds or exits with an error:
If monit succeeds in stopping galera-init, then you can use monit to restart this
node. Follow all the steps below including the steps marked Monit Restart but
omitting the steps marked Manual Redeploy.
If monit exits with the following error, then you must manually deploy this node:
Follow all the steps below including the steps marked Manual Redeploy but
omitting the steps marked Monit Restart.
Determine which Node to Bootstrap
To identify which node to bootstrap, you must find the node with the highest transaction
sequence_number. The node with the highest sequence number is the one most recently updated.
To identify the node to bootstrap:
1. SSH into the node using the procedure in BOSH SSH.
2. View the sequence number for a node by running:
/var/vcap/jobs/pxc-mysql/bin/get-sequence-number
When prompted confirm that you want to stop MySQL.
For example:
3. Record the value of sequence_number.
4. Repeat the steps above for each node in the cluster.
$ bosh -e my-env -d my-dep ssh mysql/abcd0123-a012-b345-c678-9def012345
67
Warning: include files not found '/var/vcap/monit/job/*.monitrc'
monit: action failed -- There is no service by that name
$ /var/vcap/jobs/mysql/bin/get-sequence-number
This script stops mysql. Are you sure? (y/n): y
{"sequence_number":421,"instance_id":"012abcde-f34g-567h-ijk8-9123l45
67891"}
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
184
5. After determining the sequence_number for all nodes in your cluster, identify the node with
the highest sequence_number. If all nodes have the same sequence_number, you can choose
any node as the new bootstrap node.
Bootstrap the First Node
After determining the node with the highest sequence_number, do the following to bootstrap the
node:
1. SSH into the node using the procedure in BOSH SSH.
2. Update the node state to trigger its initialization of the cluster by running:
echo -n "NEEDS_BOOTSTRAP" > /var/vcap/store/pxc-mysql/state.txt
3. Monit Restart: If in Shut Down MySQL above you successfully used monit to shut down
your galera-init process, then re-launch the mysqld process on the new bootstrap node.
1. Start the mysqld process by running:
monit start galera-init
2. It can take up to ten minutes for monit to start the mysqld process. To confirm if the
mysqld process has started successfully, run:
watch monit summary
If monit succeeds in starting the galera-init process, then the output includes the
line Process 'galera-init' running.
4. Manual Redeploy: If in Shut Down MySQL above you encountered monit errors, then
redeploy the mysqld software to your bootstrap node as follows:
1. Leave the MySQL SSH login shell and return to your local environment.
2. Target BOSH on your bootstrap node by instructing it to ignore the other nodes in
your cluster. For nodes all nodes except the bootstrap node you identified above,
run:
Where N and M are the numbers of the non-bootstrapped nodes. For example, if
you bootstrap node 0, then M=1 and N=2.
3. Turn off the BOSH Resurrector by running:
Note: Only run these bootstrap commands on the node that you identified in
Determine which Node to Bootstrap above. Non-bootstrapped nodes abandon
their data during the bootstrapping process. Therefore, bootstrapping off the wrong
node causes data loss. For information about intentionally abandoning data, see
Architecture.
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT ignore mysql/M
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT ignore mysql/N
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
185
4. Use the BOSH manifest to bootstrap your bootstrap machine by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT manifest > /tmp/manifest.yml
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT deploy /tmp/manifest.yml
Restart Remaining Nodes
After the bootstrapped node is running, restart the remaining nodes.
The procedure that you follow for restarting a node, depends on the output you got for that node
during Shut Down MySQL above. Do one of the following procedures:
Monit Restart
If in Shut Down MySQL above you successfully used monit to shut down your galera-init process,
then restart the nodes as follows:
1. SSH into the node using the procedure in BOSH SSH.
2. Start the mysqld process with monit by running:
monit start galera-init
If the Interruptor prevents the node from starting, follow the manual procedure to force the
node to rejoin the cluster. See How to manually force a MySQL node to rejoin if a node
cannot rejoin the HA (Galera) cluster in the Support knowledge base.
3. If the monit start command fails, it might be because the node with the highest
sequence_number is mysql/0.
In this case:
1. Ensure BOSH ignores updating mysql/0 by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT ignore mysql/0
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
2. Navigate to Ops Manager in a browser, log in, and click Apply Changes.
3. When the deploy finishes, run the following command from the Ops Manager VM:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT unignore mysql/0
bosh update-resurrection off
Warning: Forcing a node to rejoin the cluster is a destructive procedure.
Only follow the procedure with the assistance of Support.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
186
Where:
YOUR-ENV is the name of your environment.
YOUR-DEPLOYMENT is the name of your deployment.
Manual Redeploy
If in Shut Down MySQL above you encountered monit errors, then restart the nodes as follows:
1. Instruct BOSH to no longer ignore the non-bootstrap nodes in your cluster by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT unignore mysql/M
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT unignore mysql/N
Where N and M are the numbers of the non-bootstrapped nodes. For example, if you
bootstrap node 0, then M=1 and N=2.
2. Redeploy software to the other two nodes and have them rejoin the cluster, bootstrapped
from the node above by running:
bosh -e YOUR-ENV -d YOUR-DEPLOYMENT deploy /tmp/manifest.yml
You only need to run this command once to deploy both the nodes that you unignored in
the step above.
3. With your redeploys completed, turn the BOSH Resurrector back on by running:
Verify that the Nodes Have Joined the Cluster
The final task is to verify that all the nodes have joined the cluster.
1. Connect to the MySQL database with BOSH SSH. For instructions, see Connect to MySQL
with BOSH SSH.
2. Run the following command to output the total number of nodes in the cluster:
mysql> SHOW STATUS LIKE 'wsrep_cluster_size';
Running mysql-diag
Run mysql-diag the Using BOSH Command Line Interface (CLI)
Example Healthy Output
Example Unhealthy Output
This topic discusses how to use the mysql-diag tool in VMware Tanzu SQL with MySQL for VMs.
mysql-diag prints the state of your MySQL highly available (HA) cluster and suggests solutions if
your node fails. VMware recommends running this tool against your HA cluster before all
deployments.
mysql-diag checks the following information about the status of your HA cluster:
bosh -e YOUR-ENV update-resurrection on
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
187
Membership status of all nodes
Size as it appears to all nodes
If it needs to be bootstrapped
If replication is working
Used disk space and inodes per server
Run mysql-diag Using the BOSH Command Line Interface
(CLI)
To use the BOSH CLI to run mysql-diag, do the following:
1. Obtain the information needed to use the BOSH CLI by doing the procedure in Gather
Credential and IP Address Information.
2. SSH into your Ops Manager VM by doing the procedure in Log in to the Ops Manager VM
with SSH for your IaaS.
3. Log in to your BOSH Director by doing the procedure in Authenticate with the BOSH
Director VM.
4. Identify the VM to log in to with SSH by running the following command:
bosh -e MY-ENV -d MY-DEPLOYMENT vms
Where:
MY-ENV is the name of your environment.
MY-DEPLOYMENT is the name of your deployment.
5. Record the GUID associated with the mysql-monitor VM, also known as the jumpbox VM.
6. SSH into your mysql-monitor VM by running the following command:
bosh -e MY-ENV -d MY-DEP ssh mysql-monitor/GUID
Where:
MY-ENV is the name of your environment.
MY-DEPLOYMENT is the name of your deployment.
GUID is the GUID you recorded in the previous step.
7. View the status of your HA cluster by running the following command:
mysql-diag
Example Healthy Output
The mysql-diag command returns the following message if your canary status is healthy:
Checking canary status...healthy
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
188
Here is a sample mysql-diag output after the tool identified a healthy HA cluster:
View a larger version of this image.
Example Unhealthy Output
The mysql-diag command returns the following message if your canary status is unhealthy:
In the event of a broken HA cluster, running mysql-diag outputs actionable steps meant to
expedite the recovery of that HA cluster. Below is a sample mysql-diag output after the tool
identified an unhealthy HA cluster:
Checking canary status...unhealthy
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
189
View a larger version of this image.
About the Replication Canary
This topic describes the replication canary for an highly available (HA) cluster. The replication canary
runs on the jumpbox VM and monitors an HA cluster to ensure that replication is working.
Overview
The replication canary writes to a private dataset in the cluster and attempts to read the written
data from each node. It pauses between read and write mode to ensure that the write-sets have
been replicated. The private dataset does not use a significant amount of disk capacity.
When replication fails, the canary cannot read the data from all the nodes and does the following:
Sends a message that replication has failed to a configured email address. See Sample
Notification Email.
Deactivates client access to the cluster. See Determine If the Cluster Is Accessible.
Sample Notification Email
Warning: When replication fails, data can be lost. Contact Support immediately in
the case of replication failure.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
190
If the canary detects replication failure, it immediately sends an email through the VMware Tanzu
Application Service for VMs (TAS for VMs) notification service. You must have configured email
notifications in TAS for VMs for the replication canary to send emails.
For more information about configuring email notifications, see the
Configure Email Notifications
procedure for your IaaS in the Ops Manager documentation.
The notification service sends emails similar to the following:
Subject: CF Notification: p-mysql Replication Canary, alert 417
This message was sent directly to your email address.
{alert-code 417}
This is an email to notify you that the MySQL service's replication canary
has detected an unsafe cluster condition in which replication is not
performing as expected across all nodes.
Determine If the Cluster Is Accessible
When the canary detects replication failure, it deactivates connections to the database cluster
through the proxies. When the replication issue is resolved, the canary automatically restores client
access to the cluster. You can determine if the canary deactivated cluster access by observing the
cluster using the Switchboard API.
To determine if cluster access is deactivated, do the following:
1. Do the prerequisite procedure in Monitor Node Health.
2. To view cluster access, run the following command:
curl https://USERNAME:PASSWORD@N-HOSTNAME/v0/cluster
Where:
USERNAME is the username you recorded in step 1.
PASSWORD is the password you recorded in step 1.
N is 0, 1, or 2 depending on the proxy you want to connect to.
HOSTNAME is the hostname you recorded in step 1.
For example:
$ curl https://abcdefghijklmno:[email protected]
2e-34f5-g34612c10dba.org.dedicated-mysql.cf-app.com/v0/cluster
[
{
"currentBackendIndex":0,
"trafficEnabled":false,
"message":"Disabling cluster traffic",
"lastUpdated":"2016-07-27T05:16:29.197754077Z"
}
]
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
191
When cluster access is deactivated, trafficEnabled is set to false.
If you must restore client access to the cluster while replication is failing, contact Support.
For more information about the Switchboard API, see Monitoring Node Health.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
192
VMware Tanzu SQL with MySQL for VMs -
Developer Guide
This guide includes the following topics:
Developer Guide
Getting Started
Using Tanzu MySQL for VMs
Using Tanzu MySQL for VMs for Multi-Site Replication
Using TLS
About Data Migration in Tanzu MySQL for VMs
Migrating Data in Tanzu MySQL for VMs
About MySQL Server Defaults
Changing Defaults Using Arbitrary Parameters
Managing Tanzu MySQL for VMs
Connecting to Tanzu MySQL for VMs
Customizing Database Credentials
Using Management Tools for Tanzu MySQL for VMs
Using SSH to Connect from Outside a Deployment
Triggering Multi-Site Replication and Failover
Backing Up and Restoring with mysqldump
Monitoring Node Health for HA Clusters
Troubleshooting Instances
VMware Tanzu SQL with MySQL for VMs - Developer Guide
- Getting Started
Getting Started
Using Tanzu MySQL for VMs
Using Tanzu MySQL for VMs for Multi-Site Replication
Using TLS
About Data Migration in Tanzu MySQL for VMs
Migrating Data in Tanzu MySQL for VMs
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
193
About MySQL Server Defaults
Changing Defaults Using Arbitrary Parameters
Using VMware Tanzu SQL with MySQL for VMs
This topic provides instructions for developers using the VMware Tanzu SQL with MySQL for VMs
service for their VMware Tanzu Application Service for VMs apps.
Overview
Tanzu SQL for VMs provides a relational database for apps and devices. To use Tanzu SQL for VMs
in an app:
1. Check the service availability in the Marketplace, and see if there is an existing instance of
Tanzu SQL for VMs in your space.
See Confirm the Tanzu SQL for VMs Service Availability.
2. If there is no existing instance or you want to use a different one, create an instance of the
Tanzu SQL for VMs service in the same space as the app.
See Create a Service Instance.
3. Bind the app to the Tanzu SQL for VMs service instance, to enable the app to use MySQL.
See Bind a Service Instance to Your App.
4. Call the Tanzu SQL for VMs service in your app code, and then push your app again into the
space.
For more information on services, see Use the MySQL Service in Your App.
After you create a Tanzu SQL for VMs service instance, you can manage it over the life cycle of
your apps and data.
For more information on managing services instances, see Manage Service Instances.
The following procedures use the Cloud Foundry Command Line Interface (cf CLI).
For more information, see Managing Service Instances with the cf CLI. You can also use Apps
Manager to do the same tasks using a graphical UI. For more information, see Managing Apps and
Service Instances Using Apps Manager.
Prerequisites
To use Tanzu SQL for VMs with your TAS for VMs apps, you must:
Decide what type of plan you want from the following options: single node, leader-follower,
highly available (HA) cluster, and Multi‑Site Replication. Multi‑Site Replication is used to
deploy a leader-follower service instance across multiple foundations or data centers. Your
marketplace might not offer all of these plan types.
For more information about service plans, see Availability Options.
If you intend to use a Multi‑Site Replication plan for deploying a leader-follower service
instance across multiple foundations or data centers, review the limitations associated with
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
194
this topology.
For more information, see Multi‑Site Replication Limitations.
If you intend to use an HA cluster plan, review the limitations associated with this plan type.
For more information, see Highly Available Cluster Limitations.
Have an Ops Manager installation with Tanzu SQL for VMs installed and listed in the
Marketplace.
For how to verify availability in the Marketplace, see Confirm Service Availability below.
Have a Space Developer or Admin account on the TAS for VMs installation.
For more information, see User Roles.
Have a local machine with the following installed:
A browser
A shell
The Cloud Foundry Command-Line Interface (cf CLI). See Installing the cf CLI.
The Linux watch command. See the Linux Information Project website.
Log in to the org and space containing your app. For instructions, see Log In With the CLI.
Confirm the Tanzu SQL for VMs Service Availability
For an app to use the Tanzu SQL for VMs service, both of the following must be true:
The service must be available in the Marketplace for its space.
An instance of the service must exist in its space.
You can confirm both of these using the cf CLI as follows.
Check Service Availability in the Marketplace
To find out if a Tanzu SQL for VMs service is available in the Marketplace:
1. Enter the following command:
cf marketplace
2. If the output lists p.mysql in the service column, Tanzu SQL for VMs is available. If it is not
available, ask your operator to install it.
Check that an Instance is Running in the Space
$ cf marketplace
Getting services from marketplace in org my-org / space my-space as use
OK
service plans description
[...]
p.mysql db-small Dedicated instances of MySQL service
to provide a relational database
[...]
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
195
To confirm that a Tanzu SQL for VMs instance is running in the space:
1. Use the cf CLI or Apps Manager to log in to the org and space that contains the app.
2. Enter the following command:
cf services
3. Any p.mysql listings in the service column are service instances of Tanzu SQL for VMs in
the space.
For example:
You can bind your app to an existing instance or create a new instance to bind to your app.
Create a Service Instance
On-demand services are created asynchronously, not immediately. The watch command shows you
when your service instance is ready to bind and use.
To create an instance of the Tanzu SQL for VMs service:
1. Create a service instance by running the following command:
cf create-service p.mysql PLAN SERVICE-INSTANCE
Where:
PLAN is the name of the Tanzu SQL for VMs plan you want to use.
SERVICE-INSTANCE is a name you choose to identify the service instance. This name
appears under service in output from cf services.
2. Enter the following command and wait for the last operation for your instance to show as
create succeeded.
watch cf services
For example:
$ cf services
Getting services in org my-org / space my-space as [email protected]...
OK
name service plan bound apps last operation
my-instance p.mysql db-small create succeeded
Note: If you are deploying a leader-follower service instance across multiple
foundations, follow the procedure in Using Tanzu SQL for VMs for Multi‑Site
Replication.
$ cf create-service p.mysql db-small my-instance
Creating service my-instance in org my-org / space my-space as user@exa
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
196
If you get an error, see Troubleshooting Instances.
Bind a Service Instance to your App
For an app to use a service, you must bind the app to a service instance. You must do this after
every time you run cf push.
To push and bind an app to a Tanzu SQL for VMs instance run the following command:
1. Push your app into the same space as your Tanzu SQL for VMs service instance by running
the following command:
cf push
2. Bind your app to a Tanzu SQL for VMs instance by running the following command:
cf bind-service APP SERVICE-INSTANCE
Where:
APP is the app you want to use the MySQL service instance.
SERVICE-INSTANCE is the name you supplied when you ran cf create-service.
For example:
3. Restart your app by running the following command:
cf restart APP
Where APP is the app you want to use the MySQL service instance.
Use the MySQL Service in your App
To access the MySQL service from your app:
mple.com...
OK
$ watch cf services
Getting services in org my-org / space my-space as [email protected]...
OK
name service plan bound apps last operation
my-instance p.mysql db-small create succeeded
$ cf bind-service my-app my-instance
Binding service mydb to my-app in org my-org / space test as user@examp
le.com...
OK
TIP: Use 'cf push' to ensure your env variable changes take effect
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
197
1. Verify that your app code (or the MySQL client library that the app uses) retries in the case
of DNS timeouts.
2. Locate the connection strings listed in the VCAP_SERVICES > credentials object for your
app. For information about VCAP_SERVICES, see MySQL Environment Variables.
3. In your app code, call the MySQL service using the connection strings.
See this example: Node.js code.
Use Custom Schemas
Tanzu SQL for VMs supports multiple custom schemas. You can use custom schemas with apps
that share a MySQL service instance to isolate app data by schema. By default, service bindings use
the default schema service_instance_db.
To use custom schemas in your apps:
1. Bind your app to the custom schema by running:
cf bind-service APP SERVICE-INSTANCE -c '{"schema":"CUSTOM-SCHEMA"}'
Where:
APP is the app you want to use the custom schema.
SERVICE-INSTANCE is the name of your service instance.
CUSTOM-SCHEMA is the name of your custom schema. Valid characters include
uppercase and lowercase letters, digits, $, and _.
2. Restart your app by running:
cf restart APP
Where APP is the app you want to use the custom schema.
MySQL Environment Variables
Apps running in Ops Manager gain access to bound service instances through an environment
variable credentials hash called VCAP_SERVICES. This environment variable includes the credentials
that apps use to access service instances.
For example:
{
"p.mysql": [
{
"label": "p.mysql",
"name": "my-instance",
"plan": "db-medium",
"provider": null,
"syslog_drain_url": null,
"tags": [
"mysql"
],
"credentials": {
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
198
"hostname": "10.0.0.20",
"jdbcUrl": "jdbc:mysql://10.0.0.20:3306/service_instance_db?user=fefcbe8360854
a18a7994b870e7b0bf5\u0026password=z9z6eskdbs1rhtxt",
"name": "service_instance_db",
"password": "z9z6eskdbs1rhtxt",
"port": 3306,
"uri": "mysql://fefcbe8360854a18a7994b870e7b0bf5:z9z6eskdbs1rhtxt@10.0.0.20:33
06/service_instance_db?reconnect=true",
"username": "fefcbe8360854a18a7994b870e7b0bf5"
},
"volume_mounts": []
}
]
}
You can search for your service by the name given when the service instance was created. You can
also search using the tags or label properties. The credentials property can be used to provide
access to the MySQL protocol.
VCAP_SERVICES is only modified when an app is bound to a service instance. If you modify your
service instance, you must cf unbind-service, cf bind-service and cf restage your app to apply
the changes to VCAP_SERVICES.
Manage a Service Instance
You can manage service instances in the following ways:
Migrate your data to a different plan. See Migrate Data to a Different Plan.
Upgrade service instance individually to the latest version of Tanzu SQL for VMs. See
Upgrade an Individual Service Instance.
Change the default parameters for an existing service instance. See Change Default
Parameters on an Existing Service Instance.
Share service instances between orgs and spaces. See Share Service Instances.
Remove access to a service from an app that no longer needs it. See Unbind an App from a
Service Instance.
Delete a service instance that is not used. See Delete a Service Instance.
Note: If you use MySQL Connector/J 8.0.13 or later with Tanzu SQL for VMs, you
must modify the JDBC URL in VCAP_SERVICES to include sslMode=VERIFY_IDENTITY
and verifyServerCertificate=true. MySQL Connector/J 8.0.13 and later does not
verify TLS connections. For more information about JDBC URL syntax, see the
MySQL documentation.
Warning: If a developer rebinds an app to the Tanzu SQL for VMs service after
unbinding, they must also rebind any existing custom schemas to the app. When
you rebind an app, stored code, programs, and triggers break. For more information
about binding custom schemas, see Use Custom Schemas.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
199
Migrate Data to a Different Plan
You can use cf update-service to migrate data to a different plan. When you update a service
instance, you do not need to rebind your app or service keys. However, when you migrate data to
a new service instance the database is unavailable for several minutes.
For more information about using cf update-service, see the Cloud Foundry CLI Reference Guide.
The following table lists migration use cases for the update-service command:
Use update-service for migrating from... To...
Single Node larger Single Node
Leader-Follower larger Leader-Follower
Single Node Leader-Follower of the same or larger size
Leader-Follower Single Node of the same or larger size
To migrate a service instance to another plan:
1. View the available service plans for Tanzu SQL for VMs by running:
cf marketplace
2. Migrate data to another plan by running:
cf update-service SERVICE-INSTANCE -p PLAN
Where PLAN is the plan you want to update the service instance to.
For example:
Upgrade an Individual Service Instance
You can use cf update-service with the --upgrade flag to individually upgrade on-demand service
instances to the latest version of Tanzu SQL for VMs. When you upgrade a service instance, you do
Warning: You cannot use cf update-service to migrate data between an HA
cluster plan and a plan of another topology. If you want to do this, you must use the
cf mysql-tools plug-in instead. For more information about migrating data, see
About Data Migration in VMware Tanzu SQL with MySQL for VMs.
Warning: If you are using multi‑site replication, you must not update the Multi‑Site
Replication plan to a plan of another topology. If you do this, your replication breaks.
$ cf update-service my-instance -p db-large
Note: Before you individually upgrade service instances you must have cf CLI
v6.46.0 or later.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
200
not need to rebind your app or service keys. However, when you upgrade a service instance the
database is unavailable for several minutes.
For more information about using cf update-service, see the Cloud Foundry CLI Reference Guide.
To upgrade a single service instance:
1. Confirm that an upgrade is available for the service instance by running:
cf services
The upgrade is available when the upgrade available column in the output says yes. For
example:
2. Upgrade the service instance by running:
cf update-service SERVICE-INSTANCE-NAME --upgrade
3. When prompted, confirm that you want to upgrade.
Share Service Instances
In Tanzu SQL for VMs you can share service instances between different orgs and spaces using cf
share-service. Service instance sharing is enabled by default.
For more information about service instance sharing, see Sharing Service Instances.
To share a service instance:
1. Target the source org and space for the service instance you want to share by running:
cf target -o SOURCE-ORG -s SOURCE-SPACE
Where:
SOURCE-ORG is the source org for your service instance.
SOURCE-SPACE is the source space for your service instance.
2. Share your service instance to the destination org and space by running:
cf share-service SERVICE-INSTANCE -o DESTINATION-ORG -s DESTINATION-SPACE
Where:
SERVICE-INSTANCE is the service instance you want to share.
DESTINATION-ORG is the destination org for the service instance.
$ cf services
Getting services in org system / space system as admin...
name service plan bound apps last operation broker
upgrade available
my-instance p.mysql db-small create succeeded dedicat
ed-mysql-broker yes
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
201
DESTINATION-SPACE is the destination space for the service instance.
3. Target the destination org and space by running:
cf target -o DESTINATION-ORG -s DESTINATION-SPACE
4. Confirm the service instance was shared by running:
cf service SERVICE-INSTANCE
Where SERVICE-INSTANCE is the service instance you shared.
Unbind an App from a Service Instance
If you want stop an app from using a service you must unbind the app from the service.
1. Unbind your app by running:
cf unbind-service APP SERVICE-INSTANCE
Where:
APP is the app you want to stop using the MySQL service instance.
SERVICE-INSTANCE is the name you supplied when you ran cf create-service.
For example:
Delete a Service Instance
You cannot delete a service instance that an app is bound to.
To delete a service instance:
1. Run the following command:
cf delete-service SERVICE-INSTANCE
Where SERVICE-INSTANCE is the name of the service to delete.
For example:
$ cf unbind-service my-app my-instance
Unbinding app my-app from service my-instance in org my-org / space my-
space as [email protected]...
OK
Warning: If a developer rebinds an app to the Tanzu SQL for VMs service after
unbinding, they must also rebind any existing custom schemas to the app. When
you rebind an app, stored code, programs, and triggers break. For more information
about binding custom schemas, see Use Custom Schemas.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
202
2. Enter the following command and wait for a Service instance not found error indicating
that the instance no longer exists:
watch cf service SERVICE-INSTANCE
Purge a Service Instance
If the service instance VM is lost, then you cannot delete the service instance. However, you can
use the cf CLI to purge a service instance from the Cloud Controller database.
To purge a service instance:
1. Run:
cf purge-service-instance SERVICE-INSTANCE-NAME
Where SERVICE-INSTANCE-NAME is the name of the service instance to purge.
For example:
Using VMware Tanzu SQL with MySQL for VMs for Multi-
Site Replication
This topic provides instructions for developers using the VMware Tanzu SQL with MySQL for VMs
for replication across multiple foundations or data centers.
Overview
In Tanzu SQL for VMs v2.7.4 and later, you provision a leader-follower service instance across two
foundations using the Multi‑Site Replication topology. This leader-follower service instance is
$ cf delete-service my-instance
Are you sure you want to delete the service my-instance ? y
Deleting service my-service in org my-org / space my-space as user@exam
ple.com...
OK
$ cf purge-service-instance my-instance
WARNING: This operation assumes that the service broker responsible for
this service instance is no longer available or is not responding with
a 200 or 410, and the service instance has been deleted, leaving orphan
records in Cloud Foundry's database. All knowledge of the service insta
nce is removed from Cloud Foundry, including service bindings and servi
ce keys.
Really purge service instance my-instance from Cloud Foundry?> y
Purging service my-instance...
OK
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
203
comprised of two single node instances that are configured for replication.
For more information about the Multi‑Site Replication topology, see About Multi‑Site Replication.
Prerequisites
Before you use Tanzu SQL for VMs across multiple foundations, you must have:
Access to the Multi‑Site Replication plan. Your operator configures this plan and enables
access to it in your org and space.
Selected two foundations to deploy your Multi‑Site Replication leader and follower VMs.
Your operators must configure these foundations for multi‑site replication. For more
information configuring a Multi‑Site Replication topology, see Preparing for Multi‑Site
Replication.
Create a Multi‑Site Replication Leader-Follower Service
Instance
To create a leader-follower service across multiple foundations:
1. Check the availability of the Multi‑Site Replication plan in the Marketplace in both your
foundations. See Confirm the VMware Tanzu SQL with MySQL for VMs Service Availability.
2. Create one Multi‑Site Replication service instance on each foundation. See Create
Multi‑Site Replication Service Instances.
3. Enable replication between the Multi‑Site Replication service instances. See Configure
Multi‑Site Replication.
4. Bind the Multi‑Site Replication leader-follower service instance to your apps. See Bind a
Multi‑Site Replication Leader-Follower Service Instance to Your App.
5. Modify your app to use the Tanzu SQL for VMs service. See Use the MySQL Service in Your
App.
After you create a Multi‑Site Replication leader-follower service instance, you can manage it over
the life cycle of your apps and data. For instructions on how to manage a Tanzu SQL for VMs
service instance, see Manage Service Instances.
Create Multi‑Site Replication Service Instances
To create a leader-follower service instance across two foundations, you must create one Multi‑Site
Replication service instance in each foundation. You must configure the service instances in each
foundation for replication.
Note: The Multi‑Site Replication plan type is configured separately from the leader-
follower service plan.
Note: The secondary foundation is the foundation that the follower VM is deployed,
and usually is your disaster recovery site.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
204
To create a multi-site replication service instance in both foundations:
1. Create a Multi‑Site Replication service instance in your primary foundation:
1. Log in to the deployment for your primary foundation by running:
cf login PRIMARY-API-URL
Where PRIMARY-API-URL is the API endpoint for the primary foundation.
2. Create a primary service instance by running:
cf create-service p.mysql PLAN PRIMARY-INSTANCE
Where:
PLAN is the name of the Multi‑Site Replication plan you want to use.
PRIMARY-INSTANCE is a name you choose to identify the service instance. This
name appears under service in output from cf services.
For example:
3. (Optional) Watch the progress of the service instance creation by running:
watch cf services
Wait for the last operation for your instance to show as create succeeded.
For example:
If you get an error, see Troubleshooting Instances.
$ cf create-service p.mysql db-small primary-node
Creating service primary-node in org my-org / space my-space as a
dmin...
OK
Note: Do not name your service instance leader or follower. If you
trigger a failover or switchover, the service instances in your primary
and secondary foundations switch roles. For more information, see
Triggering Multi-Site Replication Failover and Switchover.
$ watch cf services
Getting services in org my-org / space my-space as admin...
OK
name service plan bound apps last opera
tion
primary-node p.mysql db-small create suc
ceeded
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
205
2. Create a Multi‑Site Replication service instance in your secondary foundation by repeating
step 1 and replacing references to primary with secondary. Ensure that you log in to
deployment for your secondary foundation.
Configure Multi‑Site Replication
After you create the Multi‑Site Replication service instance in primary and secondary foundations,
you must configure replication between the two service instances.
You configure replication using service keys to pass connection information between the leader
and follower VMs. You must not use these service keys for any other use case besides establishing
multi‑site replication.
Workflow for Configuring Multi‑Site Replication
The following diagram describes the workflow for configuring multi‑site replication:
View a larger version of this diagram
The steps shown in the diagram are as follows:
1. Create host-info service key.
2. Record host-info service key.
3. Update secondary service instance with host-info service key.
4. Create credentials service key.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
206
5. Record credentials service key.
6. Update primary service instance with credentials service key.
Procedure for Configuring Multi‑Site Replication
To configure replication for your Multi‑Site Replication leader-follower service instance:
1. Create a host-info service key for the service instance in your secondary foundation by
running:
cf create-service-key SECONDARY-INSTANCE SERVICE-KEY \
-c '{"replication-request": "host-info"}'
Where:
SECONDARY-INSTANCE is the name of the secondary service instance you created in
step 2 of Create Multi‑Site Replication Service Instances.
SERVICE-KEY is a name you choose for the host-info service key.
For example:
2. View the replication-credentials for your host-info service key by running:
cf service-key SECONDARY-INSTANCE SERVICE-KEY
Where:
SECONDARY-INSTANCE is the name of the secondary service instance you created in
step 2 of Create Multi‑Site Replication Service Instances.
SERVICE-KEY is the name of the host-info service key you created in step 1.
For example:
Note: The following procedure assumes you created the leader service instance in
the primary foundation and the follower service instance in the secondary
foundation. You created these service instances in Create Multi‑Site Replication
Service Instances.
$ cf create-service-key secondary-node host-info \
-c '{"replication-request": "host-info" }'
Creating service key host-info for service instance secondary-node
as admin...
OK
$ cf service-key secondary-node host-info-key
Getting key host-info-key for service instance secondary-node as admi
n...
{
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
207
3. Record the output of the previous command, and remove the top-level credentials key.
4. Log in to the deployment for your primary foundation by running:
cf login PRIMARY-API-URL
5. Update your primary service instance with the host-info service key by running:
cf update-service PRIMARY-INSTANCE -c HOST-INFO
Where:
PRIMARY-INSTANCE is the name of the primary service instance you created in step 1
of Create Multi‑Site Replication Service Instances.
HOST-INFO is the output you recorded in step 3.
For example:
"credentials": {
"replication": {
"peer-info": {
"hostname": "secondary.bosh",
"ip": "10.0.19.12",
"system_domain": "sys.secondary-domain.com",
"uuid": "ab12cd34-5678-91e2-345f-67891h234567"
},
"role": "leader"
}
}
}
Caution
In cf CLI v8, the response includes a top-level credentials key. Earlier
versions of the cf CLI do not include a top-level credentials key. This
procedure assumes that you are using cf CLI v8.
$ cf update-service primary-node -c '{"replication":{ \
"peer-info":{ \
"hostname": "secondary.bosh", \
"ip": "10.0.18.12", \
"system_domain": "sys.secondary-domain.com", \
"uuid": "ab12cd34-5678-91e2-345f-67891h234567" \
},\
"role": "leader" \
} \
}'
Updating service instance primary-node as admin...
OK
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
208
6. Watch the progress of the service instance by running:
watch cf services
Wait for the last operation for your instance to show as update succeeded.
For example:
If you get an error, see Troubleshooting Instances.
7. Create a credentials service key for the service instance in your primary foundation by
running:
cf create-service-key PRIMARY-INSTANCE SERVICE-KEY-NAME \
-c '{"replication-request": "credentials"}'
Where:
PRIMARY-INSTANCE is the name of the primary service instance you created in step 1
of Create Multi‑Site Replication Service Instances.
SERVICE-KEY-NAME is a name you choose for the credentials service key.
For example:
8. View the replication-credentials for your credentials service key by running:
cf service-key PRIMARY-INSTANCE SERVICE-KEY-NAME
Where:
PRIMARY-INSTANCE is the name of the primary service instance you created in step 1
of Create Multi‑Site Replication Service Instances.
SERVICE-KEY-NAME is the name of the credentials service key you created in step 6.
For example:
$ watch cf services
Getting services in org my-org / space my-space as admin...
OK
name service plan bound apps last operation
primary-node p.mysql db-small update succeeded
$ cf create-service-key primary-node cred-key \
-c '{"replication-request": "credentials" }'
Creating service key cred-key for service instance primary-node as ad
min...
OK
Note: The -c flag is different from the one in step 1.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
209
9. Record the output of the previous command, and remove the top-level credentials key.
10. Log in to the deployment for your secondary foundation by running:
cf login SECONDARY-API-URL
11. Update your secondary service instance with the credentials service key by running:
cf update-service SECONDARY-INSTANCE -c CREDENTIALS
Where:
SECONDARY-INSTANCE is name of the secondary service instance you created in step 2
of Create Multi‑Site Replication Service Instances.
CREDENTIALS is the output you recorded in step 8.
For example:
$ cf service-key primary-node cred-key
Getting key cred-key for service instance primary as admin...
{
"credentials": {
"replication": {
"credentials": {
"password": "a22aaa2a2a2aaaaa",
"username": "6bf07ae455a14064a9073cec8696366c"
},
"peer-info": {
"hostname": "primary.bosh",
"ip": "10.0.17.12",
"system_domain": "sys.primary-domain.com",
"uuid": "zy98xw76-5432-19v8-765u-43219t876543"
},
"role": "follower"
}
}
}
Caution
In cf CLI v8, the response includes a top-level credentials key. Earlier
versions of the cf CLI do not include a top-level credentials key. This
procedure assumes that you are using cf CLI v8.
$ cf update-service secondary-node -c '{"replication": { \
"credentials": { \
"password": "a22aaa2a2a2aaaaa", \
"username": "6bf07ae455a14064a9073cec8696366c" \
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
210
You now have a Multi‑Site Replication leader-follower service instance that is fully configured and
has replication enabled.
Bind a Multi‑Site Replication Leader-Follower Service Instance to
Your App
For an app to use a Multi‑Site Replication leader-follower service instance, you must bind your app
to your primary service instance in your primary foundation. If you want to use an active-active
topology, you must additionally bind your app to the secondary service instance in your secondary
foundation.
For information about active-passive and app-layer active-active topologies, see About Active-
Passive Topology and About App-Layer Active-Active Topology.
To bind an app to a leader-follower service instance:
1. Log in to the deployment for your primary foundation by running:
cf login PRIMARY-API-URL
2. Bind your app to your primary service instance by doing the procedure in Bind a Service
Instance to Your App.
3. (Optional) If you are using an active-active topology, you must bind the same app to your
secondary service instance in your secondary foundation. To do this, repeat the previous
steps and replace references to primary with secondary.
4. Modify your app to use the Tanzu SQL for VMs service by using the procedure in Use the
MySQL Service in Your App.
Using TLS
This topic describes how developers can use TLS to secure the communication from their apps and
local workstations to the VMware Tanzu SQL with MySQL for VMs service.
Overview
}, \
"peer-info": { \
"hostname": "primary.bosh", \
"ip": "10.0.17.12", \
"system_domain": "sys.primary-domain.com", \
"uuid": "zy98xw76-5432-19v8-765u-43219t876543" \
}, \
"role": "follower" \
} \
}'
Updating service instance primary-node as admin...
OK
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
211
For MySQL for Pivotal Cloud Foundry v2.5 and later and Tanzu SQL for VMs 2.7 and later, if your
operator has configured TLS in the tile, but did not configure TLS for existing service instances in
MySQL for Pivotal Cloud Foundry v2.4 and earlier, you must activate TLS using the procedures in
Activate TLS.
If your operator has configured TLS in the tile, new service instances have TLS activated by default.
In this case, developers don’t have to activate TLS.
After TLS is activated for your service instance, you can establish a TLS connection from your local
workstation to a Tanzu SQL for VMs service instance.
For more information on how to establish a TLS connection, see Establish a TLS Connection to a
Service Instance.
Activate TLS
The procedure for updating your app depends on the language and framework of your app. Java
and Spring apps automatically detect TLS. Apps written in other languages and frameworks must
be manually modified to use TLS.
To activate TLS on existing service instances, do one of the following:
If your app is written in Java or Spring, see Activate TLS for Java and Spring Apps.
If your app is not written in Java or Spring, see Activate TLS for Non-Spring Apps.
Prerequisites
To activate TLS for service instances, an operator must:
Complete the procedure in Preparing for TLS.
Activate TLS in the tile configuration when doing the procedure in Configure Security.
Activate TLS for Java and Spring Apps
In MySQL for Pivotal Cloud Foundry v2.5, if your operator has configured TLS in the tile, new
service instances have TLS activated by default. If your Spring app detects TLS configured in the
service instance, it must connect over TLS.
If you did not previously activate TLS in your service instance before upgrading to MySQL for
Pivotal Cloud Foundry v2.5, you must rebind your Spring apps in order to re-establish connections
to your service instance.
Warning: Mutual TLS (mTLS) is not supported in Tanzu SQL for VMs. Because of
this, the server certificate does not validate apps. If an app presents a certificate to
the MySQL server, the connection closes and a network error appears in the app
logs. To resolve this issue, you must deactivate mTLS in your apps.
Note: VMware recommends developers configure their apps to use the MySQL
Connector/J v5.1.42 or later instead of the MariaDB Connector/J.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
212
Rebind your App
If your app is bound to an existing service instance, you must rebind it after activating TLS for the
instance.
To rebind your app:
1. To stop your app, run the following command:
cf stop YOUR-APP
Where YOUR-APP is the name of your app.
2. To unbind your app from the service instance, run the following command:
cf unbind-service YOUR-APP YOUR-SERVICE-INSTANCE
Where:
YOUR-APP is the name of your app.
YOUR-SERVICE-INSTANCE is the name of your service instance.
3. To rebind your app to the service instance, run the following command:
cf bind-service YOUR-APP YOUR-SERVICE-INSTANCE
Where:
YOUR-APP is the name of your app.
YOUR-SERVICE-INSTANCE is the name of your service instance.
4. To restage your app, run the following command:
cf restage YOUR-APP
Where YOUR-APP is the name of your app.
Your app now communicates securely with the MySQL service instance.
Activate TLS for Non-Spring Apps
In order to activate TLS for apps not written in Java or Spring, you must modify the app to discover
the CA certificate in VCAP_SERVICES and specify the CA component when initiating the connection
to the database.
VCAP_SERVICES is an environment variable that exists within every container. It contains runtime-
specific information about the app, including metadata supplied by each of the services that are
bound to that app.
Warning: If a developer rebinds an app to the Tanzu SQL for VMs service after
unbinding, they must also rebind any existing custom schemas to the app. When
you rebind an app, stored code, programs, and triggers break. For more information
about binding custom schemas, see Use Custom Schemas.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
213
The metadata includes the information needed to connect to the service, such as hostnames,
usernames, and passwords.
To activate TLS for your app, do the following:
1. Modify your app to retrieve the hostname, username, password, database name, and CA
certificate for the bound Tanzu SQL for VMs service instance from the VCAP_SERVICES
environment variable.
For example, the following Node.js code initializes a variable named mysql_creds, and then
populates it with the necessary information from VCAP_SERVICES:
var mysql_creds = {} ;
var vcap_services = undefined ;
if (process.env.VCAP_SERVICES) {
vcap_services = JSON.parse(process.env.VCAP_SERVICES) ;
mysql_creds["host"] = vcap_services["p.mysql"][0]["credentials"]["hostnam
e"] ;
mysql_creds["user"] = vcap_services["p.mysql"][0]["credentials"]["usernam
e"] ;
mysql_creds["password"] = vcap_services["p.mysql"][0]["credentials"]["passw
ord"] ;
mysql_creds["port"] = vcap_services["p.mysql"][0]["credentials"]["port"] ;
mysql_creds["database"] = vcap_services["p.mysql"][0]["credentials"]["nam
e"] ;
if (vcap_services["p.mysql"][0]["credentials"]["tls"]) {
mysql_creds["ca_certificate"] = vcap_services["p.mysql"][0]["credential
s"]["tls"]["cert"]["ca"];
} else {
mysql_creds["ca_certificate"] = undefined ;
}
}
2. Modify your app to use the hostname, username, password, and CA certificate to establish
a secure connection with the bound Tanzu SQL for VMs service instance.
For example, the following Node.js function establishes a TLS connection with the MySQL
service, using the information loaded into mysql_creds:
function MySQLConnect() {
clientConfig = {
host : mysql_creds["host"],
user : mysql_creds["user"],
password : mysql_creds["password"],
port : mysql_creds["port"],
database : mysql_creds["database"]
} ;
if (mysql_creds["ca_certificate"]) {
clientConfig["ssl"] = { ca : mysql_creds["ca_certificate"] } ;
}
dbClient = mysql.createConnection( clientConfig ) ;
dbClient.connect(CALLBACK-FUNCTION) ;
}
3. Push your app with cf push.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
214
Establish a TLS Connection to a Service Instance
You can use mysql to establish a TLS connection to a Tanzu SQL for VMs service instance that has
TLS activated.
For more information about how to activate TLS for a service instance, see Activate TLS.
To establish a TLS connection to a service instance:
1. Create a new service key for the service instance with TLS activated. For example:
If the service key does not have a CA certificate under tls.cert.ca, the service key might
be stale. Create a new service key.
2. Copy the contents of the CA certificate under tls.cert.ca and paste it into a file. For
example:
1. Record the values for username, password, and hostname.
2. Use mysql to establish a TLS connection to the MySQL instance. Run the following
command:
mysql –host=HOSTNAME
–user=USERNAME
–password=PASSWORD
–ssl-ca=root.pem
–ssl-verify-server-cert
Where:
$ cf create-service-key my-service-instance my-tls-service-key
{
"hostname": "q-n3s3y1.q-g693.bosh",
"jdbcUrl": "jdbc:mysql://q-n3s3y1.q-g693.bosh:3306/service\_instance\_d
b?user=6bf07ae455a14064a9073cec8696366c\u0026password=a22aaa2a2a2aaaaa
\u0026=true",
"name": "service\_instance\_db",
"password": "a22aaa2a2a2aaaaa",
"port": 3306,
"tls": {
"cert": {
"ca": "-----BEGIN CERTIFICATE-----\...n-----END CERTIFICATE-----\n"
}
},
"uri": "mysql://6bf07ae455a14064a9073cec8696366c:a22aaa2a2a2aaaaa@q-n3s
3y1.q-g693.bosh:3306/service\_instance\_db?reconnect=true",
"username": "6bf07ae455a14064a9073cec8696366c"
}
$ pbpaste > root.pem
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
215
HOSTNAME is the value for hostname previously retrieved.
USERNAME is the value for username previously retrieved.
PASSWORD is the value for password previously retrieved.
For example:
About Data Migration in VMware Tanzu SQL with MySQL for
VMs
This topic explains how data migration works in VMware Tanzu SQL with MySQL for VMs.
Read this topic before you do the procedures in Migrating Data in Tanzu SQL for VMs.
The migrate Command
You can use the mysql-tools CLI plug-in to migrate MySQL data with the migrate command. The
migrate command does a streaming mysqldump and restore to migrate data from your source
MySQL database to a destination Tanzu SQL for VMs v2 instance.
The command supports connections over TLS. If TLS is configured in the source and destination
MySQL instances, the data is securely streamed using TLS. For information about how to configure
TLS in a Tanzu SQL for VMs v2 service instance, see Using TLS.
During data migration, the migrate command:
1. Creates a new Tanzu SQL for VMs v2 service instance in the destination space with the
same name as the source MySQL service instance name.
2. Copies the source data over to the new service instance.
3. Appends -old to the source service instance name, regardless if the source is a Tanzu SQL
for VMs service instance or a user-provided service instance.
Use Cases
The migrate command is used for most migration use cases. See Use Cases Requiring the migrate
Command.
However, many common migrations, such as from a small to a large database, can be done with the
simpler update-service command. See Use Cases Not Requiring the migrate Command.
If your use case is not listed in this topic, you might need to manually back up and restore your
database to migrate your data. See Backing Up and Restoring with mysqldump.
Use Cases Requiring the migrate Command
$ mysql --hostname=q-n3s3y1.q-g693.bosh \
--user=6bf07ae455a14064a9073cec8696366c \
--password=a22aaa2a2a2aaaaa \
--ssl-ca=root.pem \
--ssl-verify-server-cert
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
216
The following table lists migration use cases that must be done with the migrate command:
Use migrate for migrating from... To...
Single Node
Highly Available (HA) Cluster
Multi-Site
Leader-Follower
HA Cluster
Multi-Site
HA Cluster
Leader-Follower
Single Node
Multi-Site
Single Node
Leader-Follower
Off-Platform Database * Tanzu SQL for VMs v2
Availability Zone (AZ) Another AZ
MySQL for Pivotal Cloud Foundry v1 Tanzu SQL for VMs v2
* If your off-platform database has encryption at rest or the Percona PAM Authentication plug-in
enabled, you cannot use the migrate command. Instead, you must follow the procedure in Restore
from an Off-Platform Logical Backup.
Use Cases Not Requiring the migrate Command
Not all migrations require the migrate command. For example, if you are migrating a database to a
larger single node or leader-follower plan, use the simpler update-service command.
The following table lists migration use cases that can be done with the update-service command.
For instructions about using the update-service command, see Update a Service Instance.
Use update-service for migrating from... To...
Single Node larger Single Node
Leader-Follower larger Leader-Follower
Single Node Leader-Follower of the same or larger size
Leader-Follower Single Node of the same or larger size
Omitted Data
The migrate command skips the following system schemas:
Warning: The migrate command does not migrate all stored programs. The
command only migrates views and does not migrate triggers, routines, or events.
To manually migrate triggers, routines, or events, contact Support.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
217
cf_metadata: This is metadata for binding information, such as database users mapped to
GUIDs.
information_schema: This is metadata for the MySQL database.
mysql: This is a MySQL database that describes users, user accounts, and permissions.
Tanzu SQL for VMs uses it to authenticate users.
performance_schema: This is metadata about performance and server execution.
sys: This is a schema that summarizes performance_schema.
Migrating Data in VMware Tanzu SQL with MySQL for VMs
This topic describes how to migrate data from any MySQL database to a VMware Tanzu SQL with
MySQL for VMs service instance.
Overview
You can migrate data from a source database to a destination database using the migrate
command.
To migrate data in Tanzu SQL for VMs:
1. Enable the migrate command to access the source database. See Enable Source Access.
2. Migrate the data from the source database to the destination service instance. See Migrate
Data.
3. Ensure that the migration was successful. See Validate Data.
4. Rebind and re-stage apps to the new destination service instance. To save resources, you
can optionally delete the old source database. See Rebind and Re-Stage Apps.
Prerequisites
Before you do the procedures in this topic, you must have:
Read about the migrate command in About Data Migration in Tanzu SQL for VMs.
Note: sys, performance_schema, and information_schema change dynamically with
changes on a Tanzu SQL for VMs service instance. These schemas are not migrated
because the destination database has its' own version of the schemas. You do not
need to back up these schemas.
Warning: Migrating large datasets can take several hours. Data migration is linear
and depends on the hardware being used. For example, if X amount of data takes
10 minutes to migrate, then 2X amount of data takes 20 minutes to migrate using
the same hardware.
Do a test migration with small datasets to estimate how long the entire migration
takes before migrating larger datasets.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
218
An existing MySQL database that is the source of the data you want to migrate from.
MySQL source databases can be:
a MySQL for Pivotal Cloud Foundry v1 service instance
a VMware Tanzu SQL with MySQL for VMs v2 service instance
a database that is not a Tanzu SQL for VMs database
Tanzu SQL for VMs v2 installed in the destination environment you want to migrate to.
A Tanzu SQL for VMs service plan available in the destination org and space you want to
migrate your data to. This service plan must fulfill the requirements in Resource Planning.
Talk to your operator to determine which service plan is appropriate.
Resource Planning
You must ensure that the Tanzu SQL for VMs v2 service plan for your destination service instance
has your preferred VM type and persistent disk size. If the plan does not enough space on disk to
store the data, migration fails.
When choosing a service plan for your destination service instance, if you select:
A single node or leader-follower plan, ensure the persistent disk size is three times larger
than the size of the source data.
A HA cluster plan, ensure the persistent disk size is two times larger than the size of the
source data.
For more information about recommended persistent disk sizes, see Persistent Disk Usage.
Install the mysql-tools CF CLI Plug-in
VMware recommends that developers migrate data using the mysql-tools CLI plug-in with the
migrate command. For more information about mysql-tools cf CLI plug-in, see mysql-cli-plugin in
GitHub.
To install the mysql-tools cf CLI plug-in:
1. Install the plug-in by running:
cf install-plugin -r CF-Community "MysqlTools"
2. Confirm that the plug-in has successfully installed, run:
cf mysql-tools version
Note: To view available service plans, run cf marketplace. Tanzu SQL for VMs v2
appears as p.mysql and MySQL for Pivotal Cloud Foundry v1 appears as p-mysql.
Note: You must have TLS activated to use this feature. To configure and activate
TLS, see Configure TLS.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
219
Enable Source Access
The migrate command must be able to access the source database. How you activate this access
depends on where the source database is located relative to the destination space and org.
If you are:
Migrating within an Org and Space: The migrate command can access the source
without any preparation. Continue to Migrate Data.
Migrating across Spaces: If the source database is in a different space from the
destination, enable access using service instance sharing. See Source Access across Spaces.
Migrating from Off-Platform: If the source database is in a different Ops Manager
foundation from the destination or not deployed on Ops Manager, create a user-provided
service that can access the remote database. See Source Access Off-Platform.
Source Access across Spaces
If your source MySQL service instance is in a different development space from your destination
org and space, you can migrate your data by sharing the service instance to the destination org and
space. Service instance sharing is enabled by default.
To share a source MySQL service instance with your destination org and space:
1. Do the procedure in Share Service Instances.
2. Continue to Migrate Data.
Source Access Off-Platform
If your source MySQL database is in a different Ops Manager foundation or not deployed on Ops
Manager, you can migrate your data by creating a local user-provided service instance that can
access the database.
For more information on user-provided service instances, see User-Provided Service Instances.
To create a user-provided service instance to access the off-platform database:
1. Confirm that your off-platform MySQL database permits inbound and outbound network
traffic to your destination Ops Manager foundation. You might need to modify firewall rules
for your off-platform MySQL database. Talk to your platform operator for assistance.
2. If your off-platform MySQL database requires connections over TLS, verify that your Ops
Manager foundation is configured to recognize the CA certificate that the MySQL server
certificate signed in with. Talk to your platform operator for assistance.
3. Record the information needed to access your off-platform database. These values usually
include:
hostname: the domain name or IP address of the off-platform source database.
name: the name of source database.
username and password: the database account credentials.
port: The port number for the database. This number is usually 3306.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
220
If your off-platform database is a Tanzu SQL for VMs service instance, these values are in
your VCAP_SERVICES environment variable credentials hash. For more information, see
MySQL Environment Variables.
4. Create a Cloud Foundry user-provided service instance that can access the off-platform
database:
cf cups CF-DB-INSTANCE -p CREDS-STRUCT
Where:
CF-DB-INSTANCE is the name that you want to give to the new database service
instance that you are migrating to.
CREDS-STRUCT is a JSON structure that contains the off-platform database access
values you recorded in the previous step.
For example:
5. After your user-provided service instance is created, continue to Migrate Data.
Migrate Data
After your source database can access the destination space:
If your source MySQL database is running on Ops Manager, you must stop all traffic to
the service instance before you migrate your data. You can do this by stopping and
unbinding all of your apps. See Stop and Unbind Apps.
If your source MySQL database is running off-platform, do the procedure in Migrate
Data to Destination Instance.
Stop and Unbind Apps
To stop and unbind your apps:
1. Log in to your BOSH deployment by running:
cf login API-URL
When prompted, enter your credentials.
$ cf cups migrating-db -p '{"hostname": "34.192.88.212", "name": "my\_d
b", \
"username": "root", "password": "P455w0rD", "port": 3306}'
Creating user provided service migrating-db in org my-org / space my-sp
ace as admin...
OK
Note: cf cups is a shortcut for the cf create-user-provided-service
command.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
221
2. Target the org and space for the new destination service instance by running:
cf target -o DESTINATION-ORG -s DESTINATION-SPACE
3. Retrieve and record a list of your bound apps by running:
cf services
Your apps are listed the bound apps column.
4. For each bound app you recorded in the previous step:
1. Stop the app by running:
cf stop APP
2. Unbind the app by running:
cf unbind-service APP SOURCE-INSTANCE
Where:
APP is the name of your app.
SOURCE-INSTANCE is the name of your source Tanzu SQL for VMs service
instance.
5. Do the procedure in Migrate Data to Destination Instance.
Migrate Data to Destination Instance
To migrate data from your source database to your destination service instance:
1. Log in to your BOSH deployment and target the destination org and space by doing steps 1
and 2 of Stop and Unbind Apps.
2. View and select an available Tanzu SQL for VMs v2 service plans by running:
cf marketplace
Tanzu SQL for VMs v2 service plans are under p.mysql.
3. Migrate your data by running:
cf mysql-tools migrate SOURCE-INSTANCE V2-PLAN
Where:
SOURCE-INSTANCE is the name of your source MySQL service instance or user-
provided service instance.
V2-PLAN is the name of the service plan that you previously selected.
For example:
$ cf mysql-tools migrate my-instance db-small
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
222
Validate Data
After migrating your data, you must verify that the data was successfully migrated by validating the
data in the new Tanzu SQL for VMs v2.x service instance.
You can validate the data by creating an SSH tunnel to gain direct command line access to the new
Tanzu SQL for VMs v2.x service instance.
To create an SSH tunnel to the instance and validate your data:
1. Create an SSH tunnel to your Tanzu SQL for VMs v2.x service instance by doing the
following procedures in Accessing Services with SSH:
1. Push Your Host App
2. Create Your Service Key
3. Configure Your SSH Tunnel
4. Access Your Service Instance
2. From the MySQL shell, validate that the data that you expect to see has been imported
into the Tanzu SQL for VMs v2.x service instance.
3. Exit the MySQL shell and stop the SSH tunnel.
Rebind and Re-Stage Apps
To complete the migration, rebind and re-stage any bounds apps in your destination org and space.
After rebinding and re-staging your apps, VMware recommends deleting the old source database
instance to save resources.
2018/04/24 11:31:19 Creating new service instance "my-instance" for ser
vice p.mysql using plan db-small
2018/04/24 11:41:01 Unpacking assets for migration to /var/folders/dm/6
6n2j9xx02l8vs58q2whz4080000gn/T/migrate\_app\_101341527
2018/04/24 11:41:02 Started to push app
Done uploading
2018/04/24 11:41:09 Successfully pushed app
2018/04/24 11:41:10 Successfully bound app to v1 instance
2018/04/24 11:41:12 Successfully bound app to v2 instance
2018/04/24 11:41:12 Starting migration app
2018/04/24 11:41:25 Started to run migration task
2018/04/24 11:41:27 Migration completed successfully
2018/04/24 11:41:29 Cleaning up...
Note: For debugging purposes, you can add the --no-cleanup flag to the
previous command. If a migration fails, this flag preserves the migration app
and the newly-created service instance. However, if a migration succeeds,
the migration app is still deleted.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
223
To rebind and re-stage your apps and delete the source database instance:
1. Bind the app to the new service instance by running
cf bind-service APP V2-INSTANCE
Where:
APP is the name of your app.
V2-INSTANCE is the name of your Tanzu SQL for VMs v2.x service instance.
For example:
2. Re-stage the app by running:
cf restage APP
The app is now using your new Tanzu SQL for VMs v2.x service instance and should be
operational again.
3. (Optional) Delete your source database. If your source database is deployed on Ops
Manager, delete the old database instance by running in your source space and org:
cf delete-service SOURCE-INSTANCE
Where SOURCE-INSTANCE is the name of your source Tanzu SQL for VMs service instance.
About MySQL Server Defaults
This topic provides information about the defaults that VMware Tanzu SQL with MySQL for VMs
applies to its Percona Server components.
Overview
Learn about the server defaults for Tanzu SQL for VMs service plans. Most of the server defaults
are the same for all service plans, however, some server defaults differ depending on the service
plan.
For server defaults that are:
Warning: If a developer rebinds an app to the Tanzu SQL for VMs service after
unbinding, they must also rebind any existing custom schemas to the app. When
you rebind an app, stored code, programs, and triggers break. For more information
about binding custom schemas, see Use Custom Schemas.
$ cf bind-service my-app my-v2-instance
Binding service my-v2-instance to app my-app in org my-org / space my-s
pace as [email protected]...
OK
TIP: Use 'cf restage my-app' to ensure your env variable changes take e
ffect
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
224
Common to all plans, see Server Defaults for All Plans.
Specific to single node and leader-follower plans, see Server Defaults for Single Node and
Leader-Follower Plans.
Specific to highly available (HA) cluster plans, see Server Defaults for Highly Available
Cluster Plans.
Specific to multi‑site replication plans, see Server Defaults for Multi‑Site Replication Plans.
Server Defaults for All Plans
The following table lists the Tanzu SQL for VMs server defaults that are common to all service
plans:
Name Variable Name Default Notes
Audit Log audit-log OFF To set to ON, select Enable Server Activity Logging
in Monitoring. Logs are written in JSON to
/var/vcap/store/mysql_audit_logs/mysql_serve
r_audit.log.
Character
Set
character-set-server utf8 This setting defaults all character sets. You can
override this during a session.
Binary Log
Removal
expire_log_days 3 This setting is the number of days before binary log
files are automatically removed.
For more information, see the MySQL
documentation.
InnoDB
Transaction
Log
Durability
innodb_flush_log_at_trx
_commit
1 At each transaction commit, logs are written and
flushed to disk.
For more information, see the MySQL
documentation.
InnoDB
Flush
Method
innodb_flush_method fsync This setting defines the method used to flush data to
InnoDB data and log files.
For more information, see the MySQL
documentation.
InnoDB
Auto
Increment
Lock Mode
innodb-autoinc-lock-
mode
2 This setting uses the interleaved mode. This enables
multiple statements to execute at the same time.
There can be gaps in auto-incrementing columns.
InnoDB
Buffer Pool
Size
innodb-buffer-pool-size 50% of the
available memory
on each service
instance
This setting is dynamically configured to be 50% of
the available memory on each service instance.
InnoDB Log
Buffer Size
innodb-log-buffer-size 32 MB This setting defaults to 32 MB to avoid excessive
disk I/O when issuing large transactions.
Note: You can use optional parameters to change certain server defaults. For more
information, see Changing Defaults Using Arbitrary Parameters.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
225
Log Bin
Trust
Function
Creators
log-bin-trust-function-
creators
ON This setting relaxes constraints on how MySQL
writes stored procedures to the binary log.
For more information, see the MySQL
documentation.
Lower Case
Table
Names
lower-case-table-names 0 By default, all table names are case sensitive.
Operators can change this default setting on the
MySQL Configuration page and permit developers
to override the default when they create a service
instance.
For more information about using lowercase table
names, see the MySQL documentation.
Max
Allowed
Packet
max-allowed-packet 256 MB If necessary, you can change this size setting in a
session variable.
Reverse
Name
Resolution
skip-name-resolve ON This deactivates reverse DNS lookups to improve
performance. Tanzu SQL for VMs uses user
credentials, rather than hostnames, to authenticate
access. Therefore, most deployments do not need
reverse DNS lookups.
To activate reverse name resolution, deselect this
option.
Skip
Symbolic
Links
symbolic-links OFF Tanzu SQL for VMs is configured to prevent the use
of symlinks to tables. VMware recommends this
security setting to prevent users from manipulating
files on the server file system.
For more information, see Making MySQL Secure
Against Attackers.
Table
Definition
Cache
table-definition-cache 8192 For information about changing this setting, see the
MySQL documentation.
Server Defaults for Single Node and Leader-Follower Plans
In addition to the server default settings that are common to all plans, single node and leader-
follower plans use the server defaults listed in the following table:
Name Variable Name Default Notes
Max
Connections
max-connections 750 connections
per service
instance
System processes count towards this limit.
MyISAM
Recover
Options
myisam-recover-options BACKUP, FORCE This setting enables Tanzu SQL for VMs to recover
from most MyISAM problems without human
intervention.
For more information, see the MySQL
documentation.
Event
Scheduler
event-scheduler ON Tanzu SQL for VMs enables the event scheduler so
users can create and use events in their dedicated
service instances.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
226
InnoDB Log
File Size
innodb-log-file-size 256 MB Tanzu SQL for VMs clusters default to a log-file size
of 256 MB.
Collation
Server
collation-server utf8_general_ci You can override this during a session.
For instructions about viewing available and default
collations, see the MySQL documentation.
Relay Log
Recovery
relay-log-recovery ON When enabled, relay log recovery happens
automatically after server startup.
For more information, see the MySQL
documentation.
Server Defaults for Highly Available Cluster Plans
In addition to the server default settings that are common to all plans, HA cluster plans use the
server defaults listed in the following table:
Name Variable Name Default Notes
Max
Connections
max-connections 1500 connections
per service
instance
System processes count towards this limit.
MyISAM
Recover
Options
myisam-recover-options OFF This setting enables Tanzu SQL for VMs to recover
from most MyISAM problems without human
intervention.
For more information, see the MySQL
documentation.
Event
Scheduler
event-scheduler OFF Tanzu SQL for VMs enables the event scheduler so
users can create and use events in their dedicated
service instances.
InnoDB Log
File Size
innodb-log-file-size 1024 MB Tanzu SQL for VMs clusters default to a log-file size
of 256 MB.
Collation
Server
collation-server utf8_unicode_ci You can override this during a session.
For instructions about viewing available and default
collations, see the MySQL documentation.
Relay Log
Recovery
relay-log-recovery ON When enabled, relay log recovery happens
automatically after server startup.
For more information, see the MySQL
documentation.
Server Defaults for Multi‑Site Replication Plans
In addition to the server default settings that are common to all plans, multi‑site replication plans
use the server defaults listed in the following table.
Name Variable Name Default Notes
Max
Connections
max-connections 1500 connections
per service
instance
System processes count towards this limit.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
227
MyISAM
Recover
Options
myisam-recover-options OFF This setting activates Tanzu SQL for VMs to recover
from most MyISAM problems without human
intervention.
For more information, see the MySQL
documentation.
Event
Scheduler
event-scheduler OFF Tanzu SQL for VMs activates the event scheduler so
users can create and use events in their dedicated
service instances.
InnoDB Log
File Size
innodb-log-file-size 1024 MB Tanzu SQL for VMs clusters default to a log-file size
of 256 MB.
Collation
Server
collation-server utf8_unicode_ci You can override this setting during a session.
For instructions about viewing available and default
collations, see the MySQL documentation.
Relay Log
Recovery
relay-log-recovery OFF When enabled, relay log recovery happens
automatically after server startup.
For more information, see the MySQL
documentation.
Changing Defaults Using Arbitrary Parameters
This topic provides instructions for how to use optional parameters to change server defaults.
Overview
You can configure optional parameters to change certain VMware Tanzu SQL with MySQL for VMs
server defaults. You might want to configure optional parameters in the following cases:
You have a read-heavy or write-heavy app. See Workloads below.
You are migrating from a case-insensitive database. See Lowercase Table Names below.
You want to use a different character set or collation than the default. See Character Sets
below.
You want to configure leader-follower replication. See Synchronous Replication below.
The procedures in this topic use the Cloud Foundry Command Line Interface (cf CLI). You can also
use Apps Manager to perform the same tasks using a graphical UI.
Set Optional Parameters
You can change the default configuration of optional parameters by creating a new service instance
or updating an existing service instance.
To set optional parameters:
1. Do one of the following:
Note: Tanzu SQL for VMs service instances are configured by default with industry
best practices. For information about the configured server defaults, see About
MySQL Server Defaults.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
228
If you want to create a new service instance, run:
cf create-service p.mysql PLAN SERVICE-INSTANCE \
-c '{ "PARAMETER": "PARAMETER-VALUE"}'
If you want to update an existing service instance, run:
cf update-service p.mysql PLAN SERVICE-INSTANCE \
-c '{ "PARAMETER": "PARAMETER-VALUE"}'
For a list of available optional parameters, see Optional Parameters below. The -c
flag accepts a valid JSON object containing service-specific configuration
parameters, provided either in-line or in a file.
2. Verify that the cf command ran successfully by running:
watch cf services
Wait for the last operation for your instance to show as create succeeded.
For example:
Workloads
The following table describes how to use the workload optional parameter to adjust server default
settings for different workload profiles:
workload
Type String
Default mixed
Description Set this parameter to mixed, read-heavy or write-heavy. See Workload Profile Types.
Usage create-service or update-service
Workload Profile Types
The following table lists the workload profiles that developers can use to configure MySQL
instances based on their specific app workloads:
Profile Description
$ watch cf services
Getting services in org my-org / space my-space as [email protected]...\
OK
name service plan bound apps last operation
myDB p.mysql db-small create succeeded
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
229
Mixed Workload By default, each MySQL service instance is configured for a mixed workload. This workload is
equally heavy on reads and writes.
The configuration for this profile is described in About MySQL Server Defaults.
Read-Heavy
Workload
For apps that have a large number of reads, you can configure your service instances with a read-
heavy workload.
The read-heavy profile changes the following server defaults listed in About MySQL Server
Defaults:
innodb_buffer_pool_size is increased to 75% of the available memory on each service
instance.
innodb_flush_method is set to O_DIRECT.
Write-Heavy
Workload
For apps that write to the database frequently, you can configure your service instances with a
write heavy workload.
The write heavy profile changes the following server defaults that are listed in About MySQL Server
Defaults:
innodb_buffer_pool_size is increased to 75% of the available memory on each service
instance.
innodb_flush_method is set to O_DIRECT.
innodb_log_file_size is increased to 1 GB.
max_allowed_packets is increased to 1 GB.
Lowercase Table Names
If you are migrating a database from a system that was case insensitive, you can enable lowercase
table names to change all table names to lowercase.
For example, if your database had the table names TableName and TABLEname, when you enable
lowercase table names both of the names change to tablename and are interpreted as the same
table.
For more information, see the MySQL documentation.
The following table describes how to use the enable_lower_case_table_names optional parameter:
enable_lower_case_table_names
Type Boolean
Defaul
t
Set by the operator in the Mysql Configuration pane in the tile. See Configure MySQL.
Descri
ption
The operator can set a default for this parameter and permit developers to override the default.
If you set this to true, table names are stored in lowercase. See About Lowercase Table Names.
Usage create-service or update-service
Warning: Before you enable this feature, ensure all tables have lowercase names. Tables
with uppercase names are inaccessible after enabling lowercase table names.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
230
Character Sets
The following table describes how to use the default-charset and default-collation optional
parameters to change the character sets used in databases:
default-charset
Type String
Default utf8
Descripti
on
You can set this to any MySQL 5.7 supported character set. For information about character sets and
collations, see the MySQL documentation.
Usage create-service or update-service
default-collation
Type String
Default utf8_general_ci
Descripti
on
The default-collation changes based on the default-charset. To set the default-collation, first set
the default-charset.
For instructions for viewing available and default collations, see the MySQL documentation.
Usage create-service or update-service
Synchronous Replication for Leader-Follower
If you use a leader-follower service instance, Tanzu SQL for VMs supports synchronous replication
in addition to the default asynchronous replication. In sync replication, data does not get committed
to the leader node until the follower acknowledges the commit and can replicate it.
The guarantee of redundancy gives sync replication an advantage over asynchronous replication in
data integrity. However, depending on latency, sync replication reduces the performance of write
operations.
The following table describes how to use the replication_mode optional parameter:
replication_mode
Type String
Default async
Description Set this parameter to one of the following:
semi-sync: This enables sync replication on a leader-follower service instance.
async: This restores the default asynchronous replication for a leader-follower service instance.
Usage create-service or update-service
Note: replication_mode does not work for single node, HA cluster, or multi-site
replication plans.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
231
Synchronous Replication Timeout
By default, the timeout for sync replication is set to approximately 292 million years. Therefore, the
leader always waits for the follower to confirm receipt of the transaction. This guarantees that if the
leader is lost, a redundant copy of the data exists on the follower.
The following table describes how to use the semi_sync_ack_timeout_in_ms optional parameter:
semi_sync_ack_timeout_in_ms
Type Integer
Default
2
63
milliseconds
Description Sets the timeout in milliseconds for the leader to acknowledge a replication operation.
Usage create-service or update-service
Backup Schedule
The following table describes how to use the backup-schedule optional parameter:
backup-schedule
Type Cron expression
Defa
ult
The operator sets the default
Desc
ripti
on
Enter a cron expression using standard syntax. The cron expression sets the backup schedule for your service
instance. For example, entering a cron expression of 15 10 ** * triggers a backup 10:15 AM every day. Test
your cron expression using a website such as Crontab Guru.
Usag
e
create-service or update-service
Optimize for Short Words
The following table describes how to use the optimize_for_short_words optional parameter:
optimize_for_short_words
Note: In Tanzu SQL for VMs, replication is called sync, rather than semi-sync. This is
because it is as synchronous as possible given the limits of MySQL. For more
information about MySQL semi-sync replication, see the MySQL documentation.
Note: When the replication mode timeout is reached, the replication mode
automatically reverts to asynchronous without any user intervention. You can
manually override this timeout by setting a lower value.
Note: Configuring a cron expression overrides the default schedule for your service instance.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
232
Ty
pe
Boolean
Def
aul
t
false
Des
cri
pti
on
Set this parameter to true to change the MySQL system variable innodb_ft_min_token_size to 1. This allows
shorter words to be stored in the InnoDB full-text index.
Because this has the side effect of increasing the size of the index, you must monitor the memory usage of the
service instance and choose a larger VM type when necessary. Also, the operator needs to prevent the index from
becoming too large and ineffective by removing entries as described in the following paragraphs. How often this
needs to be done depends on the workload and how much data is changed in the full-text index.
For more information about the innodb_ft_min_token_size system variable, see the MySQL documentation.
To remove full-text index entries for deleted records or old records:
1. Edit my.cnf to set innodb_optimize_fulltext_only=ON.
For single node and leader-follower plans, the path to the file is
/var/vcap/jobs/mysql/config/my.conf.
For HA cluster and Multi‑Site Replication plans the path to the file is /var/vcap/jobs/pxc-
mysql/config/my.conf.
2. Run OPTIMIZE TABLE on the indexed tables.
3. When the optimization is done, set innodb_optimize_fulltext_only=OFF so that the query behaves
normally for other tables.
For more information about InnoDB full-text index deletion, see the MySQL documentation.
Usa
ge
create-service or update-service
WSREP Applier Threads
The following table describes how to use the wsrep_applier_threads optional parameter:
wsrep_applier_threads
Type Integer
Defa
ult
4
Desc
ripti
on
Specifies the number of threads that can apply replication transactions in parallel in a high-availability cluster.
VMware MySQL for VMs defaults to 4 threads, whereas Percona XtraDB Cluster 5.7 defaults to 1.For further
details, see the Percona XtraDB Cluster Docs.
Usag
e
create-service or update-service as per Set Optional Parameters
Note: This parameter only applies to "high-availablity" service plans, and is rejected for
service plans using other topologies.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
233
VMware Tanzu SQL with MySQL for VMs - Developer Guide
- Managing MySQL
This guide covers the following topics:
Managing Tanzu MySQL for VMs
Connecting to Tanzu MySQL for VMs
Customizing Database Credentials
Using Management Tools for Tanzu MySQL for VMs
Using SSH to Connect from Outside a Deployment
Triggering Multi-Site Replication and Failover
Backing Up and Restoring with mysqldump
Monitoring Node Health for HA Clusters
Troubleshooting Instances
Developer Guide - Managing MySQL for VMs - Connecting
To MySQL
This guide covers the following topics:
Connecting to Tanzu MySQL for VMs
Customizing Database Credentials
Using Management Tools for Tanzu MySQL for VMs
Using SSH to Connect from Outside a Deployment
Customizing Database Credentials
This topic provides instructions for developers to customize access credentials and privileges for
VMware Tanzu SQL with MySQL for VMs service instances.
Overview
You can customize database credentials by creating service keys with custom properties. For
example, you can create read-only access credentials to activate desktop tools to connect to your
databases.
The following procedures use the Cloud Foundry Command Line Interface (cf CLI). You can also
use Apps Manager to do the same tasks using a graphical user interface. For information about
Apps Manager, see Getting Started with Apps Manager.
Create Read-only Access Credentials
Tanzu SQL for VMs enables space developers to create read-only credentials to give to users who
need read-only access to the database. These users can audit and monitor the database without
mutating or changing any data.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
234
To create and find read-only credentials for an existing service instance:
1. Create a new read-only service key for a service instance by running:
cf create-service-key SERVICE-INSTANCE-NAME KEY-NAME -c '{ "read-only": true }'
For example:
2. Retrieve the read-only credentials from the service key by running:
cf service-key SERVICE-INSTANCE-NAME KEY-NAME
For example:
Note: Any user that can create a service key can provision a fully privileged service
key.
$ cf create-service-key mydb mykey1 -c '{ "read-only": true }'
Creating service key mykey1 for service instance mydb as admin...
OK
$ cf service-key mydb mykey1
{
"hostname": "a7113e41-7254-4f5a-a0cf-a88b052c8b10.mysql.service.intern
al",
"jdbcUrl": "jdbc:mysql://a7113e41-7254-4f5a-a0cf-a88b052c8b10.mysql.se
rvice.internal:3306/service_instance_db?user=973eb219bd554dfc9794bc29a3
01bcb1\u0026password=zr3aqa847tzm6cls\u0026sslMode=VERIFY_IDENTITY\u002
6useSSL=true\u0026requireSSL=true\u0026serverSslCert=/etc/ssl/certs/ca-
certificates.crt",
"name": "service_instance_db",
"password": "zr3aqa847tzm6cls",
"port": 3306,
"tls": {
"cert": {
"ca": "-----BEGIN CERTIFICATE-----\nMIIDDzCCAfegAwIBAgIUW0tF3p3wubz+
0GMH/850aVUIPnUwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAxMMVG9vbHNtaXRoc0NBM
B4XDTIwMDkyMTA2MjcxMFoXDTIx\nMDkyMTA2MjcxMFowFzEVMBMGA1UEAxMMVG9vbHNtaX
Roc0NBMIIBIjANBgkqhkiG\n9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv5lmGmSCIkV2w1axS/v
Gk7GjQHnTtjhme4cO\nvT1Nbv6oWqt0Tlm+2gzGb8W7A6SsIEN33ltq4LTWEFK8t0htphDe
1xkAf1Eq7jWM\nnS9aFnXyEuqw5fzWAjQMMqd3JvvZ2Z85o9NaHdi+XOlQAv9UHlWkjaSAv
FoRyaC7\npI0GNF8/QpvHORdPxpyGey/LtE8FxSKb8EL1y430LT7N/PxmVmFnySItlMbBiX
cA\nTkosY+9IswMwrVyYBwN65UoC7sKomjrloVNHhErm5pZv1hlEvEK116wiNY//9Wav\nA
mUneQ4LpjMPYXDGhHL04mMc2ySsrFW0lI8zcYzbEQBUQN5ovwIDAQABo1MwUTAd\nBgNVHQ
4EFgQUyCp0znZlP1d+vQ9U4tpzs1g/hrAwHwYDVR0jBBgwFoAUyCp0znZl\nP1d+vQ9U4tp
zs1g/hrAwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOC\nAQEAfh26fULdpurm
RdE9KKcRGVY56fFk2SbxITTIoHULtQY5pzau9KVOKGl2+czM\n875QC1YviBoonQZE8LSA1
A1gaj9s+XT5/fCGRagU/ODZX/sBDJMQJjaN+/QFRhom\nXKHZ+1nCPJqSiGGDJOANtZT1Xl
fz+cKreuDfPysAA+s5row17CUIcYcC0WTNgVE9\nGdkjzF9ZDakLHekkQ9F2nMmEhwRTwxw
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
235
Create Custom Username Credentials
Tanzu SQL for VMs enables space developers to create custom usernames for service keys and
service bindings. You can create these credentials for users that want to access the database with a
specific username.
To create and find custom username credentials for an existing service instance:
1. Create a new service key and username for a service-instance by running:
cf create-service-key SERVICE-INSTANCE-NAME KEY-NAME -c '{ "username": "NEW-USE
RNAME" }'
For example:
2. Retrieve the credentials from the service key by running:
cf service-key SERVICE-INSTANCE-NAME KEY-NAME
For example:
neqJzcTFqDgWiIZpzkF6Ck90Ay43mpc7N\nU/osEYJlW10NJy8+wq11yZ50T3Z8EFkkbzo9
QipnfW1byY+JstVeR0uLmUzNmkyy\nNBUf8fcYBdCLr2lDvOiUGhRw6w==\n-----END CE
RTIFICATE-----\n\n"
}
},
"uri": "mysql://973eb219bd554dfc9794bc29a301bcb1:zr3aqa847tzm6cls@a711
3e41-7254-4f5a-a0cf-a88b052c8b10.mysql.service.internal:3306/service_in
stance_db?reconnect=true",
"username": "973eb219bd554dfc9794bc29a301bcb1"
}
Caution
In cf CLI v8, the response includes a top-level credentials key. Earlier
versions of the cf CLI do not include a top-level credentials key.
Note: Any user that can create a service key can provision a fully privileged service
key.
$ cf create-service-key mydb mykey2 -c '{ "username": "myuser" }'
Creating service key mykey2 for service instance mydb as admin...
OK
$ cf service-key mydb mykey2
{
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
236
Using Management Tools for VMware Tanzu SQL with
MySQL for VMs
This topic provides instructions for tool developers to use for access to their VMware Tanzu SQL
with MySQL for VMs databases.
Overview
"hostname": "a7113e41-7254-4f5a-a0cf-a88b052c8b10.mysql.service.intern
al",
"jdbcUrl": "jdbc:mysql://a7113e41-7254-4f5a-a0cf-a88b052c8b10.mysql.se
rvice.internal:3306/service_instance_db?user=myuser\u0026password=bdjq5
o19ax4suzmg\u0026sslMode=VERIFY_IDENTITY\u0026useSSL=true\u0026requireS
SL=true\u0026serverSslCert=/etc/ssl/certs/ca-certificates.crt",
"name": "service_instance_db",
"password": "bdjq5o19ax4suzmg",
"port": 3306,
"tls": {
"cert": {
"ca": "-----BEGIN CERTIFICATE-----\nMIIDDzCCAfegAwIBAgIUW0tF3p3wubz+
0GMH/850aVUIPnUwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAxMMVG9vbHNtaXRoc0NBM
B4XDTIwMDkyMTA2MjcxMFoXDTIx\nMDkyMTA2MjcxMFowFzEVMBMGA1UEAxMMVG9vbHNtaX
Roc0NBMIIBIjANBgkqhkiG\n9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv5lmGmSCIkV2w1axS/v
Gk7GjQHnTtjhme4cO\nvT1Nbv6oWqt0Tlm+2gzGb8W7A6SsIEN33ltq4LTWEFK8t0htphDe
1xkAf1Eq7jWM\nnS9aFnXyEuqw5fzWAjQMMqd3JvvZ2Z85o9NaHdi+XOlQAv9UHlWkjaSAv
FoRyaC7\npI0GNF8/QpvHORdPxpyGey/LtE8FxSKb8EL1y430LT7N/PxmVmFnySItlMbBiX
cA\nTkosY+9IswMwrVyYBwN65UoC7sKomjrloVNHhErm5pZv1hlEvEK116wiNY//9Wav\nA
mUneQ4LpjMPYXDGhHL04mMc2ySsrFW0lI8zcYzbEQBUQN5ovwIDAQABo1MwUTAd\nBgNVHQ
4EFgQUyCp0znZlP1d+vQ9U4tpzs1g/hrAwHwYDVR0jBBgwFoAUyCp0znZl\nP1d+vQ9U4tp
zs1g/hrAwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOC\nAQEAfh26fULdpurm
RdE9KKcRGVY56fFk2SbxITTIoHULtQY5pzau9KVOKGl2+czM\n875QC1YviBoonQZE8LSA1
A1gaj9s+XT5/fCGRagU/ODZX/sBDJMQJjaN+/QFRhom\nXKHZ+1nCPJqSiGGDJOANtZT1Xl
fz+cKreuDfPysAA+s5row17CUIcYcC0WTNgVE9\nGdkjzF9ZDakLHekkQ9F2nMmEhwRTwxw
neqJzcTFqDgWiIZpzkF6Ck90Ay43mpc7N\nU/osEYJlW10NJy8+wq11yZ50T3Z8EFkkbzo9
QipnfW1byY+JstVeR0uLmUzNmkyy\nNBUf8fcYBdCLr2lDvOiUGhRw6w==\n-----END CE
RTIFICATE-----\n\n"
}
},
"uri": "mysql://myuser:bdjq5o19ax4suzmg@a7113e41-7254-4f5a-a0cf-a88b05
2c8b10.mysql.service.internal:3306/service_instance_db?reconnect=true",
"username": "myuser"
}
Caution
In cf CLI v8, the response includes a top-level credentials key. Earlier
versions of the cf CLI do not include a top-level credentials key.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
237
You can access your Tanzu SQL for VMs databases by using the following tools:
MySQLWeb Database Management App.
cf CLI MySQL Plug-in.
Desktop Tools.
MySQLWeb Database Management App
The MySQLWeb app provides a web UI for managing Tanzu SQL for VMs databases. The free app
lets you view and operate on tables, indexes, constraints, and other database structures, and
directly run SQL commands.
You can run the MySQLWeb app in the following ways:
Standalone on your own machine.
Deployed to Ops Manager.
If you deploy MySQLWeb to Ops Manager, you can configure it in the deployment manifest to
automatically bind to a specific service instance.
See the MySQLWeb code repo for how to install and run MySQLWeb.
cf CLI MySQL Plug-in
To connect to your Tanzu SQL for VMs databases from a command line, use the cf CLI MySQL
plug-in. The plug-in lets you:
Inspect databases for debugging.
Manually adjust database schema or contents in development environments.
Dump and restore databases.
To install the cf CLI MySQL plug-in, run the following:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
238
For more information, see the cf-mysql-plugin repository.
Desktop Tools
You can connect your Tanzu SQL for VMs databases to a desktop tool. For example, MySQL
Workbench or Sequel Pro, if you have the credentials for your MySQL service instance.
If you do not have credentials for your MySQL service instance, follow the procedure in Create
Read-only Access Credentials.
To connect your databases to a desktop tool do the following:
1. To retrieve the credentials from your MySQL service instance service key, run the following
command:
cf service-key SERVICE-INSTANCE MYSQL-SERVICE-KEY
Where:
SERVICE-INSTANCE is the name of your service instance.
KEY-NAME is the name of your service key.
For example:
2. Record the values for the following:
hostname
name
$ cf install-plugin -r "CF-Community" mysql-plugin
$ cf service-key mydb mykey1
{
"hostname": "q-n3s3y1.q-g693.bosh",
"jdbcUrl": "jdbc:mysql://q-n3s3y1.q-g693.bosh:3306/service_instance
_db?user=6bf07ae455a14064a9073cec8696366c\u0026password=a22aaa2a2a2aaaa
a\u0026=true",
"name": "service\_instance\_db",
"password": "a22aaa2a2a2aaaaa",
"port": 3306,
"uri": "mysql://6bf07ae455a14064a9073cec8696366c:a22aaa2a2a2aaaaa@q
-n3s3y1.q-g693.bosh:3306/service_instance_db?reconnect=true",
"username": "6bf07ae455a14064a9073cec8696366c"
}
Caution
In cf CLI v8, the response includes a top-level credentials key. Earlier
versions of the cf CLI do not include a top-level credentials key.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
239
password
port
username
3. Configure an SSH tunnel using the values for hostname and port that you recorded in the
previous step. For information on configuring an SSH tunnel, see Configure your SSH
tunnel.
4. Configure a connection in your desktop tool using the values for hostname, name, password,
port, and username that you recorded in the previous step.
Establishing a Connection from Outside a Deployment
This topic provides instructions for developers for establishing a connection to a VMware Tanzu
SQL with MySQL for VMs service instance from outside your deployment.
Overview
If you want to connect to a MySQL service instance from outside the VMware Tanzu Application
Service for VMs deployment where your Tanzu SQL for VMs service instance is located, you can do
one of the following:
Connect Using an SSH Tunnel
Connect Using an IP Address
You may want to use an SSH tunnel or IP address for establishing a connection in the following
cases:
Connecting from a local workstation. For more information, see Tanzu SQL for VMs Tools.
Connecting from outside where your Tanzu SQL for VMs service instance is located.
Connecting from a legacy app that is not in your TAS for VMs deployment.
Taking a manual backup to be stored locally. For more information, see Backing Up and
Restoring with mysqldump.
Connect Using an SSH Tunnel
Any app deployed within your deployment can resolve BOSH DNS hostnames and forward traffic to
MySQL service instances using an SSH tunnel.
Prerequisite
To connect using an SSH tunnel, you must have SSH access to app containers. For more
information, see Configuring SSH Access for TAS for VMs.
Procedure
To connect to a MySQL instance using an SSH tunnel, follow the procedures in Accessing Services
with SSH.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
240
Connect Using an IP Address
You can also connect remotely to the MySQL service instance by directly using its IP address.
You can obtain the IP address using one the following methods:
Using cf SSH. For more information, see Obtain the IP Address Using cf SSH.
Using the DNS Lookup app. For more information, see Obtain the IP Address Using the
DNS Lookup App.
Using the BOSH CLI. For more information, see Obtain the IP Address Using the BOSH CLI.
Obtain the IP Address Using cf SSH
If you have limited access to the platform you can obtain the IP address through cf SSH.
Prerequisites
To obtain the IP address using cf SSH, you must have the following:
SSH access to app containers. For more information, see Configuring SSH Access for TAS
for VMs.
An app deployed on the same deployment as the MySQL service instance.
Procedure
To obtain the IP address using cf SSH, follow these steps:
1. To retrieve your service key, run the following command:
cf service-keys SERVICE-INSTANCE
Where SERVICE-INSTANCE is the name of your Tanzu SQL for VMs service instance.
2. To obtain the BOSH DNS hostname of the MySQL service instance, run the following
command:
cf service-key SERVICE-INSTANCE SERVICE-KEY
For example:
Note: By connecting to the MySQL service instance directly using its IP address,
the client is responsible to handle cases in which the IP address of the service
instance will change transparently. This can happen when a VM is recreated, or
during a failover.
$ cf service-key my-mysql my-service-key
Getting key my-service-key for service instance my-mysql as admin...
{
"hostname": "q-n1a2b3.q-c456.bosh",
"jdbcUrl": "jdbc:mysql://10.10.10.5:3306/cf_e2d148a8_1baa_4961_b314_24
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
241
3. Record the value of hostname.
4. To SSH into the app, run the following command:
$ cf ssh APP-NAME
Where APP-NAME is the name of your app deployed on the same deployment as the MySQL
service instance.
5. To obtain the IP address of the service instance, run the following command:
$ dig +short BOSH-DNS-HOSTNAME
Where BOSH-DNS-HOSTNAME is the BOSH DNS hostname that you retrieved from step 2.
The output of the command is the IP address.
Obtain the IP Address Using the DNS Lookup App
See the following screenshot of the DNS Lookup UI:
If your deployment has app SSH disabled, you can deploy the DNS Lookup app to your
deployment. The DNS Lookup app resolves BOSH DNS hostnames.
For more information, see the dns-lookup repository in GitHub.
Obtain the IP Address Using the BOSH CLI
If you have access to the BOSH CLI, you can obtain the IP address through it.
Prerequisite
To connect using the BOSH CLI, you must have access to the BOSH CLI. Ask your operator for
access to the BOSH CLI.
Procedure
To connect using the BOSH CLI, follow these steps:
31f57037e5?user=my_unique_name\u0026password=123456789",
"name": "cf_e2d148a8_1baa_4961_b314_2431f57037e5",
"password": "123456789",
"port": 3306,
"uri": "mysql://my_unique_name:[email protected]:3306/cf_e2d148a8_1
baa_4961_b314_2431f57037e5?reconnect=true",
"username": "myuser"
}
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
242
1. To obtain the credentials and IP address needed to use the BOSH CLI, follow the
procedure in Gather Credential and IP Address Information.
2. To SSH into your Ops Manager VM, follow the procedure in Log in to the Ops Manager VM
with SSH for your IaaS.
3. To log in to your BOSH Director, follow the procedure in Authenticate with the BOSH
Director VM.
4. To obtain the MySQL service instance GUID, run the following command:
cf service SERVICE-INSTANCE --guid
Where SERVICE-INSTANCE is the name of your Tanzu SQL for VMs service instance.
5. To use the BOSH CLI to view the IP address, run one of the following commands
depending on your topology:
If your MySQL service instance topology is single node or highly available (HA)
cluster, run the following command:
bosh -d service-instance_GUID instance
Where GUID is the GUID of the service instance retrieved above.
If you are using an HA cluster, record the IP address of any of your instances.
For example:
If your MySQL service instance topology is leader-follower, run the following
command:
bosh -d service-instance_GUID run-errand inspect
Record the IP address for the instance marked Role: leader.
$ bosh -d service-instance\_b2e92d56-27e2-407e-90e0-c34612c10dba
\
instances
Using environment 'https://10.0.0.6:25555' as client 'admin'
Task 21409. Done
Deployment 'service-instance_0b26c8d1-2795-438e-bcbf-339e77daf88
e'
Instance Process State AZ IP
s
mysql/1373022d-4eab-46d3-8fd1-a12067edf597 running z2 1
0.0.17.14
1 instances
Succeeded
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
243
For example:
Triggering Multi-Site Replication Failover and Switchover
The topic describes how to trigger a failover and switchover when using a Multi‑Site Replication
plan.
Overview
You can trigger a failover or switchover to redirect traffic to a secondary foundation. You can only
trigger a failover or switchover in Tanzu SQL for VMs v2.7.3 and later.
For information about when to trigger a failover or switchover, see About Failover and Switchover.
Before you trigger a failover or switchover, you must verify that the follower service instance is
healthy. See Verify Follower Health.
Instance mysql/ca0ed8b5-7590-4cde-bba8-7ca2935f2bd0
Exit Code 0
Stdout 2018/04/03 18:08:46 Started executing command: inspect
2018/04/03 18:08:46
IP Address: 10.0.8.11
Role: leader
Read Only: false
Replication Configured: false
Replication Mode: async
Has Data: true
GTID Executed: 82ddc607-710a-404e-b1b8-a7e3ea7ec063:1-1
8
2018/04/03 18:08:46 Successfully executed command: insp
ect
Stderr -
Instance mysql/37e4b6bc-2ed6-4bd2-84d1-e59a91f5e7f8
Exit Code 0
Stdout 2018/04/03 18:08:46 Started executing command: inspect
2018/04/03 18:08:46
IP Address: 10.0.8.10
Role: follower
Read Only: true
Replication Configured: true
Replication Mode: async
Has Data: true
GTID Executed: 82ddc607-710a-404e-b1b8-a7e3ea7ec063:1-1
8
2018/04/03 18:08:46 Successfully executed command: insp
ect
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
244
Verify Follower Health
Before you trigger a failover or switchover, you must verify that the follower service instance is
healthy. If your follower service instance is unhealthy, contact Support.
To verify the service instance:
1. Log in to the deployment for your secondary foundation by running:
cf login SECONDARY-API-URL
Where SECONDARY-API-URL is the API endpoint for your secondary foundation.
2. Record the GUID of the follower service instance by running:
cf service SERVICE-INSTANCE-NAME --guid
Where SERVICE-INSTANCE-NAME is the name of the follower service instance.
For example:
3. Obtain the credentials and IP address needed to SSH into the Ops Manager VM by
following the procedure in Gather Credential and IP Address Information.
4. SSH into the Ops Manager VM by following the procedure in Log in to the Ops Manager
VM with SSH.
5. From the Ops Manager VM, log in to your BOSH Director by following the procedure in
Authenticate with the BOSH Director VM.
6. View the health of the service instance by running:
bosh -d service-instance_GUID instance
For example:
Note: The procedures in this topic assume that you created the leader service
instance in the primary foundation and the follower service instance in the
secondary foundation.
$ cf service my-follower --guid
12345678-90ab-cdef-1234-567890abcdef
$ bosh -d service-instance_12345678-90ab-cdef-1234-567890abcdef instanc
e
Using environment 'https://10.0.0.6:25555' as client 'admin'
Task 21409. Done
Deployment 'service-instance_12345678-90ab-cdef-1234-567890abcdef'
Instance Process State AZ IPs
mysql/1373022d-4eab-46d3-8fd1-a12067edf597 running z2 10.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
245
7. Ensure that the service instance is running. If the service instance is failing, contact
Support.
Trigger a Failover
To trigger a failover:
1. Promote the Follower
2. Delete or Purge the Former Leader
3. Create a Follower
4. Reconfigure Multi‑Site Replication
Promote the Follower
To promote the follower service instance to leader:
1. Log in to the deployment for your secondary foundation by running:
cf login SECONDARY-API-URL
Where SECONDARY-API-URL is the API endpoint for your secondary foundation.
2. Promote the follower service instance to leader by running:
cf update-service SECONDARY-INSTANCE \
-c '{"initiate-failover":"promote-follower-to-leader"}'
For example:
0.17.14
1 instances
Succeeded
Warning: You can only trigger failover if you do not need to recover the leader
service instance.
Note: If you try to promote a leader-follower, highly available cluster, or single node
service instance to leader or make it read-only, you get an error message similar to:
Updating service instance haDB as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broker
error: 1 error occurred:
* the configuration parameter 'initiate-failover' is not a valid option
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
246
3. If this command fails, do one of the following:
If you have unapplied local transactions on the follower service instance, wait for the
transactions to be applied and run the command again. The error message looks like
the following:
Updating service instance secondary-node as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broke
r error: Promotion of follower failed - has 1 transactions still unapplie
d
If the leader service instance is still reachable and in read-write mode, follow the
procedure in Trigger a Switchover instead. The error message looks like the
following:
Updating service instance secondary-node as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broke
r error: Promotion of follower failed - the leader is still writable
4. Watch the progress of the service instance update by running:
watch cf services
Wait for the last operation for your instance to show as update succeeded.
For example:
5. Reconfigure your global DNS load balancer to direct all traffic to apps in your secondary
foundation. See Configure Your GLB.
Delete or Purge the Former Leader
When you do a failover, you cannot manually recover the leader service instance. After you
promote the follower service instance to leader, you remove the former leader service instance.
Otherwise, the service instance can recover in read-write mode.
$ cf update-service secondary-node \
-c '{"initiate-failover":"promote-follower-to-leader"}'
Updating service instance secondary-node as admin...
OK
$ watch cf services
Getting services in org my-org / space my-space as admin...
OK
name service plan bound apps
last operation
secondary-node p.mysql db-pxc-single-node-small
update succeeded
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
247
The way you remove the service instance, depends on whether the VM has been lost or not.
To remove the former leader service instance:
1. Log in to the deployment for your primary foundation by running:
cf login PRIMARY-API-URL
Where PRIMARY-API-URL is the API endpoint for the primary foundation.
2. Do one of the following:
If the VM is lost, purge the service instance by doing the procedure Purge a Service
Instance in
Using VMware Tanzu SQL with MySQL for VMs
.
If the VM is not lost:
1. Remove all app bindings by following the procedure Unbind an App from a
Service Instance in
Using VMware Tanzu SQL with MySQL for VMs
.
2. Delete the service keys from the former leader service instance.
3. Delete the service instance by following the procedure Delete a Service
Instance in
Using VMware Tanzu SQL with MySQL for VMs
.
Create a Follower
To reconfigure Multi‑Site Replication between two instances, a new follower without any data must
be created in the primary foundation.
To create a follower:
1. Log in to the deployment for your primary foundation by running:
cf login PRIMARY-API-URL
Where PRIMARY-API-URL is the API endpoint for the primary foundation.
2. Create a service instance using the same Multi‑Site Replication plan that was used to create
the original instance:
Follow the procedure in Create a Service Instance in
Using VMware Tanzu SQL with
MySQL for VMs
.
Do not name your service instance follower because, if in the future you trigger a
failover or switchover, this instance can no longer be the follower.
Reconfigure Multi‑Site Replication
The follower in the primary foundation needs to catch up to the newly promoted leader in the
secondary foundation.
Note: If the foundation is lost, you purge the service instance after
following the steps to recover the foundation's Cloud Controller
database in Restoring Deployments from Backup with BBR.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
248
Reconfigure Multi‑Site Replication so that the follower in the primary foundation receives the data
from the leader in the secondary foundation.
To reconfigure:
1. Follow the procedure in Reconfigure Multi‑Site Replication.
Trigger a Switchover
To trigger a switchover:
1. Promote the Follower
2. Reconfigure Multi‑Site Replication
Promote the Follower
Before you promote the follower service instance, you must make the leader service instance,
which is in the primary foundation, a read-only format.
To make the leader read-only and promote the follower to leader in the secondary foundation:
1. Log in to the deployment for your primary foundation by running:
cf login PRIMARY-API-URL
Where PRIMARY-API-URL is the API endpoint for the primary foundation.
2. Set the service instance that is currently the leader to read-only by running:
cf update-service PRIMARY-INSTANCE \
-c '{"initiate-failover":"make-leader-read-only"}'
For example:
3. Watch the progress of the service instance update by running:
Note: If you try to promote a leader-follower, highly available cluster, or single node
service instance to leader or make it read-only, you get an error message similar to
the following:
Updating service instance haDB as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broker
error: 1 error occurred:
* the configuration parameter 'initiate-failover' is not a valid option
$ cf update-service primary-node \
-c '{"initiate-failover":"make-leader-read-only"}'
Updating service instance primary-node as admin...
OK
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
249
watch cf services
Wait for the last operation for your instance to show as update succeeded.
4. Log in to the deployment for your secondary foundation by running:
cf login SECONDARY-API-URL
Where SECONDARY-API-URL is the API endpoint for your secondary foundation.
5. Promote the service instance in the secondary foundation to leader by running:
cf update-service SECONDARY-INSTANCE \
-c '{"initiate-failover":"promote-follower-to-leader"}'
If this command fails, do one of the following:
If you have unapplied local transactions on the follower service instance, wait for the
transactions to be applied and then run the command again. The error message
might look like the following:
Updating service instance secondary-node as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broke
r error: Promotion of follower failed - has 1 transactions still unapplie
d
If the leader service instance is still reachable and in read-write mode, follow the
procedure in Trigger a Switchover instead. The error message might look like the
following:
Updating service instance secondary-node as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broke
r error: Promotion of follower failed - the leader is still writable
6. Watch the progress of the service instance update by running:
watch cf services
Wait for the last operation for your instance to show as update succeeded.
Reconfigure Multi‑Site Replication
To establish a connection between the service instances in the primary and secondary foundations,
you must reconfigure replication. Reconfiguring replication is similar to the procedure in Configure
Multi‑Site Replication except the service instance in the primary foundation is the follower and the
service instance in the secondary foundation is the leader.
Warning: To successfully trigger a switchover, the follower dataset must be a
subset of the leader dataset. This means that Tanzu SQL for VMs has not written
new data to the follower. The follower must also be no more than 3 days behind the
leader.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
250
Workflow for Reconfiguring Multi‑Site Replication
The following diagram describes the workflow for reconfiguring multi-site replication:
View a larger version of this diagram
The steps shown in the diagram are:
1. Create host-info service key.
2. Record host-info service key.
3. Update secondary service instance with host-info service key.
4. Create credentials service key.
5. Record credentials service key.
6. Update primary service instance with credentials service key.
Procedure for Reconfiguring Multi‑Site Replication
To reconfigure replication for the service instances:
If your follower instance does not satisfy these requirements, you must create a
new Multi‑Site Replication service instance and reconfigure replication using this
new, empty instance as the follower.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
251
1. Log in to the deployment for your primary foundation by running:
cf login PRIMARY-API-URL
2. Create a host-info service key for the service instance in your primary foundation:
cf create-service-key PRIMARY-INSTANCE SERVICE-KEY \
-c '{"replication-request": "host-info"}'
Where:
PRIMARY-INSTANCE is the name of the follower service instance in the primary
foundation.
SERVICE-KEY is a name you choose for the host-info service key.
For example:
3. View the replication-credentials for your host-info service key by running:
cf service-key PRIMARY-INSTANCE SERVICE-KEY
Where:
PRIMARY-INSTANCE is the name of the follower service instance in the primary
foundation.
SERVICE-KEY is the name of the host-info service key you created in previous step.
For example:
$ cf create-service-key primary-node host-info \
-c '{"replication-request": "host-info" }'
Creating service key host-info for service instance primary-node as adm
in...
OK
$ cf service-key primary-node host-info-key
Getting key host-info-key for service instance primary-node as admin...
{
"credentials": {
"replication": {
"peer-info": {
"hostname": "primary.bosh",
"ip": "10.0.19.12",
"system_domain": "sys.primary-domain.com",
"uuid": "ab12cd34-5678-91e2-345f-67891h234567"
},
"role": "leader"
}
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
252
4. Record the output of the previous command, and remove the top-level credentials key.
5. Log in to the deployment for your secondary foundation by running:
cf login SECONDARY-API-URL
6. Update your leader service instance in the secondary foundation with the host-info service
key by running:
cf update-service SECONDARY-INSTANCE -c HOST-INFO
Where:
SECONDARY-INSTANCE is the name of the leader service instance in the secondary
foundation.
HOST-INFO is the output you recorded in previous step.
For example:
7. Monitor the progress of the service instance update by running:
watch cf services
Wait for the last operation for your instance to show as update succeeded.
8. Create a credentials service key for the service instance in your secondary foundation by
running:
}
}
Caution
In cf CLI v8, the response includes a top-level credentials key. Earlier
versions of the cf CLI do not include a top-level credentials key. This
procedure assumes that you are using cf CLI v8.
$ cf update-service secondary-node -c '{"replication":{ \
"peer-info":{ \
"hostname": "primary.bosh", \
"ip": "10.0.18.12", \
"system_domain": "sys.primary-domain.com", \
"uuid": "ab12cd34-5678-91e2-345f-67891h234567" \
},\
"role": "leader" \
} \
}'
Updating service instance secondary-node as admin...
OK
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
253
cf create-service-key SECONDARY-INSTANCE SERVICE-KEY-NAME \
-c '{"replication-request": "credentials"}'
Where:
SECONDARY-INSTANCE is the name of the service instance in the secondary
foundation.
SERVICE-KEY-NAME is a name you choose for the credentials service key.
For example:
9. View the replication-credentials for your credentials service key by running:
cf service-key SECONDARY-INSTANCE SERVICE-KEY-NAME
Where:
SECONDARY-INSTANCE is the name of the leader service instance in the secondary
foundation.
SERVICE-KEY-NAME is the name of the credentials service key you created in step 7.
For example:
$ cf create-service-key secondary-node cred-key \
-c '{"replication-request": "credentials" }'
Creating service key cred-key for service instance secondary-node as us
OK
Note: The -c flag is different than the one in step 2.
$ cf service-key secondary-node cred-key
Getting key cred-key for service instance secondary as admin...
{
"credentials": {
"replication": {
"credentials": {
"password": "a22aaa2a2a2aaaaa",
"username": "6bf07ae455a14064a9073cec8696366c"
},
"peer-info": {
"hostname": "secondary.bosh",
"ip": "10.0.17.12",
"system_domain": "sys.secondary-domain.com",
"uuid": "zy98xw76-5432-19v8-765u-43219t876543"
},
"role": "follower"
}
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
254
10. Record the output of the previous command, and remove the top-level credentials key.
11. Log in to the deployment for your primary foundation by running:
cf login PRIMARY-API-URL
12. Update the follower service instance in the primary foundation with the credentials service
key by running:
cf update-service PRIMARY-INSTANCE -c CREDENTIALS
Where:
PRIMARY-INSTANCE is name of the follower service instance in the primary foundation.
CREDENTIALS is the output you recorded in the previous step.
For example:
13. Watch the progress of the service instance update by running:
watch cf services
Wait for the last operation for your instance to show as update succeeded.
}
}
Caution
In cf CLI v8, the response includes a top-level credentials key. Earlier
versions of the cf CLI do not include a top-level credentials key. This
procedure assumes that you are using cf CLI v8.
$ cf update-service primary-node -c '{"replication": { \
"credentials": { \
"password": "a22aaa2a2a2aaaaa", \
"username": "6bf07ae455a14064a9073cec8696366c" \
}, \
"peer-info": { \
"hostname": "secondary.bosh", \
"ip": "10.0.17.12", \
"system_domain": "sys.secondary-domain.com", \
"uuid": "zy98xw76-5432-19v8-765u-43219t876543" \
}, \
"role": "follower" \
} \
}'
Updating service instance primary-node as admin...
OK
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
255
You now have a leader-follower service instance successfully configured, where the leader
is in your secondary foundation and your follower is in the primary foundation.
If this command fails and you get one of the following errors, you must create a new
Multi‑Site Replication service instance and reconfigure replication using this new, empty
instance as the follower.
If your follower service instance is more than 3 days behind leader, you get error
message similar to the following:
If your follower has a divergent dataset from the leader, you get error message
similar to the following:
In either case, you must create a new multi-site service instance and reconfigure replication
using this new, empty instance as the follower.
14. Reconfigure your global DNS load balancer to direct traffic to the correct foundations of
your choice. See Configure Your GLB.
Backing Up and Restoring with mysqldump
This topic describes how to manually create a logical backup with mysqldump and restore a MySQL
database with the backup.
Overview
A logical backup reproduces the database table structure and data, without copying the InnoDB
data files. The backup contains SQL statements to re-create the data. Because of this, restoring a
$ cf update-service primary-node -c /tmp/credentials-key.json
Updating service instance primary-node as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Servi
ce broker error: Establishing Replication Failed - follower is to
o far behind Leader to start replication
Leader GTIDs offering: "487e6056-6e93-11ea-8c96-42010a010806:5-9"
Follower GTIDs missed: "487e6056-6e93-11ea-8c96-42010a010806:1-9"
Try again with an empty instance or contact your operator to trou
bleshoot
$ cf update-service primary-node -c /tmp/credentials-key.json
Updating service instance primary-node as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Servi
ce broker error: Establishing Replication Failed - the follower h
as divergent data
Leader GTIDs applied: "bd2ff185-6947-11ea-80d8-42010a000808:1-20"
Follower GTIDs applied: "c1abd2a4-6947-11ea-8099-42010a010807:1-1
5"
Try again with an empty instance or contact your operator to trou
bleshoot
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
256
logical backup takes longer than restoring a physical backup. While the backup is running,
operations against the database are stalled.
You might want to create a logical backup in the following use cases:
Migrating a Tanzu SQL for VMs database to an off-platform database.
Backing up specific individual Tanzu SQL for VMs databases.
Editing table definitions or insert statements before restoring the Tanzu SQL for VMs
database.
Migrating an off-platform database with encryption at rest or the Percona PAM
Authentication plug-in enabled to a Tanzu SQL for VMs database. See Restore from an Off-
Platform Logical Backup.
In most cases, if you want to copy a Tanzu SQL for VMs database you can use the migrate
command. For information about using the migrate command, see About Data Migration in Tanzu
SQL for VMs.
You can create a physical backup by following the procedure in Backing Up and Restoring VMware
Tanzu SQL with MySQL for VMs.
Operators can configure physical backups of your MySQL database in Tanzu SQL for VMs tile. For
information about backups configured by operators, see Configuring Automated Backups.
Back Up and Restore a Tanzu SQL for VMs Logical Backup
To back up and restore a Tanzu SQL for VMs logical backup:
1. Create a logical backup of your database. See Create a Tanzu SQL for VMs Logical Backup.
2. Restore the logical backup to a Tanzu SQL for VMs service instance. For more information
about restoring a logical backup, see Restore from a Tanzu SQL for VMs Logical Backup.
Create a Tanzu SQL for VMs Logical Backup
Tanzu SQL for VMs deactivates remote admin access to MySQL databases. You must create a read
only service key to access the database for the service instance you want to back up.
To back up your Tanzu SQL for VMs data manually:
1. Create and retrieve read-only access credentials by following the procedure in Create
Read-only Access Credentials.
2. From the output of the previous step, record the following values:
hostname: The MySQL BOSH DNS hostname
password: The password for the user that can be used for backups of the service
instance database
username: The username for the user that can be used for backups of the service
instance database
Note: When you take a logical backup, Tanzu SQL for VMs does not send an email
notification.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
257
3. Connect to your service instance, by either using an SSH tunnel or by connecting directly to
its IP address. For more information, see Establish a Connection to a Service Instance from
Outside Your Deployment.
4. View a list of your databases by running:
mysql --user=USERNAME --password=PASSWORD \
--host=MYSQL-IP \
--silent --execute='show databases'
Where:
USERNAME is the username retrieved from the output of cf service-key.
PASSWORD is the password retrieved from the output of cf service-key.
MYSQL-IP is the MySQL IP address. This value is 0 if you are connecting using an SSH
tunnel.
For example:
5. Do not back up the following databases:
cf_metadata
information_schema
mysql
performance_schema
sys
6. For each remaining database, back up by running:
mysqldump --set-gtid-purged=off --single-transaction \
--user=USERNAME --password=PASSWORD \
--host=MYSQL-IP \
--databases DB-NAME > BACKUP-FILE
Where:
USERNAME is the username retrieved from the output of cf service-key.
PASSWORD is the password retrieved from the output of cf service-key.
MYSQL-IP is the MySQL IP address.
DB-NAME is the name of the database.
BACKUP-FILE is a name you create for the backup file. Use a different filename for
each backup.
The --set-gtid-purged=off flag enables you to restore the backup without admin
privileges.
$ mysql --user=abcdefghijklm --password=123456789 \
--host=10.10.10.5 \
--silent --execute='show databases'
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
258
For example:
For more information about the mysqldump utility, see mysqldump in the MySQL
Documentation.
Restore from a Tanzu SQL for VMs Logical Backup
To restore a logical backup:
1. (Optional) If you want to create a new service instance, do the procedure in Create a
Service Instance.
2. Retrieve the credentials for the service instance you are restoring the backup to by
following steps 1 and 2 in Create a Tanzu SQL for VMs Logical Backup. You can either
restore the backup to an existing service instance or the one you created in step 1.
3. Connect to your service instance, by either using an SSH tunnel or by connecting directly to
its IP address. For more information, see Establish a Connection to a Service Instance from
Outside Your Deployment.
4. Restore your data from the SQL file on your local machine by running:
mysql --user=USERNAME --password=PASSWORD --host=MYSQL-IP < BACKUP-FILE
Where:
USERNAME is the username retrieved from the output of cf service-key.
PASSWORD is the password retrieved from the output of cf service-key.
MYSQL-IP is the MySQL IP address.
BACKUP-FILE is the name of your backup artifact.
Restore from an Off-Platform Logical Backup
This section assumes that you have already created a logical backup for your off-platform database
using mysqldump.
If you want to restore a logical backup from an off-platform database that has encryption at rest or
the Percona PAM Authentication plug-in enabled, you cannot use the migrate command.
To restore an off-platform logical backup to a Tanzu SQL for VMs database:
1. If your database has encryption at rest enabled, delete all instances of ENCRYPTION='Y' from
your backup artifact.
2. Retrieve your service instance GUID by running:
cf service SERVICE-INSTANCE-NAME --guid
$ mysqldump --set-gtid-purged=off --single-transaction \
--user=abcdefghijklm --password=123456789 \
--host=10.10.10.5 \
--databases canary\_db > canary\_db.sql
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
259
Record the output.
3. Copy the backup artifact to the service instance by running:
bosh -d service-instance_GUID scp ./BACKUP-FILE mysql:/tmp/
Where GUID is the GUID you recorded in the previous step.
4. SSH into the service instance by running:
bosh -d service-instance_GUID ssh
5. Restore your backup artifact into mysql by running:
mysql --defaults-file=/var/vcap/jobs/mysql/config/mylogin.cnf \
-D SERVICE-INSTANCE-NAME < /tmp/BACKUP-FILE
Monitoring Node Health
Monitor Node Health
Prerequisite
Monitor Node Health Using the Dashboard
Monitor Node Health Using the API
Node Health
Healthy
Unhealthy
Unresponsive
This topic describes how to obtain the health status for each node in a MySQL database cluster
node in a highly available (HA) cluster plan.
If you cannot write or connect to your app, you should check the health status of your nodes. You
can observe the health status of nodes through a proxy using the Switchboard dashboard or API
endpoint. You can also view the number of client connections routed through a proxy to each
node. For more information about proxies, see MySQL Proxy.
Monitor Node Health
You can monitor the health status of your nodes by doing one of the following:
Monitor Node Health Using the Dashboard
Monitor Node Health Using the API
Prerequisite
To connect to Switchboard, you must obtain credentials.
To obtain credentials for accessing the Switchboard dashboard and API, do the following:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
260
1. Run the following command:
cf service YOUR-HA
Where YOUR-HA is the name of your HA cluster.
2. Record the dashboard hostname, username, and password.
Monitor Node Health Using the Dashboard
To monitor node health using the Switchboard dashboard, do the following:
1. To view a list of proxies in your browser, navigate to the hostname that you recorded in
Prerequisite above. For example, proxy.123abc45-67d8-912e-34f5-
g34612c10dba.org.dedicated-mysql.cf-app.com
2. When prompted, enter the username and password that you recorded in Prerequisite
above.
3. Click the link for the proxy that you want to use to monitor node health.
4. If you are prompted, enter the username and password that you recorded in Prerequisite
above.
$ cf service myHA
Showing info of service myHA in org acceptance / space example as admi
n...
name: myHA
service: p.mysql
tags:
plan: db-ha-small
description: Dedicated instances of MySQL
documentation:
dashboard: proxy.123abc45-67d8-912e-34f5-g34612c10dba.org.dedicated-
mysql.cf-app.com
(username: 'abcdefghijklmno', password: '012345678912345')
Showing status of last operation from service myHA...
status: update succeeded
message: Instance update completed
started: 2018-11-20T01:25:55Z
updated: 2018-11-20T01:30:33Z
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
261
Monitor Node Health Using the API
You can also use the Switchboard API to obtain the information that is shown on the Switchboard
dashboard.
For example, you might want to use the API to write your own app to monitor the cluster.
To monitor node health using the Switchboard API, do the following:
1. To monitor node health, run the following command:
curl https://USERNAME:PASSWORD@N-HOSTNAME/v0/backends
Where:
USERNAME is the username you recorded in Prerequisite above.
PASSWORD is the password you recorded in Prerequisite above.
N is either 0, 1, or 2 depending on the proxy you want to connect to.
HOSTNAME is the hostname you recorded in Prerequisite above.
The above command outputs a JSON object similar to the following:
$ curl https://abcdefghijklmno:[email protected]
2e-34f5-g34612c10dba.org.dedicated-mysql.cf-app.com/v0/backends
[
{
"host": "a-b1234c5d6.e-f891.bosh",
"port": 6033,
"healthy": true,
"name": "backend-0",
"currentSessionCount": 0,
"active": true,
"trafficEnabled": true
},
{
"host": "a-b1234c5d6.e-f891.bosh",
"port": 6033,
"healthy": true,
"name": "backend-1",
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
262
Node Health Status
When determining where to route traffic, the proxy queries an HTTP healthcheck process running
on the node. This healthcheck can return as either healthy or unhealthy, or the node can be
unresponsive.
Healthy
If the healthcheck process returns HTTP status code 200, the proxy includes the node in its pool of
healthy nodes.
When a new or resurrected nodes rejoin the cluster, the proxy continues to route all connections
to the currently active node. In the case of failover, the proxy considers all healthy nodes as
candidates for new connections.
Unhealthy
If the healthcheck returns HTTP status code 503, the proxy considers the node unhealthy.
This happens when a node becomes non-primary. For more information, see Cluster Scaling
Behavior.
"currentSessionCount": 0,
"active": false,
"trafficEnabled": true
},
{
"host": "a-b1234c5d6.e-f891.bosh",
"port": 6033,
"healthy": true,
"name": "backend-2",
"currentSessionCount": 0,
"active": false,
"trafficEnabled": true
}
]
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
263
The proxy severs existing connections to newly unhealthy node. The proxy routes new connections
to a healthy node, assuming such a node exists. Clients are expected to handle reconnecting on
connection failure should the entire cluster become inaccessible.
Unresponsive
If node health cannot be determined due to an unreachable or unresponsive healthcheck endpoint,
the proxy considers the node unhealthy. This may happen if there is a network partition or if the
VM running the node and healthcheck died.
Troubleshooting On-Demand Instances
This topic provides techniques that app developers can use to begin troubleshooting on-demand
instances.
Troubleshoot Errors
Start here if you are responding to a specific error or error messages.
Common Service Errors
No Metrics from Log Cache
Symptom You receive no metrics when running the cf tail command.
Cause This might happen because the Firehose is deactivated in the TAS for VMs tile.
Solution
Ask your operator to ensure that the V2 Firehose check box is activated, and the Enable Log
Cache syslog ingestion check box is deactivated in the TAS for VMs tile. For more information
about configuring these check boxes, see Enable Syslog Forwarding.
If Instances or Database are Inaccessible
You might experience the following in a leader-follower or Multi‑Site Replication topology, or
during upgrades:
Temporary Outages
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
264
Apps Cannot Write to the Database
Apps Are Inoperable
Apps Cannot Connect to the Database
Apps Cannot Connect to the Database after Upgrade from v2.8
MySQL Connector/J v5.1.41 or Earlier
Mutual TLS
Java Apps Cannot Connect after Buildpack Update
Temporary Outages
Symptom VMware Tanzu SQL with MySQL for VMs service instances can become temporarily
inaccessible during upgrades and VM or network failures.
Solution
For more information, see Service Interruptions.
Apps Cannot Write to the Database
Symptom You have a leader-follower or Multi‑Site Replication topology, and your apps can no longer
write to the database.
Cause
If you have a leader-follower topology, the leader VM might be read-only. If you can no longer
read to the database as well, your persistent disk might be full.
If you have a Multi‑Site Replication topology, your leader VM might be down.
Solution
If you have a leader-follower topology and the leader VM is read-only, for how to troubleshoot
this problem, see Both Leader and Follower Instances Are Read-Only.
If your apps can no longer read to the database as well, your persistent disk might be full. For
more information about troubleshooting inoperable apps, see Apps are Inoperable.
If you have a Multi‑Site Replication topology and your leader VM is down, to resolve this issue,
you can trigger a failover to the follower VM. For more information about troubleshooting this
problem, see Triggering Multi-Site Replication Failover and Switchover.
Apps Are Inoperable
Symptom Your apps become inoperable. Read, write, and cf CLI operations do not work.
Cause Your persistent disk might be full.
Solution
Contact your operator to check if your persistent disk is full. For more information about
troubleshooting this problem, see Persistent Disk is Full.
Apps Cannot Connect to the Database
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
265
Symptom Apps can fail to connect to the database.
Cause
When your app uses an incompatible version of MySQL Connector/J.
When your app uses mutual TLS (mTLS).
Solution
See MySQL Connector/J v5.1.41 or Earlier.
See Mutual TLS.
See Java Apps Cannot Connect after Buildpack Update.
Apps Requiring TLS Cannot Connect to the Database after Upgrade from v2.8
Symptom
Apps fail to connect to the database, and, in the app logs, you see certificate errors such as:
And, the following are true:
MySQL Connector/J v8.0.13–21 is used.
TLS is enabled for the apps.
Tanzu SQL for VM was recently upgraded from v2.8.
Cause Versions 8.0.13 to 8.0.21 of the MySQL Connector/J do not support wildcards or subject
alternative names (SANs) in the TLS certificate.
Solution
To resolve the error:
1. Ensure that the service instance has been upgraded to v2.9. For information about
upgrading instances, see Upgrading VMware Tanzu SQL with MySQL for VMs.
2. Rebind the apps to the upgraded instances.
MySQL Connector/J v5.1.41 or Earlier
Symptom
Apps cannot connect to the database when TLS is enabled and the apps are using MySQL
Connector/J v5.1.41 or earlier.
Cause
You see errors about certificates.
For example:
java.security.cert.CertificateException: Server certificate
identity check failed.
Caused by: javax.net.ssl.SSLHandshakeException: Received fat
al alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Alerts.java:19
2) ~[na:1.8.0_152]
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
266
Solution
If you cannot update the MySQL Connector/J, do the workaround in How to deactivate
KeyManager and TrustManager in Container Security Provider Framework in the
Javanbuildpack in the Support Knowledge Base.
Mutual TLS
Symptom Apps cannot connect to the database when TLS is activated and your apps use mTLS.
Cause You see network errors in your app logs.
Solution
To resolve this issue deactivate mTLS in your apps.
Java Apps Cannot Connect after Buildpack Update
Symptom After updating a Java app to use Java buildpack v4.38 or later, the app cannot connect to the
database over TLS.
In the app logs, you see errors such as:
Cause By default, the new version of Java deactivates TLS v1.0 and v1.1.
Solution
Update the app to use MySQL Connector/J v5.1.49 or later or MySQL Connector/J v8.0.19 or
later. This ensures that TLS v1.2 is used.
Failed Backup or Restore with the adbr Plug-in
If you get errors when working with the ApplicationDataBackupRestore (adbr) plug-in for the Cloud
Foundry Command Line Interface (cf CLI) tool, see:
“400” Error during Backup or Restore
“500” Error during Backup or Restore
“502” Error during Backup or Restore
“Status: Restore failed” after adbr Restore
"400" Error during Backup or Restore
Symptom
When running cf adbr backup or cf adbr restore, an error occurs.
For example:
javax.net.ssl.SSLHandshakeException: No appropriate protocol
(protocol is deactivated or cipher suites are inappropriate)
$ cf adbr backup myDB
Failed to backup service instance "myDB": failed due to se
rver error, status code: 400
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
267
Cause The broker on the VM is not running or is in an unhealthy state.
Solution
Verify the health of the broker VM and review the logs for the broker.
"500" Error during Backup or Restore
Symptom
When running cf adbr backup or cf adbr restore, an error occurs.
For example:
Cause The service instance agent is not running or is in an unhealthy state.
Solution
Verify the health of the service instance VM and review the logs for the service instance.
"502" Error during Backup or Restore
Symptom
When running cf adbr backup or cf adbr restore, an error occurs.
For example:
Cause The VM is down, stopped, or in an unhealthy state.
Solution
Verify the health of the broker VM and review the logs for the broker.
"Status: Restore failed" after adbr Restore
Symptom
When running cf adbr get-status after restoring to a service instance, adbr returns Restore
failed.
For example:
Cause A possible cause is that the database on the new service instance is not empty. For more
information, see The New Database Must Be Empty in
Backing Up and Restoring VMware Tanzu
SQL with MySQL for VMs.
$ cf adbr backup myDB
Failed to backup service instance "myDB": failed due to se
rver error, status code: 500
$ cf adbr backup myDB
Failed to backup service instance "myDB": failed due to se
rver error, status code: 502
$ cf adbr get-status myTargetDB
Getting status of service instance myTargetDB in org my-org
/ space system as admin...
[Thu Feb 25 22:33:58 UTC 2021] Status: Restore failed
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
268
Solution
To resolve the error and complete the restore:
1. Determine if the database is empty by reviewing the log /var/vcap/sys/log/mysql-
restore/mysql-restore.stderr.log on the service instance.
2. If any GTIDs (global transaction identifiers) are printed in the logs, then the database
is not empty.
Delete the service instance and create a new service instance to restore the backup
to.
3. If the log does not contain any GTIDs, then the restore failed for another reason.
Review other logs on the service instance and, if necessary, contact Support.
For general information about the adbr plug-in, see Backing Up and Restoring VMware Tanzu SQL
with MySQL for VMs.
Persistent Disk Usage Is Increasing
If you have set the optimize_for_short_words parameter to true and you are alerted that
persistent disk usage is high, then you might need to optimize the indexed tables.
Persistent Disk Usage Is Increasing
Symptom
You have set the optimize_for_short_words optional parameter to true and the persistent disk is
filling up.
For information about the parameter, see Optimize for Short Words. Cause Over time, data has
been deleted from your database and the full-text index has become too large. Solution Remove
full-text entries for deleted or old records by following the instructions in Optimize for Short Words.
For information about monitoring disk usage, see Monitoring and KPIs.
Techniques for Troubleshooting
See the following sections for troubleshooting techniques when using the Cloud Foundry
Command-Line Interface (cf CLI) to perform basic operations on a Tanzu SQL for VMs service
instance.
Basic cf CLI operations include create, update, bind, unbind, and delete.
Understand a Cloud Foundry (CF) Error Message
Failed operations (create, update, bind, unbind, delete) result in an error message. You can retrieve
the error message later by running the cf CLI command cf service INSTANCE-NAME.
$ cf service myservice
Service instance: myservice
Service: super-db
Bound apps:
Tags:
Plan: dedicated-vm
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
269
Use the information in the Message field to debug further. Provide this information to Support when
filing a ticket.
The task-id field maps to the BOSH task ID. For more information on a failed BOSH task, use the
bosh task TASK-ID.
The broker-request-guid maps to the portion of the On-Demand Broker log containing the failed
step. Access the broker log through your syslog aggregator, or access BOSH logs for the broker by
typing bosh logs broker 0. If you have more than one broker instance, repeat this process for
each instance.
Find Information about Your Service Instance
You might need to find the name, GUID, or other information about a service instance. To find this
information, do the following:
1. Log into the space containing the instance or failed instance.
2. If you do not know the name of the service instance, run cf services to see a listing of all
service instances in the space. The service instances are listed in the name column.
3. To retrieve more information about a specific instance, run cf service SERVICE-INSTANCE-
NAME
4. To retrieve the GUID of the instance, run cf service SERVICE-INSTANCE-NAME --guid
The GUID is useful for debugging.
Description: Dedicated Instance
Documentation url:
Dashboard:
Last Operation
Status: create failed
Message: Instance provisioning failed: There was a problem completing your re
quest.
Please contact your operations team providing the following information:
service: redis-acceptance,
service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089,
broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac,
task-id: 442,
operation: create
Started: 2017-03-13T10:16:55Z
Updated: 2017-03-13T10:17:58Z
$ cf login
$ cf services
Getting services in org my-org / space my-space as [email protected]...
OK
name service plan bound apps last operation
my-instance p.mysql db-small create succeeded
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
270
Use the Knowledge Base (Community)
Find the answer to your question and browse product discussions and solutions by searching the
VMware Tanzu Knowledge Base.
File a Support Ticket
You can file a support ticket here. Be sure to provide the error message from cf service YOUR-
SERVICE-INSTANCE.
To expedite troubleshooting, if possible, provide your service broker logs, service instance logs, and
BOSH task output. Your cloud operator should be able to obtain these from your error message.
You can file a ticket with Support. Be sure to provide the error message from cf service YOUR-
SERVICE-INSTANCE.
To expedite troubleshooting, if possible, provide your service broker logs, service instance logs, and
BOSH task output. Your cloud operator can obtain these from your error message.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
271
Backing Up and Restoring VMware Tanzu
SQL with MySQL for VMs
This topic describes how to back up and restore a physical backup to a VMware Tanzu SQL with
MySQL for VMs service instance. Both operators and developers can use the procedures in this
topic.
Operators can configure automated backups in the Tanzu SQL for VMs tile by following the
procedures in Configuring Automated Backups.
Developers can manually take physical backups by following the procedure in this topic. Developers
can also create logical backups using mysqldump. For more information about logical backups, see
Backing Up and Restoring with mysqldump.
Overview
You can back up and restore a Tanzu SQL for VMs service instance using the Cloud Foundry
Command Line Interface (cf CLI). The backup artifacts you create in this topic are physical backups.
The following procedures do not create logical backup artifacts. Restoring a physical backup is
faster than restoring a logical backup and has less downtime.
You might want to back up or restore a service instance in the following use cases:
Disaster recovery
Troubleshooting
Note: The backup and restore capability described in this topic restores a
running
service instance's backup to a new instance. It is not intended to list or restore
backups created by a deleted service instance. For more information about
restoring a backup from a deleted service instance, see Manually Restoring From
Backup.
Note: In Tanzu SQL for VMs v2.9.0 to v2.9.2, the Application Developer Backup
and Restore (ADBR) feature stored backup artifacts in the
root
directory in external
storage.
As of v2.9.3, backup artifacts are organized under subfolders in external storage.
They are now stored under p.mysql > service-instance_GUID > yyyy > mm > dd.
Old backup artifacts stored in the root directory can still be accessed through the cf
CLI using the adbr plug-in.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
272
Testing
About Backups
Automated backups do the following:
Periodically create and upload backups for restoring all of the databases used by a service
instance.
Operate without locking apps out of the database. There is no downtime.
Include a metadata file that contains the critical details for the backup, including the
calendar time of the backup.
Encrypt backups within the Tanzu SQL for VMs VM. Unencrypted data is never transferred
outside the Tanzu SQL for VMs deployment.
Run at scheduled time. Operators configure the default schedule in Ops Manager.
Developers can change the backup schedule for an individual service instance using an
arbitrary parameter. For more information, see Backup Schedule.
Automated backups fail if the service broker is unavailable. For example, during an upgrade.
About Backup Files and Metadata
When Tanzu SQL for VMs runs a backup, it uploads two files with Unix epoch-timestamped
filenames:
The encrypted data backup file, mysql-backup-TIMESTAMP.tar.gpg
A metadata file, mysql-backup-TIMESTAMP.txt
The metadata file contains information about the backup that looks similar to the following:
ibbackup_version = 2.4.5
end_time = 2017-04-24 21:00:03
lock_time = 0
binlog_pos =
incremental = N
encrypted = N
server_version = 5.7.16-10-log
Note: You must configure backups in the Tanzu SQL for VMs tile. You cannot be
deactivate this feature.
Note: As of v2.9, only the leader node is backed up for a leader-follower
deployment.
Warning: If Tanzu SQL for VMs fails to upload a backup, the backup artifact remains
on the persistent disk. This can cause the persistent disk to fill up faster. If the
persistent disk is full, apps become inoperable. For instructions on how to
troubleshoot persistent disk errors, see Persistent Disk is Full.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
273
start_time = 2017-04-24 21:00:00
tool_command = --user=admin --password=... --stream=tar tmp/
innodb_from_lsn = 0
innodb_to_lsn = 2491844
format = tar
compact = N
name =
tool_name = innobackupex
partial = N
compressed = N
uuid = fd13cf26-2930-11e7-871e-42010a000807
tool_version = 2.4.5
The most important entries in the metadata file are the following:
start_time: Use this to determine which transactions are captured in the backup.
server_version: Use this to determine potential incompatibilities when restoring an
instance with the backup artifact.
The backup process does not interrupt the MySQL service, however backups only reflect
transactions that are completed before the start_time.
Prerequisite: adbr Plug in
Before you can manually back up or restore a service instance, you must have installed the
ApplicationDataBackupRestore (adbr) plug-in for the Cloud Foundry Command Line Interface (cf
CLI) tool.
To install the plug-in, run:
cf install-plugin -r CF-Community "ApplicationDataBackupRestore"
Manually Back Up a Service Instance
To manually trigger a backup outside of the configured schedule:
1. Back up your service instance by running:
cf adbr backup SOURCE-INSTANCE
Where SOURCE-INSTANCE is the service instance you are backing up.
2. View the status of the backup process by running:
Note: The metadata file sets compressed encrypted as N, which indicates that the
backup is not compressed or encrypted. However, the backup uploaded by Tanzu
SQL for VMs is both compressed and encrypted. This is a known issue.
Note: The procedures on this page assume that you are using the adbr plug-in
v0.3.0 or later.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
274
cf adbr get-status SOURCE-INSTANCE
For example:
Restore a Service Instance
When restoring a service instance, you create a new service instance and then restore the backup
to it. If you are restoring leader-follower or HA cluster service instance, you update the plan after
restoring the data. Finally, you rebind and restage apps that use the service instance.
About Restoring a Service Instance
Before you begin the restore procedure review the following important concepts:
The Topology of the Instance You Restore To
When to Use a Different Restore Method
The New Database Must Be Empty
The Topology of the Instance You Restore To
You can only restore to a single node or a Multi‑Site Replication service instance. If you want a
different topology, update the instance after restoring the data:
To restore to… First, restore to… Then, update to…
Leader-Follower Single node Leader-Follower
HA cluster Multi‑Site Replication HA cluster
Multi‑Site Replication Multi‑Site Replication
n/a
Single node Single node
n/a
Operators must ensure that the persistent disk in the single node plan is at least as large as the
persistent disk of the largest leader-follower plan. Similarly, they must ensure that the persistent
disk in the Multi‑Site Replication plan is at least as large as the persistent disk of the largest HA
cluster plan. For information about persistent disk sizing recommendations, see Persistent Disk
Usage.
When to Use a Different Restore Method
There are some cases where you cannot use the following procedure:
Do not use the procedure in this section to restore a
backup…
Instead, see…
$ cf adbr get-status myDB
Getting status of service instance myDB in org my-org / space my-spac
e as user...
[Fri May 28 18:08:25 UTC 2021] Status: Backup was successful. Uploade
d 3.2M
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
275
created with Tanzu SQL for VMs v2.8 or earlier Restoring Tanzu SQL for VMs.
from a deleted service instance Manually Restoring from Backup.
to a different foundation Manually Restoring from Backup.
The New Database Must Be Empty
Do not write anything to the database in the new service instance until after you have restored the
backup to it. For example, the following actions write to the database:
You create service keys.
You configure the service instance for Multi‑Site Replication.
Tanzu SQL for VMs takes an automatic backup.
If the database is not empty, then the adbr plug-in fails to restore the data and you need to delete
the service instance and create another.
Procedure to Restore a Service Instance
To restore a service instance:
1. Create a new MySQL service instance to restore to by running:
cf create-service p.mysql NEW-PLAN DESTINATION-INSTANCE
Where:
NEW-PLAN is the name of the service plan for your new service instance. The plan you
choose depends on the service instance topology that you are restoring. If the
topology that you are restoring is:
Single node or leader-follower: Select a single node plan.
Multi‑Site Replication or HA cluster: Select a Multi‑Site Replication plan.
DESTINATION-INSTANCE is a name that you choose to identify the service instance.
You restore the backup artifact to this service instance. For more information, see Create a
Service Instance.
2. View the available backup artifacts for your service instance by running:
cf adbr list-backups SOURCE-INSTANCE --limit NUMBER
Where:
SOURCE-INSTANCE is the name of the service instance.
--limit NUMBER is an optional flag to specify how many backups to list. Without the
flag, the command lists five backups.
For example:
$ cf adbr list-backups myDB --limit 1
Getting backups of service instance myDB in org my-org / space my-spa
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
276
Record the Backup ID from the output.
3. Restore the service instance by running:
cf adbr restore DESTINATION-INSTANCE BACKUP-ID
Where is BACKUP-ID is the ID you recorded in the previous step.
For example:
4. View the status of the restore process by running:
cf adbr get-status DESTINATION-INSTANCE
For example:
If the status is Restore failed, see The New Database Must Be Empty.
5. If you are restoring a leader-follower or HA cluster service instance, update the plan by
running:
cf update-service DESTINATION-INSTANCE -p PLAN
Where PLAN is the plan for the topology that you want to restore.
6. Determine if your app is bound to a service instance by running:
cf services
For example:
7. If an app is currently bound to a service instance, unbind it by running:
ce as user...
Backup ID Time of Backup
a41bf723-2538-4020-9d16-50cccb7b7c8d_1589825284 Fri May 28 18:08:04
UTC 2021
$ cf adbr restore myTargetDB a41bf723-2538-4020-9d16-50cccb7b7c8d_15898
25284
$ cf adbr get-status myTargetDB
Getting status of service instance myTargetDB in org my-org / space m
y-space as user...
[Mon May 31 22:29:24 UTC 2021] Status: Restore was successful
$ cf services
Getting services in org my-org / space my-space as user...
OK
name service plan bound apps last operation
myDB p.mysql db-small my-app create succeeded
myTargetDB p.mysql db-small create succeeded
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
277
cf unbind-service APP-NAME SOURCE-INSTANCE
8. Update your app to bind to the new service instance by running:
cf bind-service APP-NAME DESTINATION-INSTANCE
9. Restage your app by running:
cf restage APP-NAME
Troubleshooting the adbr Plug-in
If you get HTTP error codes when working with the adbr plug-in, see Failed Backup or Restore
with adbr Plug-in in
Troubleshooting On-Demand Instances
.
Monitor Backups
It is particularly important to verify that automated backups are being taken according to schedule.
A common cause of failure for automated backups is the persistent disk filling up.
There are two ways to check that backups are being made:
Use the adbr Plug-in to List Backups
Use Healthwatch to Confirm Automated Backups
Use the adbr Plug-in to List Backups
The adbr plug-in is a quick way to list backups for a service instance. The plug-in lists the last five
backups for a service instance. The list includes both automated and manual backups.
1. List the backups for an instance using the adbr plug-in by running:
cf adbr list-backups SOURCE-INSTANCE --limit NUMBER
Where:
SOURCE-INSTANCE is the name of the service instance.
--limit NUMBER is an optional flag to specify how many backups to list. Without the
flag, the command lists five backups.
For example:
$ cf adbr list-backups myDB
Getting backups of service instance myDB in org my-org / space my-space
as admin...
Backup ID Time of Backup
f4df63d3-ece1-4b53-99a4-4b55ad10af2f_1596675600 Wed Jun 2 01:00:00 U
TC 2021
f4df63d3-ece1-4b53-99a4-4b55ad10af2f_1596675420 Wed Jun 2 00:57:00 U
TC 2021
f4df63d3-ece1-4b53-99a4-4b55ad10af2f_1596675240 Wed Jun 2 00:54:00 U
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
278
Use Healthwatch to Confirm Automated Backups
You can use Healthwatch to confirm that automated backups are being taken on the schedule
configured by the operator. For information about configuring the schedule for automated backups,
see Configuring Automated Backups.
For each backup of every service instance, Tanzu SQL for VMs emits a metric called
last_successful_backup.
1. Monitor the last_successful_backup metric through Healthwatch.
For details about the metric, see Hours Since Last Successful Backup in
Monitoring and
KPIs for VMware Tanzu SQL with MySQL for VMs
.
A healthy backup metric shows a saw-tooth plot. The blue stepped line is the time elapsed
since the last backup. The dotted red line represents the scheduled time when an
automated backup is expected to occur. When a backup is taken, the time elapsed resets
to zero.
View a larger version of this image
TC 2021
f4df63d3-ece1-4b53-99a4-4b55ad10af2f_1596675000 Wed Jun 2 00:50:00 U
TC 2021
f4df63d3-ece1-4b53-99a4-4b55ad10af2f_1596674700 Wed Jun 2 00:45:00 U
TC 2021
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
279
Monitoring and KPIs for VMware Tanzu SQL
with MySQL for VMs
This topic describes how to monitor the health of the VMware Tanzu SQL with MySQL for VMs
service using logs, metrics, and Key Performance Indicators (KPIs) that are generated by Tanzu
SQL for VMs component VMs.
For more information about logging and metrics in VMware Tanzu Application Service for VMs, see
Overview of Logging and Metrics.
About Metrics
Metrics are regularly generated log entries that report measured component states. The default
metrics polling interval is 30 seconds for MySQL instances. For the service broker, the default is 60
seconds.
You can configure the MySQL instance interval in Configure Monitoring in the Tanzu SQL for VMs
tile. For more information, see Configure Monitoring.
Metrics are long, single lines of text with the format:
origin:"p.mysql" eventType:ValueMetric timestamp:1496776477930669688 deployment:"servi
ce-instance_2b5a001f-2bf3-460c-aee6-fd2253f9fb0c" job:"mysql" index:"b09df494-b731-4d0
6-a4b0-c2985ceedf4c" ip:"10.0.8.4" valueMetric:<name:"/p.mysql/performance/open_files"
value:24 unit:"file" >
Access MySQL Metrics
To access MySQL metrics:
Use the Indicator Protocol Dashboard
Use Log Cache
Use Grafana
You can use Grafana to visually view metrics for Tanzu SQL for VMs service instances.
1. Install the Healthwatch tile in Ops Manager. For information about installing Healthwatch,
see the Healthwatch for VMware Tanzu documentation.
2. View the Grafana dashboard by navigating to:
Note: This section requires Healthwatch v2.x.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
280
https://grafana.YOUR-SYSTEM-DOMAIN
For more information about using Grafana dashboards, see the Healthwatch for VMware Tanzu
documentation.
Use the Indicator Protocol Dashboard
You can use the Indicator Protocol dashboard to view Key Performance Indicators (KPIs) for Tanzu
SQL for VMs service instances. The dashboard includes a graphical representation of the KPIs and
information about recommended alert thresholds.
For more information about the Indicator Protocol dashboard, see Indicator Protocol Dashboard
(Beta).
For information about Tanzu SQL for VMs KPIs, see KPIs for MySQL Service Instances.
To access KPIs using the Indicator Protocol dashboard:
1. Install the Healthwatch tile in Ops Manager. See Installing and Configuring Pivotal
Healthwatch.
2. Enable the Indicator Protocol add-on in Ops Manager. See Enable the Indicator Protocol
Add-on.
3. View the Indicator Protocol Dashboard by navigating to:
https://healthwatch.YOUR-SYSTEM-DOMAIN/indicator-dashboards
The following image shows an example of the Indicator Protocol dashboard for a highly available
(HA) cluster service instance:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
281
View a larger version of this image
Use Log Cache
To access metrics for Tanzu SQL for VMs service instances, you can use Loggregator's Log Cache
feature with the Log Cache CLI plugin. Log Cache is enabled by default.
To access metrics for on-demand service instances:
1. Install the cf CLI plugin by running:
cf install-plugin -r CF-Community "log-cache"
2. To access metrics for a service instance, run:
Note: To use this feature the V2 Firehose must be enabled and Enable Log Cache
syslog ingestion must be disabled in the TAS for VMs tile. For more information
about configuring these checkboxes, see Enable Syslog Forwarding.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
282
cf tail SERVICE-INSTANCE-NAME
Two useful flags to append to this command are:
-f, --follow: Append metrics logs to stdout as they are generated, rather than
returning a fixed number of metrics or metrics over a fixed interval.
--json: Output metrics logs as envelopes in JSON format. For a complete list of cf
tail options, see the Log Cache CLI repository in GitHub.
Where SERVICE-INSTANCE-NAME is the name of your service instance.
For example:
For more information about the metrics output, see the Key Performance Indicators and
Component Metrics sections below.
For more information about how to enable Log Cache and about the cf tail command, see
Enable Log Cache.
KPIs for MySQL Service Instances
KPIs are metrics for MySQL service instances that you can monitor for two purposes:
To ensure high performance.
To discover emerging issues.
KPIs can be either raw component metrics or derived metrics generated by applying formulas to
raw metrics.
VMware provides the following KPIs as general alerting and response guidance for typical Tanzu
SQL for VMs installations. VMware recommends that you alter the alert measures by observing
historical trends. You can also create your KPIs that are specific to your environment using the
available component metrics.
For a list of all the Tanzu SQL for VMs component metrics, see Component Metrics.
Server Availability
$ cf tail -f my-instance | egrep 'connected|available|persistent_disk_u
sed_percent'
2019-05-17T11:25:59.48-0700 [my-instance] GAUGE /p.mysql/available:1.00
0000 boolean
2019-05-17T11:26:29.49-0700 [my-instance] GAUGE /p.mysql/system/persist
ent_disk_used_percent:17.000000 percentage
2019-05-17T11:26:29.49-0700 [my-instance] GAUGE /p.mysql/performance/th
reads_connected:**6**.000000 connection
2019-05-17T11:26:59.50-0700 [my-instance] GAUGE /p.mysql/available:1.00
0000 boolean
2019-05-17T11:27:29.50-0700 [my-instance] GAUGE /p.mysql/system/persist
ent_disk_used_percent:17.000000 percentage
2019-05-17T11:27:29.50-0700 [my-instance] GAUGE /p.mysql/performance/th
reads_connected:**7**.000000 connection
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
283
/p.mysql/available
Description If the MySQL Server is currently responding to requests. This indicates if the component is available.
Use: If the server does not emit heartbeats, it is offline.
Origin: Doppler/Firehose
Type: boolean
Frequency: 30 s
Recommended
measurement
Average over last 5 minutes
Recommended
alert
thresholds
Yellow warning: N/A
Red critical: < 1
Recommended
response
Check the MySQL Server logs for errors. You can find the instance by targeting your MySQL
deployment with BOSH and inspecting logs for the instance. For more information, see Failing Jobs
and Unhealthy Instances.
If your service plan is an highly available (HA) cluster, you can also run mysql-diag to check logs for
errors.
Persistent Disk Used
/p.mysql/system/persistent_disk_used_percent
Description The percentage of disk used on the persistent file system.
Use: MySQL cannot function correctly if there is not sufficient free space on the file systems. Use
these metrics to ensure that you have disks large enough for your user base.
Origin: Doppler/Firehose
Type: Percent
Frequency: 30 s (default)
Recommended
measurement
Maximum of persistent disk used of all of nodes
Recommended
alert thresholds
Single Node and Leader Follower:
Yellow warning: > 25%
Red critical: > 30%
Highly Available Cluster:
Yellow warning: > 80%
Red critical: > 90%
Recommended
response
Upgrade the service instance to a plan with larger disk capacity.
For Tanzu SQL for VMs v2.9 and later, if you set the optimize_for_short_words parameter to
true, then see Persistent Disk Usage Is Increasing before upgrading the service.
Ephemeral Disk Used
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
284
/p.mysql/system/ephemeral_disk_used_percent
Description The percentage of disk used on the ephemeral file system.
Use: MySQL cannot function correctly if there is not sufficient free space on the file systems. Use
these metrics to ensure that you have disks large enough for your user base.
Origin: Doppler/Firehose
Type: Percent
Frequency: 30 s (default)
Recommended
measurement
Maximum disk used of all nodes
Recommended
alert thresholds
Yellow warning: > 80%
Red critical: > 95%
Recommended
response
Upgrade the service instance to a plan with larger disk capacity.
CPU Utilization Percent
/p.mysql/performance/cpu_utilization_percent
Description CPU time being consumed by the MySQL service.
Use: A node that experiences context switching or high CPU usage becomes unresponsive. This
also affects the ability of the node to report metrics.
Origin: Doppler/Firehose
Type: Percent
Frequency: 30 s (default)
Recommended
measurement
Average over last 10 minutes
Recommended alert
thresholds
Yellow warning: > 80 Red critical: > 90
Recommended
response
Determine what is using so much CPU. If it is from normal processes, update the service
instance to use a plan with larger CPU capacity.
Connections
/p.mysql/variables/max_connections
/p.mysql/net/max_used_connections
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
285
Description The maximum number of connections used over the maximum permitted number of simultaneous
client connections.
Use: If the number of connections drastically changes or if apps are unable to connect, there might be
a network or app issue.
Origin: Doppler/Firehose
Type: count
Frequency: 30 s
Recommended
measurement
max_used_connections / max_connections
Recommended
alert
thresholds
Yellow warning: > 80 %
Red critical: > 90 %
Recommended
response
If this measurement mets or exceeds 80% with exponential growth, monitor app usage to ensure
everything is working.
When approaching 100% of max connections, apps may be experiencing times when they cannot
connect to the database. The connections/second for a service instance vary based on app instances
and app utilization.
Queries Delta
/p.mysql/performance/queries_delta
Description The number of statements executed by the server over the last 30 seconds.
Use: The server should always be processing some queries. If the server does not process any
queries, the server is non-functional.
Origin: Doppler/Firehose
Type: count
Frequency: 30 s
Recommended
measurement
Average over last 2 minutes
Recommended alert
thresholds
Red critical: 0
Recommended
response
Investigate the MySQL server logs, such as the audit log, to understand why query rate
changed and determine appropriate action.
Highly Available Cluster WSREP Ready
/p.mysql/galera/wsrep_ready
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
286
Description Shows whether each cluster node can accept queries. Returns only 0 or 1. When this metric is 0,
almost all queries to that node fail with the error:
ERROR 1047 (08501) Unknown Command
Use: Discover when nodes of a cluster have been unable to communicate and accept
transactions.
Origin: Doppler/Firehose
Type: boolean
Frequency: 30 s (default)
Recommended
measurement
Average of values of each cluster node, over the last 5 minutes
Recommended alert
thresholds
Yellow warning: < 1
Red critical: 0 (cluster is down)
Recommended
response
Run mysql-diag and check the MySQL Server logs for errors.
Ensure no infrastructure event is affecting intra-cluster communication.
Ensure that wsrep_ready is not set to off by using the query:
SHOW STATUS LIKE 'wsrep_ready';.
Highly Available Cluster WSREP Cluster Size
/p.mysql/galera/wsrep_cluster_size
Description The number of cluster nodes with which each node is communicating normally.
Use: When running in a multi-node configuration, this metric indicates if each member of the
cluster is communicating normally with all other nodes.
Origin: Doppler/Firehose
Type: count
Frequency: 30 s (default)
Recommended
measurement
(Average of the values of each node / cluster size), over the last 5 minutes
Recommended alert
thresholds
Yellow warning: < 3 (availability compromised)
Red critical: < 1 (cluster unavailable)
Recommended
response
Run mysql-diag and check the MySQL Server logs for errors.
Highly Available Cluster WSREP Cluster Status
/p.mysql/galera/wsrep_cluster_status
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
287
Description Shows the primary status of the cluster component that the node is in.
Values are:
Primary: Node has a quorum.
Non-primary: Node has lost a quorum.
Disconnected: Node is unable to connect to other nodes.
Use: Any value other than Primary indicates that the node is part of a non-operational component.
This occurs in cases of multiple membership changes that result in a loss of quorum.
Origin: Doppler/Firehose
Type: integer (see above)
Frequency: 30 s (default)
Recommended
measurement
Sum of each of the nodes, over the last 5 minutes
Recommended
alert thresholds
Yellow warning: < 3
Red critical: < 1
Recommended
response
Check node status to ensure that they are all in working order and able to receive write-
sets
Run mysql-diag and check the MySQL Server logs for errors
Hours Since Last Successful Backup
/p.mysql/p.mysql/last_successful_backup
Description Using the configured backup schedule for the service instance as a threshold, this metric shows
how many hours have passed since the last successful backup.
Recommended
measurement
Hours elapsed since the last successful backup
Recommended
alert thresholds
Red critical: Backup did not run on schedule.
Recommended
response
Check the adbr-agent logs for the service instance to determine why recent automated backups
are failing.
Component Metrics
In addition to the KPIs, the MySQL service emits the followings metrics for monitoring and alerting:
MySQL Metrics
Disk Metrics
Leader-Follower Metrics
Highly Available Cluster Metrics
MySQL Metrics
The metrics that all Tanzu SQL for VMs service instances emit:
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
288
/p.mysql/available
Description Indicates if the local database server is available and responding.
Unit boolean
/p.mysql/variables/max_connections
Description The maximum permitted number of simultaneous client connections.
Unit count
/p.mysql/variables/open_files_limit
Description The number of files that the operating system permits mysqld to open.
Unit files
/p.mysql/variables/read_only
Description Whether the server is in read-only mode
Unit boolean
/p.mysql/performance/questions
Description The number of statements run by the server since the server started or the last FLUSH STATUS.
This includes only statements sent to the server by clients and not statements run within stored
programs, unlike the queries variable.
Unit count
/p.mysql/performance/queries
Description The number of statements run by the server, excluding COM_PING and COM_STATISTICS. Differs
from questions in that it also counts statements run within stored programs. Not affected by
FLUSH STATUS.
Unit count
/p.mysql/performance/queries_delta
Description The change in the /performance/queries metric since the last time it was emitted.
Unit integer greater than or equal to zero
/p.mysql/innodb/buffer_pool_pages_free
Description The amount of free space, measured in pages, in the InnoDB Buffer Pool.
Unit count
/p.mysql/innodb/buffer_pool_pages_total
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
289
Description The total amount of free space, measured in pages, in the InnoDB Buffer Pool containing data.
Unit count
/p.mysql/innodb/buffer_pool_pages_data
Description The number of pages in the InnoDB Buffer Pool containing data. The number includes both dirty
and clean pages.
Unit count
/p.mysql/innodb/row_lock_current_waits
Description The number of row locks currently being waited for by operations on InnoDB tables.
Unit count
/p.mysql/innodb/data_read
Description The amount of data read since the server started.
Unit bytes
/p.mysql/innodb/data_written
Description The amount of data written since the server started.
Unit bytes
/p.mysql/innodb/mutex_os_waits
Description The number of mutex OS waits.
Unit events/second
/p.mysql/innodb/mutex_spin_rounds
Description The number of mutex spin rounds.
Unit events/second
/p.mysql/innodb/mutex_spin_waits
Description The number of mutex spin waits.
Unit events/second
/p.mysql/innodb/os_log_fsyncs
Description The number of fsync() writes done to the InnoDB redo log files.
Unit count
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
290
/p.mysql/innodb/row_lock_time
Description Total time spent in acquiring row locks.
Unit milliseconds
/p.mysql/innodb/row_lock_waits
Description The number of times a row lock had to be waited for since the server started.
Unit count
/p.mysql/net/connections
Description The number of connection attempts to the server, both successful and unsuccessful, to the
MySQL server.
Unit count
/p.mysql/net/max_used_connections
Description The maximum number of connections that have been in use simultaneously since the server
started.
Unit count
/p.mysql/performance/com_delete
Description The number of delete commands since the server started or the last FLUSH STATUS.
Unit count
/p.mysql/performance/com_delete_multi
Description The number of delete-multi commands since the server started or the last FLUSH STATUS.
Applies to DELETE statements that use multiple-table syntax.
Unit count
/p.mysql/performance/com_insert
Description The number of insert commands since the server started or the last FLUSH STATUS.
Unit count
/p.mysql/performance/com_insert_select
Description The number of insert-select commands since the server started or the last FLUSH STATUS.
Unit count
/p.mysql/performance/com_replace_select
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
291
Description The number of replace-select commands since the server started or the last FLUSH STATUS.
Unit count
/p.mysql/performance/com_select
Description The number of select commands since the server started or the last FLUSH STATUS. If a query
result is returned from query cache, the server increments the qcache_hits status variable, not
com_select.
Unit count
/p.mysql/performance/com_update
Description The number of update commands since the server started or the last FLUSH STATUS.
Unit count
/p.mysql/performance/com_update_multi
Description The number of update-multi commands since the server started or the last FLUSH STATUS.
Applies to UPDATE statements that use multiple-table syntax.
Unit count
/p.mysql/performance/created_tmp_disk_tables
Description The number of internal on-disk temporary tables created by the server while executing
statements.
Unit count
/p.mysql/performance/created_tmp_files
Description The number of temporary files created by mysqld.
Unit count
/p.mysql/performance/created_tmp_tables
Description The number of internal temporary tables created by the server while executing statements.
Unit count
/p.mysql/performance/cpu_utilization_percent
Description The percent of the CPU in use by all processes on the MySQL node.
Unit percent
/p.mysql/performance/open_files
Description The number of regular files currently open, which were opened by the server.
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
292
Unit count
/p.mysql/performance/open_tables
Description The number of tables that are currently open.
Unit count
/p.mysql/performance/opened_tables
Description The number of tables that have been opened.
Unit count
/p.mysql/performance/open_table_definitions
Description The number of currently cached table definitions or .frm files.
Unit count
/p.mysql/performance/opened_table_definitions
Description The number of .frm files that have been cached.
Unit count
/p.mysql/performance/qcache_hits
Description The number of query cache hits. The query cache and qcache_hits metric is deprecated as of
MySQL v5.7.20.
Unit count
/p.mysql/performance/slow_queries
Description The number of queries that have taken more than long_query_time seconds.
Unit count
/p.mysql/performance/table_locks_waited
Description The total number of times that a request for a table lock could not be granted immediately and
a wait was needed.
Unit count
/p.mysql/performance/threads_connected
Description The number of currently open connections.
Unit count
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
293
/p.mysql/performance/threads_running
Description The number of threads that are not sleeping.
Unit count
/p.mysql/rpl_semi_sync_master_tx_avg_wait_time
Description The average time the leader has waited for the follower to accept transactions.
Unit microseconds
/p.mysql/rpl_semi_sync_master_no_tx
Description The number of transactions committed without follower acknowledgement.
Unit count
/p.mysql/rpl_semi_sync_master_wait_sessions
Description The current number of connections waiting for a sync commit. For more information about
sync replication, see About Synchronous Replication.
Unit count
Disk Metrics
The disk usage metrics that all Tanzu SQL for VMs services emit:
/p.mysql/system/persistent_disk_used_percent
Description The percentage of disk used on the persistent file system.
Unit percent
/p.mysql/system/persistent_disk_used
Description The amount of space used on the persistent disk.
Unit KB
/p.mysql/system/persistent_disk_free
Description The amount of space available on the persistent disk.
Unit KB
/p.mysql/system/persistent_disk_inodes_used_percent
Description The percentage of persistent disk inodes used by both the system and user applications.
Unit percent
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
294
/p.mysql/system/persistent_disk_inodes_used
Description The number of inodes used on the persistent disk.
Unit count
/p.mysql/system/persistent_disk_inodes_free
Description The number of inodes available on the persistent disk.
Unit count
/p.mysql/system/ephemeral_disk_used_percent
Description The percentage of disk used on the ephemeral file system.
Unit percent
/p.mysql/system/ephemeral_disk_used
Description The amount of space used on the ephemeral disk.
Unit KB
/p.mysql/system/ephemeral_disk_free
Description The amount of space available on the ephemeral disk.
Unit KB
/p.mysql/system/ephemeral_disk_inodes_used_percent
Description The percentage of ephemeral disk inodes used by both the system and user applications.
Unit percent
/p.mysql/system/ephemeral_disk_inodes_used
Description The number of inodes used on the ephemeral disk.
Unit count
/p.mysql/system/ephemeral_disk_inodes_free
Description The number of inodes available on the ephemeral disk.
Unit count
Leader-Follower Metrics
The metrics that leader-follower VMs emit:
/p.mysql/follower/is_follower
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
295
Description Shows whether a node is the follower VM.
Unit boolean
/p.mysql/follower/seconds_behind_master
Description The number of seconds the follower is behind in applying writes from the leader. For example,
a follower might have copied writes from the leader that are timestamped up to 11:00am, but
has only applied writes up to 8:00am. Normal values for this metric depend on your apps.
Unit seconds
/p.mysql/follower/seconds_since_leader_heartbeat
Description The number of seconds that elapse between the leader heartbeat and the replication of the
heartbeat in the follower. You can use this metric to determine how far behind the follower is
from the leader. Normal values for this metric depends on your app.
Unit seconds
/p.mysql/follower/relay_log_space
Description The total size of all existing relay log files.
Unit bytes
/p.mysql/follower/slave_io_running
Description Shows whether the I/O thread has started and has connected to the leader VM.
Unit boolean
/p.mysql/follower/slave_sql_running
Description Shows whether the SQL thread has started.
Unit boolean
Highly Available Cluster Metrics
The metrics that HA clusters emit:
/p.mysql/galera/wsrep_cluster_size
Description The current number of nodes in the HA cluster.
Unit count
/p.mysql/galera/wsrep_local_recv_queue
Description The current length of the local receive queue, in messages.
Unit count
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
296
/p.mysql/galera/wsrep_local_send_queue
Description The current length of the local send queue, in messages.
Unit count
/p.mysql/galera/wsrep_local_index
Description This node index in the cluster (base 0).
Unit count
/p.mysql/galera/wsrep_local_state
Description The local state of the node. Possible states include:
1 = JOINING
2 = DONOR/DESYNCED
3 = JOINED
4 = SYNCED
Unit integer
/p.mysql/galera/wsrep_ready
Description Shows whether the node can accept queries.
Unit boolean
/p.mysql/galera/wsrep_cluster_status
Description Shows the primary status of the cluster component that the node is in. Values are:
Primary: Node has a quorum.
Non-primary: Node has lost a quorum.
Disconnected: Node is unable to connect to other nodes.
Unit Status code
/p.mysql/galera/wsrep_flow_control_paused
Description Proportion of time, as a unit interval (0 to 1), that replication was paused due to flow control
since the server started or last FLUSH STATUS. This metric is a measure of how much replication
lag is slowing down the cluster.
Unit float
/p.mysql/galera/wsrep_flow_control_sent
Description Number of FC_PAUSE or flow control pause events sent by this node. Unlike many status
variables, the counter for this metric does not reset every time you run the query.
Unit count
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
297
/p.mysql/galera/wsrep_flow_control_recv
Description Number of FC_PAUSE or flow control pause events received by this node. Unlike many status
variables, the counter for this metric does not reset every time you run the query.
Unit count
VMware SQL with MySQL for Tanzu Application Service 2.9
VMware by Broadcom
298
