Consider the following situation: the database of a WordPress application deployed from a store to a GCP-hosted Kubernetes cluster becomes corrupted. Causes for database corruption may vary. Often the cause may be a faulty custom plugin, but it may also happen during common administrative tasks such as data import/export. The effect is however almost always the same - your web-page is not operational until the database and data are restored.
WordPress uses a relational database to store blog articles, their related objects and metadata, and the local file system to store assets, such as pictures in a blog post. On the Kubernetes platform the container's file system is not suitable to store data due to its ephemeral nature, as a result WordPress requires Persistent Volumes to store data.
By default, WordPress requires two pods, one for the application server (wordpress-b68f79995-t7sjh) and one for the MySQL server (wordpress-mysql-5788678cf7-wvr4v). The following figure shows Persistent Volume Claims, Persistent Volumes, and Pods which are using those volumes.
We can easily identify the MySQL and WordPress Persistent Volumes by their claims. To backup this application data, we need to backup both Persistent Volumes.
In following example, we identify Nodes (VMs) that are running our application pods, using the kubectl command:
On GCP, we see the VM instance hosting the two pods is named "gke-cluster1-default-pool-4846a3a4-1r39".
In HYCU Backup and Recovery service on GCP, the instance from the example is represented as following:
Protecting the Kubernetes persistent volumes
Applications hosted on Kubernetes have a myriad of options how to store their data. For persistent volumes which reside on worker nodes, you can simply enable data protection through backup of their instances on GCP.
In HYCU Backup and Recovery service all your Kubernetes cluster workers will be visible in the Instances panel.
Make sure you are on the right Google Cloud Platform project.
- Select the worker node instances that you want to back up.
- Click Policies. The Policies dialog box appears.
- From the list of policies, select the desired backup policy.
- Click Assign to assign the backup policy to the selected instances.
For more information on choosing the right backup policy for your needs, prerequisites and details on backing up instances see "Chapter 4 - Defining your backup strategy" and "Chapter 5 - Protecting instances" in the HYCU Backup and Recovery as a Service for GCP User Guide. (https://docs.gcp.hycu.com/HYCU-Backup-and-Recovery-as-a-Service-for-GCP-User-Guide.pdf)
Restoring the application
Please note that depending on what was corrupted and your Kubernetes configuration, you may need to restore all worker nodes.
To manage your worker node instance backup snapshots, sign into HYCU service and navigate to the Instances panel:
- In the Instances panel, click the worker node instance that you want to restore to open the Details section.
- In the Details section that appears at the bottom of the screen, select the desired restore point.
- Click Restore Instance. The Instance Restore Options dialog box opens.
- Select Restore As and click Next.
5. Enter arbitrary a name for the instance.
6. From the Disks drop-down list, select the instance disk that you want to restore.
7. Click Restore.
The restored VM instance needs to be deleted, since we only need restored application disks.
To delete the VM instance in GCP:
- Navigate to Compute Engine.
- In Compute Engine select the VM Instances view.
- Select the restored VM instance and delete it.
Export Persistent Volume configuration files using the kubectl command. See the following figure for an example:
Because we need to change the Persitent Volume configuration, the old Persistent Volumes need to be deleted. See the following example:
To find the restored disks name in GCP, navigate to the Compute Engine view and select Disks. The name will be formatted like "hycu-disk-<id>-<old disk name>", see the following image for an example.
Using a text editor open the exported PV files, delete the claimRef section and replace the pdName value with the restored disk name, as highlighted in following example.
Use the kubectl command to apply the changed PV files.
For GCP to be able to attach the restored disks to the cluster we need to delete the current running pods which are still referencing the deleted disks. See the following example:
After the new set of pods is started, the application should be up and running.
The old disks that are no longer used can be deleted.
Restoring the original disk names (optional)
To keep the name of the disks as it was before the restore you can follow the next steps:
- In GCP navigate to Compute Engine and then to Disks.
- Locate the disks that were restored and in the Actions column select Create snapshot.
- Enter a name (arbitrary) and click Create.
- After the snapshot is created navigate to the Disks panel again and in the upper menu select Create Disk.
- Enter the original disk name for the name of the disk. Select Snapshot as the Source type. From the Source snapshot drop-down menu, select the snapshot you created. Set Region and Zone so that they match the cluster.
- After all the settings are set, click Create.
- If you deleted the persistent volume yaml files, export them again.
- Delete persistent volumes from the cluster using the same commands as in the previous section. See the following example:
- Edit the persistent volume files similarly as in the previous section - delete the claimRef section and enter the original disk name for the pdName value.
- Apply the changed files using the kubectl command:
- Delete the pods again.
- Delete the disks that are no longer in use.