Changes

Page history
Opencell filesystem divided authored by Gerard Funosas's avatar Gerard Funosas
Hide whitespace changes
Inline Side-by-side
Restore-instance-with-backup.md
View page @ 74bdb470
We are using the [`backups_role`](https://github.com/coopdevs/backups_role/) to manage the backups in the OC instances. This role uses Backblaze (B2) and Restic to encrypt and upload the backup.
## Context
We are using the `backups_role` to create and save the snapshot in B2 with Restic.
In other projects with this backups strategy, we use the `restic` command-line tool to download and decrypt the snapshot from B2. This process take 10-30 mins. In this project, this strategy spends more than 5h and we need to change the strategy to download and decrypt a snapshot.
We must follow the next steps:
## Needed data
We need the keys of Backblaze and the Restic password. These secrets are saved in the [`opencell-provisioning`](https://gitlab.com/coopdevs/opencell-provisioning/) project. Show the decrypted secrets with:
```
$ ansible localhost -m debug -a var="<var_name>" -e "@inventory/host_vars/<host>/backups.yml" --ask-vault-pass
```
> In production:
> ```
> $ ansible localhost -m debug -a var="<var_name>" -e "@inventory/host_vars/opencell.coopdevs.org/secrets.yml" --ask-vault-pass
> ```
We need to list the next vars to use in the process:
* Backblaze Key: `backups_role_b2_app_key`
* Backblaze Key ID: `backups_role_b2_app_key_id`
* Restic password: `backups_role_restic_repo_password`
## Prepare the server
We have two options at this point: Create a new server or restore the backup in the current server.
### Create and provision a new server
Buy a new VPS and execute the [`opencell-provisioning`](https://gitlab.com/coopdevs/opencell-provisioning/) to prepare the server.
### Create and provision a new server
If you want to use the same server you need to be sure that you have space to download all the B2 bucket in the server. You can check the size of the bucket in https://secure.backblaze.com/b2_buckets.htm and searching the bucket. Then, if you don't have the space in the server, we can attach a new volume (double the size of the bucket to be more comfortable). This new volume attached can be find inside the server running `df -h`:
```
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 151G 119G 26G 83% /
/dev/sdc 246G 105G 128G 46% /mnt/HC_Volume_8507646 --> The new volume mounted on /mnt/HC_Volume_8507646
```
## Download backup from B2
To download the bucket we need to install and config the [`b2` command-line tool](https://www.backblaze.com/b2/docs/quick_command_line.html):
1. Access to the server where you want to restore:
```
$ ssh <user>@<host>
```
2. Install `b2` with `pip`:
```
$ pip install b2
```
3. Configure `b2`:
```
$ b2 authorize-account [<Backblaze Key ID>] [<Backblaze Key>]
```
4. With the `b2` configured we can download the bucket. You can find the bucket name listing the buckets with `b2 list-buckets`. Be sure that you start a `screen` session to avoid lost the output of the download process, that can take a lot of time:
```
$ screen
$ b2 list-buckets
$ b2 sync --threads 25 b2://<bucket-name> <path-to-save-the-bucket>
```
## Extract the snapshot with Restic
We use the `restic` command-line tool to interact with the encrypted bucket.
1. Install `restic`. If the server had installed the `backup_role`, `restic` was installed, else please follow the installation instructions: https://restic.readthedocs.io/en/stable/020_installation.html#installation
2. Browse snapshots with `restic snapshots` and select which snapshot you want to restore.
```
$ restic snapshots -r <path-of-bucket>
```
3. Use the `restic` tool to decrypt the snapshot:
```
$ sudo restic -r <path-of-bucket> restore <snapshot-id> --target <path-to-restore-snapshot>
```
## Restore backup
0. Access to the server and change to the app user:
```
$ ssh <user>@<host>
$ sudo su - opencell
```
1. Stop the OpenCell container:
```
$ docker stop opencell
```
2. Drop the OpenCell database:
```
$ docker exec -ti postgres dropdb -U opencell opencell
```
3. Create the new OpenCell DB:
```
$ docker exec -ti postgres createdb -U opencell opencell
```
4. Restore the dump in the new DB. Inside the snapshot, we can find a dump of OpenCell, a dump of Keycloak and the OC filesystem with all the printed invoices:
```
# Estructura interna del snapshot
/opt/backup/.tmp/pg_dump_opencell.sql
/opt/backup/.tmp/pg_dump_keycloak.sql
/home/opencell/input-files/opencell-version.txt
/home/opencell/input-files/opencell-admin.properties
/home/opencell/opencelldata
```
```
$ cat <snapshot-path>/opt/backup/.tmp/pg_dump_opencell.sql | docker exec -i postgres psql -U opencell opencell
$ cat <snapshot-path>/opt/backup/.tmp/pg_dump_keycloak.sql | docker exec -i postgres psql -U opencell keycloak
```
5. Restore the filesystem from the backup overriding the `opencelldata` folder:
```
$ sudo cp -r <snapshot-path>/home/opencell/opencelldata /home/opencell/opencelldata
$ sudo chown -R 1000 /home/opencell/opencelldata
```
6. Restore the OC properties:
```
$ sudo cp -r <snapshot-path>/home/opencell/input-files/opencell-admin.properties /home/opencell/input-files/opencell-admin.properties
$ sudo chown -R 1000 /home/opencell/input-files/opencell-admin.properties
```
6. Up the OC container:
```
$ docker-compose up -d
We are using the [`backups_role`](https://github.com/coopdevs/backups_role/) to manage the backups in the OC instances. This role uses Backblaze (B2) and Restic to encrypt and upload the backup.
## Context
We are using the `backups_role` to create and save the snapshot in B2 with Restic.
In other projects with this backups strategy, we use the `restic` command-line tool to download and decrypt the snapshot from B2. This process take 10-30 mins. In this project, this strategy spends more than 5h and we need to change the strategy to download and decrypt a snapshot.
We must follow the next steps:
## Needed data
We need the keys of Backblaze and the Restic password. These secrets are saved in the [`opencell-provisioning`](https://gitlab.com/coopdevs/opencell-provisioning/) project. Show the decrypted secrets with:
```
$ ansible localhost -m debug -a var="<var_name>" -e "@inventory/host_vars/<host>/backups.yml" --ask-vault-pass
```
> In production:
> ```
> $ ansible localhost -m debug -a var="<var_name>" -e "@inventory/host_vars/opencell.coopdevs.org/secrets.yml" --ask-vault-pass
> ```
We need to list the next vars to use in the process:
* Backblaze Key: `backups_role_b2_app_key`
* Backblaze Key ID: `backups_role_b2_app_key_id`
* Restic password: `backups_role_restic_repo_password`
## Prepare the server
We have two options at this point: Create a new server or restore the backup in the current server.
### Create and provision a new server
Buy a new VPS and execute the [`opencell-provisioning`](https://gitlab.com/coopdevs/opencell-provisioning/) to prepare the server.
### Create and provision a new server
If you want to use the same server you need to be sure that you have space to download all the B2 bucket in the server. You can check the size of the bucket in https://secure.backblaze.com/b2_buckets.htm and searching the bucket. Then, if you don't have the space in the server, we can attach a new volume (double the size of the bucket to be more comfortable). This new volume attached can be find inside the server running `df -h`:
```
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 151G 119G 26G 83% /
/dev/sdc 246G 105G 128G 46% /mnt/HC_Volume_8507646 --> The new volume mounted on /mnt/HC_Volume_8507646
```
## Download backup from B2
To download the bucket we need to install and config the [`b2` command-line tool](https://www.backblaze.com/b2/docs/quick_command_line.html):
1. Access to the server where you want to restore:
```
$ ssh <user>@<host>
```
2. Install `b2` with `pip`:
```
$ pip install b2
```
3. Configure `b2`:
```
$ b2 authorize-account [<Backblaze Key ID>] [<Backblaze Key>]
```
4. With the `b2` configured we can download the bucket. You can find the bucket name listing the buckets with `b2 list-buckets`. Be sure that you start a `screen` session to avoid lost the output of the download process, that can take a lot of time:
```
$ screen
$ b2 list-buckets
$ b2 sync --threads 25 b2://<bucket-name> <path-to-save-the-bucket>
```
## Extract the snapshot with Restic
We use the `restic` command-line tool to interact with the encrypted bucket.
1. Install `restic`. If the server had installed the `backup_role`, `restic` was installed, else please follow the installation instructions: https://restic.readthedocs.io/en/stable/020_installation.html#installation
2. Browse snapshots with `restic snapshots` and select which snapshot you want to restore.
```
$ restic snapshots -r <path-of-bucket>
```
3. Use the `restic` tool to decrypt the snapshot:
```
$ sudo restic -r <path-of-bucket> restore <snapshot-id> --target <path-to-restore-snapshot>
```
## Restore backup
0. Access to the server and change to the app user:
```
ssh <user>@<host>
sudo su - opencell
```
1. Stop the OpenCell container:
```
docker stop opencell
```
2. Drop the OpenCell & Keycloak database:
```
docker exec -ti postgres dropdb -U opencell opencell
docker exec -ti postgres dropdb -U opencell keycloak
```
3. Create the new OpenCell & Keycloak DB:
```
docker exec -ti postgres createdb -U opencell opencell
docker exec -ti postgres createdb -U opencell keycloak
```
4. Restore the dump in the new DB.
We have the OC filesystem snapshot separated from the Opencell & Keycloak DB dump, which is found in another bucket in BackBlaze.
- OC DB bucket:
```
/mnt/HC_Volume_100466521/backup/pg_dump_opencell.sql
/mnt/HC_Volume_100466521/backup/pg_dump_keycloak.sql
/home/opencell/input-files/opencell-version.txt
/home/opencell/input-files/opencell-admin.properties
```
- OC filesystem bucket:
```
/mnt/HC_Volume_9375116/opencelldata
```
The OC filesystem is not growing, since invoices are not stored in it anymore. Therefore, we are not updating its content in its corresponding bucket.
```
cat <db-path>/pg_dump_opencell.sql | docker exec -i postgres psql -U opencell opencell
cat <db-path>/pg_dump_keycloak.sql | docker exec -i postgres psql -U opencell keycloak
```
5. Prepare the filesystem from the backup:
```
sudo chown -R 1000 <filesystem-path>/opencelldata
```
6. Prepare the OC properties:
```
sudo chown -R 1000 <properties-path>/opencell-admin.properties
```
6. Make sure the <properties-path> and the <filesystem-path> are well placed in the `docker-compose.yml', listed in services / opencell / volumes
8. Up the OC container:
```
$ docker-compose up -d
```
\ No newline at end of file