Re-installing Matrix-Synapse server and migrating data
I was running a Matrix-Synapse server on Ubuntu 18.04 LTS and recently noticed that it did not receive any updates anymore. It would not have come as a surprise to me if I had read the release notes. ;-) Unfortunately I could not simply upgrade to Ubuntu 20.04 LTS but had to reinstall the operating system - thus losing all data. Of course I’m running backups but I never actually tried to recover from them… It turned out to be quite easy.
I’m trying to cover all the stuff you’ll have to backup and recover if you ever have to re-install Matrix-Synapse server and migrate the data to the new installation. Of course depending on your setup details may differ but hopefully basically it would be the same.
Things to backup
You might want to backup some basic OS configuration, like everything you customize once it’s installed, i.e. sshd configuration/keys, custom scripts, firewall configuration, etc.).
Synapse usually (depending on your setup) requires a couple of additional services:
- webserver (i.e. nginx)
- database (Postgresql)
- TURN server (Coturn)
nginx
My setup is using nginx as a reverse proxy. It’s a good idea to take backups of at least /etc/nginx/sites-available/<YOUR_CONFIG> and /etc/nginx/nginx.conf. You might also want to backup your certificates (or just generate new ones later, i.e. by Let’s Encrypt).
Postgresql
Depending on your version you should probably take a backup of /etc/postgresql/<VERSION>/main. And one VERY important thing: your database (replace DB_NAME accordingly):
1
$ su - postgresql -c "pg_dump DB_NAME > backup.dump"
Coturn
The configuration file to backup would be /etc/turnserver.conf. I created a custom logrotate configuration for Coturn’s logs so I backed that up as well.
Matrix-Synapse
Your main configuration file of Matrix-Synapse probably is /etc/matrix-synapase/homeserver.yml, but it may be a good idea to backup the directory completely. You should also create a backup of all files uploaded in chats, profile pictures etc. Check media_store_path: in your configuration file and backup that folder.
If you backed up all of the above you should at least not suffer any data loss. In worst case you’d have to configure some settings again for stuff you forgot to backup.
Restore
Once your (new) server is ready (again) you can start to customize and install everyting. First do all the OS stuff, like sshd configuration, fail2ban, hardening, firewall, add user(s), what ever…
Then install the required software again. In case of Ubuntu, once you’ve added the Matrix-Synapse repository, run:
1
$ apt install postgresql nginx coturn matrix-synapse-py3 snapd fuse
snapd and fuse are optional in case you’d like to install certbot to manage Let’s Encrypt certificates.
Postgresql
When upgrading from Ubuntu 18.04 to 20.04 the Postgresql version will increase from 10 to 12. So I did not just copy the configuration files from the backup to /etc/postgresql/12/main, but I compared the old and new main configuration file postgresql.conf and altered the new file where appropriate. The same is true for pg_hba.conf. You should do both before actually re-creating the database user and the database. Depending on your password encryption settings Matrix-Synapse server will otherwise not be able to login to the database. (You’ll have to reset the database password then - took a while until I remembered the issue). Anyway, once configuration files are OK, restart Postgresql and create your Synapse database and user again. You’ll find everything in your homeserver.yml, like database name, database user name and password. This should leave you with an “empty” database. To restore the contents of your old database you will need the backup.dump file created before:
1
$ su - postgresql -c "psql -d <DATABASE_NAME> -f /path/to/backup.dump"
nginx
Before adjusting the configuration of nginx I first generated some new certificates by invoking certbot. If you also use certificates with Coturn don’t forget to generate them as well. Then create the reverse proxy configuration file for Synapse, save it in /etc/nginx/sites-available/ and create a symlink in /etc/nginx/sites-enabled. Remove configuration files not required. Don’t restart nginx yet.
Coturn
Also Coturn came as new version. I again compared old and new configuration file but you can probably just copy the old one back. Then restart Coturn.
Matrix-Synapse
Probably you can just copy your old configuration file over the new one and you should be fine. Remember to restore the files of media_store_path:. Restart Matrix-Synapse then. If it fails check /var/log/matrix-synapse/homeserver.log. If all is fine restart nginx now as well.
Your Matrix-Synapse server should be available again with all chats and files at the same place where you left them!
HINT: you should always check the status of required services after restarting them (systemctl satus ...). You might encounter warnings because the old configuration file is maybe using deprecated settings or stuff like that.