Security

Disaster recovery & BI

A word about Sveltekit adapters.

21RISK is built with Sveltekit, a modern application framework that makes a clear abstraction so developers don’t have to think about the target deployment. This makes it very flexible to change where the application code is deployed. As of june 2023, Sveltekit supports the following deployment targets:

AdapterTarget
@sveltejs/adapter-cloudflareCloudflare Pages
@sveltejs/adapter-cloudflare -workersCloudflare Workers
@sveltejs/adpater-netlifyNetlify
@sveltejs/adapter-nodeNode servers
@sveltejs/adapter-staticStatic site generation
@sveltejs/adapter-vercelVercel (The adapter 21RISK is currently using, since may 2023.)

Failover strategy

To avoid any downtime we utilise a comprehensive failover strategy on all avenues of our application. This helps ensure that any regional or local incidents do not impact performance and result in downtime.

  • Vercel uses AWS Global Accelerator and Anycast network to automatically reroute traffic to another region in case of regional failure.
  • Edge functions and Edge Middleware switch to another region automatically using Cloudflare's automatic failover feature
  • Postgres cloud failover - ensures that data is replicated to multiple regions.
  • Postgres cloud provider backups - We do periodic backups to multiple different cloud providers to ensure that even in the event of a total loss of the cloud provider AWS we still have ensured our data.

Disaster Recovery Plan

A Disaster Recovery Plan (DRP) is essential for businesses of all sizes, including your small development team. The purpose of a DRP is to ensure that your team can respond to a disaster or other emergency that affects information systems and minimize the effect on the operation of the business.

Risk Assessment

An ordered list of the most critical incidents to affect our system: This list provides the foundation for the steps we take in order to minimize the impact if they should occur.

  • Regional at Planetscale (postgres) -> Failover strategy
  • Complete Outage at Planetscale (postgres) -> Move to alternate processing site
  • Outage at Doppler -> Move to alternate processing site
  • Outage at SSOready -> Move to alternate processing site
  • Regional outage at Vercel -> Failover strategy
  • Complete outage at Vercel -> Move to alternate processing site
  • Outage at AWS -> Move to alternate processing site
  • Outage at Cloudflare -> Move to alternate processing site
  • Outage at Github -> Move to alternate processing site

Identify Critical Assets

Identify which data, hardware, and software are critical to business operations. Prioritize their recovery in the event of a disaster.

  • Planetscale (Postgres)
  • SSOready
  • Cloudflare (DNS, files)
  • Google Cloud (file backups)
  • Github
  • pnpm
  • Doppler
  • AWS

Backup and Restore Procedures

ServiceBackup policyRecovery Point Objective
PlanetscaleWe have an extensive backup policy, described in this document. It’s cross-region. The backups are full, nothing is left out.full
Cloudflare R2 (customer files)Twice a day, new files that have passed their virus scan are copied to a Google Cloud Storage bucket in the EU, outside Cloudflare.full, usually within 12 hours; a file whose virus scan is delayed is copied on the run after its scan
Cloudflare R2 (public files)Public files are stored only in Cloudflare R2, which is designed for 99.999999999% annual durability; they have no separate backup yet.partial
GithubAll changes to the code is in version control, running on Github.comfull

Disaster Response

Step 1 Better stack will automatically contact Alex and Andreas, when downtime in the system is identified.

Step 2 Depending on the severity of the plan, a physical meetup is arranged with the priority to protect/verify all backups are in place.

Recovery Procedures

Document the steps necessary to recover each of your critical assets. This should include steps to restore data from backups, how to ensure that restored systems are secure, and how to validate that systems are functioning correctly after recovery.

ServiceSteps to validateSteps to recover
PlanetscaleNavigate to planetscale.com, login and verify backups are there.Create a new target cluster, and restore the backup.
Cloudflare R2 (customer files)Log into Google Cloud, open the backup bucket in Cloud Storage and check that recent files are there.Copy the files to the alternate provider and point assets.21risk.com at it.
Cloudflare R2 (public files)Log into Cloudflare, open R2 and check the public files bucket.Copy the files to the alternate provider and point p-assets.21risk.com at it.
SourceCodeConnect to the 21RISK/eddystone repository and download the codebase. (Alternatively use Github codespaces, where the code is also present).

Alternate Processing Sites: For critical systems

In the event of a disaster at one of our current service providers, we would migrate the following services to the alternative providers to ensure functionality. 28 Please note that the alternate providers are not actively used currently.

ServiceCurrentAlernative
DatabasePlanetscaleSelf-hosted on Digital Ocean / Heroku
HostingVercelHeroku
File storageCloudflare R2AWS S3
AuthenticationSSOReadyAuth0
Version controlGithubGitlab
Secrets managementDoppler1Password