On-demand backups with the Compose API and Node.js

Published

You need your latest backup and you need it automatically? We can show you how to create an on-demand backup and retrieve it automatically using the Compose API and Node.js

We recently showed you how to use the API and Node.js to retrieve the automated backups made by Compose deployments every day. The API is pretty straightforward for those cases - find a backup record, download the associated file from the backup record's link. To create and retrieve an on-demand backup is a little more intricate as we'll have to make the backup program wait. Ride along with the code, and we'll show you why.

First up, we need to create our latest backup. For that the 2016-07-post-deployments-backups endpoint comes into play. POSTing at this endpoint with a deployment id in the path will start an on-demand backup:

let startBackup = (deploymentid, options) => {  
    fetch(`${apibase}/deployments/${deploymentid}/backups`, { method: "POST", headers: apiheaders })
        .then(function (res) {
            return res.json();
        })

But this endpoint doesn't wait till it can return a backup id or backup record. It returns immediately with a Compose API recipe. Recipes are how we track the progress of tasks in Compose. When there's work to be done a recipe is found to do that task and it is run. Each recipe has an id and we can query another endpoint with that recipe id to see how it has progressed. That progress is represented by the status field of the recipe.

As an aside, recipes are forever on Compose, once run, they form a part of the history of a deployment. You can even query a deployment for all the recipes ever run against it, including the ones use to create it initially. That way, we know what happened to any deployment. The more you know.

Back to the task at hand, for this we'll print out some of the recipe information first, then call up a new function, pollForRecipeComplete() giving it the recipe id to work with.

        .then(function (recipe) {
            console.log(`Recipe Id: ${recipe.id}
Status:        ${recipe.status}  
`);
            pollForRecipeComplete(recipe.id)
        })
        .catch(function (err) {
            console.log(err);
        });
}

So, the key to checking a recipe's status is the 2016-07-get-recipe endpoint. By using GET on this endpoint with the recipe id, we can get a current copy of the recipe.

let pollForRecipeComplete = (recipeid) => {  
    fetch(`${apibase}/recipes/${recipeid}`, { headers: apiheaders })
        .then(function (res) { return res.json() })

We now have the current recipe JSON and we can check for whether the recipe has run to completion. At that point the on-demand backup will be ready. We'll know that if the status becomes "complete":

        .then(function (recipe) {
            if (recipe.status == "complete") {
                process.stdout.write("\n");
                return getLatestOndemand(recipe.deployment_id);
                }
            else {
                process.stdout.write('.');
                setTimeout(pollForRecipeComplete, 10000, recipeid);
            }
        })
};

If it is complete, we return the result of calling getLatestOndemand() with the deployment_id the recipe was working on. We'll come back to that in a moment. The else side of the if simply writes out a dot and then sets up the pollForRecipeComplete function to be called again in ten seconds.

Ok, it's time to get the latest on-demand backup in getLatestOndemand(). The recipe, in its current incarnation, can't tell us about the backup it made, so we have to download the on-demand backup list so we can search through that for its work:

let getLatestOndemand = (deploymentid) => {  
    fetch(`${apibase}/deployments/${deploymentid}/backups`, { headers: apiheaders })
        .then(function (res) {
            return res.json();
        })
        .then(function (json) {
            let backups = json["_embedded"].backups;

Getting the list of backups is essentially the same as the listBackups() function in the previous article, using the 2016-07-get-deployment-backups endpoint to retrieve the JSON list. The difference is that we aren't going to list them. Instead, we are going to search for a backup of type "on_demand". There's only one on-demand backup associated with a deployment at any time so this should be the backup we're looking for:

            for (let backup of backups) {
                if(backup.type=="on_demand") {
                    getBackup(backup.deployment_id,backup.id)
                    return true;
                }
            }
            console.log("No on demand backup found")
            return false;
        })
        .catch(function (err) {
            console.log(err);
        });
}

Once we find the backup id, we call the getBackup() function we created in the previous article, giving it the deployment id and backup id. And we're done...

Well, nearly. To really finish this we need to add a line to the yargs command line definition:

yargs.version("0.0.1")  
    ...
    .command("start <deploymentid>", "Start on demand backup", {}, (argv) => startBackup(argv.deploymentid))
    ...
    .help()
    .argv;

Now we can run our command:

$ backupomatplus.js start 55f694344d847d005d000009
Recipe Id: 58a1bf06e1baf600140002cf  
Status:        running

..
Going to download rethinkery_2017-02-13_14-13-37_utc_on_demand  
Done  
$

You'll find the code for this in backupomatplus.js in the backupmat repository on Github. Of course, the Compose API is there for you to make the integrations that best suit your application. We'll be taking a wider look at the API in a future article, covering all the API's functionality.


If you have any feedback about this or any other Compose article, drop the Compose Articles team a line at articles@compose.com. We're happy to hear from you.

attribution Pixabay

Dj Walker-Morgan
Dj Walker-Morgan is Compose's resident Content Curator, and has been both a developer and writer since Apples came in II flavors and Commodores had Pets. Love this article? Head over to Dj Walker-Morgan’s author page and keep reading.