Migrations in DynamoDB with NodeJS

main

When you work with frameworks such as ASP.NET or Django, ORM (Entity Framework, Django ORM) magically creates migration, and with DynamoDB, it may seem strange that you need to write migrations by yourself. But it is the price of using NoSQL and rejecting ORMs, but don't worry, managing migrations by yourself isn't that scary.

Current State

Before we jump into migrations, I want to show you how we know the current state of tables.

const TABLE_NAME_POSTFIX = `_${process.env.ENV}`
const TABLES_NAMES = {
users: `users_${process.env.ENV}`,
projects: `projects_${process.env.ENV}`,
// we will use it know the index of last migration
projects: `management_${process.env.ENV}`,
// other tables names ...
}
const TABLES_PARAMS = [
tableParams(TABLES_NAMES.users, 'id'),
tableParams(TABLES_NAMES.projects, 'id'),
tableParams(TABLES_NAMES.management, 'id'),
// other tables params ...
]
view raw tableStates.js hosted with ❤ by GitHub

The function that turns the table name and primary key into parameters for DynamoDB comes from this tiny library with common utils.

Create and Delete Tables

First, we want to delete tables we don't need anymore and create new ones. To do this, we compare the existing DynamoDB tables' names with our tables parameters. If there is a mismatch, we execute appropriate operations to delete and create tables accordingly to the current state.

const database = new AWS.DynamoDB()
const neededTables = TABLES_PARAMS.map(({ TableName }) => TableName)
const existingTables = await database
.listTables()
.promise()
.then(data => data.TableNames)
const dbsToDelete = _.without(existingTables, ...neededTables)
const dbsToCreate = _.without(neededTables, ...existingTables)
await Promise.all(
dbsToDelete.map(TableName =>
database.deleteTable({ TableName }).promise()
)
)
await Promise.all(
dbsToCreate.map(TableName =>
database
.createTable(TABLES_PARAMS.find(t => t.TableName === TableName))
.promise()
)
)
view raw prepareDB1.js hosted with ❤ by GitHub

Running Migrations

To know which migration was executed last time, we have the table with migration information. It looks like this:

migration item
migration item

All our migrations are simply an array of asynchronous functions that perform operations on tables items.

const aws = require('aws-sdk')
module.exports = [
async () => {...},
async () => {...},
async () => {...},
async () => {
const documentClient = new aws.DynamoDB.DocumentClient()
const projects = await documentClient
.scan({
TableName: 'Projects',
ProjectionExpression: 'id',
FilterExpression: 'attribute_not_exists(creationTime)'
})
.promise()
.then(({ Items }) => Items)
return Promise.all(
projects.map(({ id }) =>
documentClient
.update({
TableName: 'Projects',
Key: { id },
UpdateExpression: 'set creationTime = :creationTime',
ExpressionAttributeValues: {
':creationTime': Date.now()
}
})
.promise()
)
)
},
async () => {...},
async () => {...},
]
view raw migrations.js hosted with ❤ by GitHub

To run these migrations, we take the index of the last executed one and run all migrations, starting from the index. If there was no migration item in the management table, it means that tables are empty, and there is no need to run migrations.

const documentClient = new AWS.DynamoDB.DocumentClient()
const migration = await documentClient
.get({
TableName: 'Management',
Key: { id: 'migration' }
})
.promise()
.then(({ Item }) => Item)
if (
migration.lastMigration !== undefined &&
migration.lastMigration < migrations.length
) {
const migrationsToRun = migrations.slice(migration.lastMigration + 1)
await Promise.all(migrationsToRun.map(migration => migration()))
}
await documentClient
.put({
TableName: 'Management',
Item: {
id: 'migration',
lastMigration: migrations.length - 1
}
})
.promise()
view raw prepareDB2.js hosted with ❤ by GitHub

Finally, we will have a script for database preparation. We can run it automatically in our CI/CD pipeline before deploying the new version of the back-end.

require('dotenv').config()
require('../src/utils/aws').setupAWS()
const _ = require('lodash')
const AWS = require('aws-sdk')
const { TABLES_PARAMS } = require('../src/constants/db')
const migrations = require('../src/migrations')
const prepareDB = async () => {
const database = new AWS.DynamoDB()
const neededTables = TABLES_PARAMS.map(({ TableName }) => TableName)
const existingTables = await database
.listTables()
.promise()
.then(data => data.TableNames)
const dbsToDelete = _.without(existingTables, ...neededTables)
const dbsToCreate = _.without(neededTables, ...existingTables)
await Promise.all(
dbsToDelete.map(TableName =>
database.deleteTable({ TableName }).promise()
)
)
await Promise.all(
dbsToCreate.map(TableName =>
database
.createTable(TABLES_PARAMS.find(t => t.TableName === TableName))
.promise()
)
)
const documentClient = new AWS.DynamoDB.DocumentClient()
const migration = await documentClient
.get({
TableName: 'Management',
Key: { id: 'migration' }
})
.promise()
.then(({ Item }) => Item)
if (
migration.lastMigration !== undefined &&
migration.lastMigration < migrations.length
) {
const migrationsToRun = migrations.slice(migration.lastMigration + 1)
await Promise.all(migrationsToRun.map(migration => migration()))
}
await documentClient
.put({
TableName: 'Management',
Item: {
id: 'migration',
lastMigration: migrations.length - 1
}
})
.promise()
}
prepareDB()
view raw prepareDB.js hosted with ❤ by GitHub

It is a simple implementation of migrations mechanism :) If you would like to know about any other topic related to this, please let me know!