Custom Extensions for Directus

For the Sabdivisen.com project I needed a headless CMS that would provide a great experience for the client and also for myself. Most importantly, I wanted a CMS that I could be confident I could extend easily if I found myself needing an unique feature. One such feature I knew I would need early in the project planning was a way to allow the client to upload their KML files and transparently convert them to GeoJSON for use by the frontend. This will avoid requiring the client to pre-process files and hopefully make the experience more seamless for them as they can continue to use the file formats they are familar with.

After reviewing several options I decided on using Directus. However, adding custom extensions to Directus has one hang-up, it requires either an Enterprise Cloud subscription or you must self-host. I opted to self-host Directus.

Self-Hosting Directus

To begin, I organized the project files as follows for reference. The backend directory contains code and data for Directus while the frontend directory contains Sabdivisen.com’s public Vue.js website.

.
├── backend
│   ├── extensions
│   └── uploads
├── frontend
│   └── ...
└── docker-compose.yml

The docker-compose.yml file below is a slightly modified version of the sample found in the documentation provided by Directus and highlighted below is the key line that mounts our local extensions directory for loading extensions into Directus.

version: '3'
services:
  database:
    image: postgis/postgis:13-master
    volumes:
      - ./data/database:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: 'directus'
      POSTGRES_PASSWORD: 'directus'
      POSTGRES_DB: 'directus'
 
  cache:
    image: redis:6
 
  directus:
    image: directus/directus:10.1.0
    ports:
      - 8055:8055
    volumes:
      - ./backend/uploads:/directus/uploads
      - ./backend/extensions:/directus/extensions
    depends_on:
      - cache
      - database
    environment:
      KEY: '255d861b-5ea1-5996-9aa3-922530ec40b1'
      SECRET: '6116487b-cda1-52c2-b5b5-c8022c45e263'
 
      DB_CLIENT: 'pg'
      DB_HOST: 'database'
      DB_PORT: '5432'
      DB_DATABASE: 'directus'
      DB_USER: 'directus'
      DB_PASSWORD: 'directus'
 
      CACHE_ENABLED: 'true'
      CACHE_STORE: 'redis'
      REDIS: 'redis://cache:6379'
 
      ADMIN_EMAIL: 'admin@example.com'
      ADMIN_PASSWORD: 'd1r3ctu5'
 
      # Make sure to set this in production
      # (see https://docs.directus.io/self-hosted/config-options#general)
      # PUBLIC_URL: 'https://directus.example.com'

The next time Directus is restarted the local extensions directory will be populated as seen below.

.
├── backend
│   ├── extensions
│   │   ├── displays
│   │   ├── endpoints
│   │   ├── hooks
│   │   ├── interfaces
│   │   ├── layouts
│   │   ├── modules
│   │   ├── operations
│   │   └── panels
│   └── uploads
├── frontend
│   └── ...
└── docker-compose.yml

Directus will scan these new dirctories for any Javascript modules and attempt to load them as an extension. Directus will see a file such as ./backend/extensions/endpoints/my-endpoint/index.js and attempt to load the my-endpoint custom API endpoint or ./backend/extensions/hooks/my-hook/index.js and attempt to load the my-hook custom API hook. Now we can write an extension.

Writing the Extension

The documentation for extensions is decent but I feel some examples are shallow or require you to dig around the docs to see the whole picture, so I’ll write the following out almost step-by-step.

Extension Types

There are many extesion types as you may have noticed when Directus populated the extensions directory. The 8 extension types can be combined to create even greater customizations. In the docs they describe this as an extension bundle and provide tools to help build them.

I chose to write an API Hook extension because it allows us to run custom logic whenver specific events occur. The hook would allow us to inspect a subdivision payload, read the KML file, do some processing to generate the GeoJSON data from the KML contents and finally commit the modified payload to the database.

I should mention that this could be done with a webhook to an external service as well. Although, as a webook the custom logic would be done asynchronously and we would not be able to interrupt the database transaction so that’s why I went with API hook extension instead.

Creating The Data Model

First, I created a Subdivisions collection in Directus which means behind the scenes Directus will create a table in the database for our subdivision data and Directus will provide us an API to interact with this table.

For this demostration we only need two fields in our table. A File type field named kml is created to store the uploaded KML file and a Map type field named polygon is created to store the GeoJSON data.

You can polish things further by making the polygon field hidden or read-only so that only a new KML file can overwrite the polygon value. Then, you can update the access controls so the API response will not contain the kml field and your frontend will only receive the prepared GeoJSON data in the polygon field.

Actually Writing the Extension

First let’s create a no-op extension. This hook will do nothing but prove that Directus is configured correctly and our extensions are loading successfully. Notice this is a hook extension that uses the filter event, there are other events you may consider for your usecase.

I wrote the API hook extension in the file ./backend/extensions/hooks/kml-converter/index.js so that Directus knows this code is a hook and it is called kml-converter.

import { defineHook } from "@directus/extensions-sdk";
 
export default defineHook(
  ({ filter }) => {
 
    filter("items.create", async (input, { collection }, context) => {
      return input;
    });
 
    filter("items.update", async (input, { collection }, context) => {
      return input;
    });
  }
)

When you restart the Directus container you should see the following log which shows our extension has successfully loaded.

directus | [04:58:00.180] INFO: Loaded extensions: kml-converter

However, this extension listens to every create and update event which is unnecessary for our use-case. We can add the collection name to our filter to limit our hook to only events that create or update the Subdivisions collection. Or alternatively, I could have used an if statement to check the value of collection is equal to Subdivisions.

import { defineHook } from "@directus/extensions-sdk";
 
export default defineHook(
  ({ filter }) => {
 
    filter("Subdivisions.items.create", async (input, { collection }, context) => {
      return input;
    });
 
    filter("Subdivisions.items.update", async (input, { collection }, context) => {
      return input;
    });
  }
)

Now the hook can act on the correct events, so I create a dummy function convertKmlToGeoJSON which is where I will actually do the real work of converting the KML file to GeoJSON soon. Also, take note that the update hook specifically checks if the kml property exists on input because if the KML file does not change during an update then it won’t be present on the input object and we can return as we don’t need to update the existing GeoJSON data.

import { defineHook } from "@directus/extensions-sdk";
 
export default defineHook(
  ({ filter }) => {
 
    const convertKmlToGeoJSON = async (kmlAssetId, context) => {
      return undefined;
    }
 
    filter("Subdivisions.items.create", async (input, { collection }, context) => {
      const polygon = await convertKmlToGeoJSON(input.kml, context);
      input.polygon = polygon;
      return input;
    });
 
    filter("Subdivisions.items.update", async (input, { collection }, context) => {
      if (!input.hasOwnProperty('kml')) {
        return input;
      }
      const polygon = await convertKmlToGeoJSON(input.kml, context);
      input.polygon = polygon;
      return input;
    });
  }
)

Adding Packages to the Environment

To convert the KML files easily I wanted to use the togeojson package that was built for converting KML to GeoJSON 🙄…duh. But this package is not available in Directus by default so it is not available to the extension either.

There is another way to add packages, see the Conclusion at end of this post for more information.

To install additional packages into the Directus environment I had to modify the docker-compose.yml file and include a new Dockerfile in the backend directory. Instead of using the official Directus image I am building a new image ontop of the official image.

services:
  directus:
    # image: directus/directus:10.1.0
    container_name: directus
    build:
      context: backend
      dockerfile: Dockerfile
  ...

I go through the hassle of installing pnpm because I already use it for the frontend codebase and wanted consistency… 🤷

FROM directus/directus:10.1.0
 
USER root
RUN corepack enable && corepack prepare pnpm@8.3.1 --activate
 
USER node
RUN pnpm install togeojson xmldom stream-to-string

Later I had to install a couple other packages to handle the KML file itself as you will see.

The Final Extension with Error handling

Now, I can put the whole thing together. Only 3 lines are actually converting the KML file (thanks togeojson). The remaining lines are handling the KML file itself and handling errors.

File type fields in Directus are a relationship to the Assets table so the field only contains the asset ID. I had to use the AssetsService to obtain the file stream so that I could access its contents.

The great majority of lines are error handling. I found when the extension fails the user will only receive a generic internal server type error which certainly doesn’t help them. Instead I raise the InvalidPayloadException where ever I can to provide useful information to the user, especialy since many errors were due to bad KML files and with useful error messages the client could then fix it themselves.

I renamed the function here convertKmlToPolygon to match its use as I only want the coordinates field from the GeoJSON object.

import { defineHook } from "@directus/extensions-sdk";
import { DOMParser } from "xmldom";
import toGeoJSON from "togeojson";
import streamToString from "stream-to-string";
 
export default defineHook(
  ({ filter }, { exceptions, services }) => {
    const { InvalidPayloadException } = exceptions;
    const { AssetsService } = services;
 
    const convertKmlToPolygon = async (kmlAssetId, context) => {
      // get the kml file
      const assets = new AssetsService(context);
      const { stream } = await assets.getAsset(kmlAssetId, {});
      
      // convert kml to geojson
      const doc = await streamToString(stream);
      const kml = new DOMParser().parseFromString(doc)
      const gj = toGeoJSON.kml(kml);
 
      // sanity check
      if (!gj.hasOwnProperty('features')) {
        throw new InvalidPayloadException('KML file does not have features.');
      }
 
      // grab geometry data from geojson and remove Z value from coordinates
      let geometry = gj.features?.[0]?.geometry
      if (geometry === undefined) {
        throw new InvalidPayloadException('KML file does not have geometry.')
      }
      const coords = geometry.coordinates[0].map((coord) => [coord[0], coord[1]])
      geometry.coordinates = [coords];
      return geometry;
    }
 
    filter("Subdivisions.items.create", async (input, { collection }, context) => {
      try {
        const polygon = await convertKmlToPolygon(input.kml, context);
        input.polygon = polygon;
        return input;
      } catch (error) {
        console.error(error);
        throw new InvalidPayloadException(`Failed to convert KML file. ${error}`);
      }
    });
 
    filter("Subdivisions.items.update", async (input, { collection }, context) => {
      // return if `kml` column is not updated
      if (!input.hasOwnProperty('kml')) {
        return input;
      }
      try {
        const polygon = await convertKmlToPolygon(input.kml, context);
        input.polygon = polygon;
        return input;
      } catch (error) {
        console.error(error);
        throw new InvalidPayloadException(`Failed to convert KML file. ${error}`);
      }
    });
  }
)

Conclusion

The way I added this extension to Directus is one way to do so but probably best for small extensions. The other way is with the create-directus-extension utility which you can read more about here. The advantage being you can keep dependencies of your extensions seperate from Directus and other extensions and publish your extension as a package for others to use.