While AWS does a good job of running these jobs, we've had a really hard time running cron jobs, data backfill tasks, and one-off scripts on AWS. That's where Airplane comes in.
Airplane is awesome because it gives you nice infrastructure around running your own code.
We give Airplane our main backend Docker image and then every Airplane task calls that one Dockerfile with a different command. This is nice since I didn't have to learn anything special to use Airplane, I can use what we are already deploying to AWS.
This did take some time to set up the first time since I was a very early Airplane customer. But it sounds like the onboarding has gotten a lot easier.
Below is what our Airplane dashboard looks like:
I've redacted the task names, but you can think about them in a few different categories:
Before Airplane, if we wanted to delete a user or add a flag, we would have to SSH into one of our production machines. This was really annoying in that it was error prone, we didn't have auditing, and it required an engineer with access to production.
Airplane lets you run scripts from their dashboard. And the logging if f'ing incredible. You can always click into an old script and see who ran it, what the parameters were, and look at the logs. You even get an email / Slack if the task fails!
I really really love the built in logging. Managing scheduled tasks and one off scripts on AWS is a nightmare and I was planning on building a whole system around this until I found Airplane!
Here is what one of the task detail pages looks like. And if I click into any run I can see exactly what happened. It's incredible.
Tip for Airplane — It would be sweet if I could define a result of the the task and then see it in this list.
For Glow, we use this to periodically backfill data that changes on a daily basis. For Luma, we use this to send guest email reminders for events.
Airplane has a pretty nice scheduler that lets you define the task either in Cron format or a more humane way.
For Glow, we are pulling in and processing a ton of data. While new data gets added by a service on AWS, Airplane gives us an incredible interface for running tasks to populate old data.
Again this benefits from the increased logging I mentioned above. If we are backfilling in a few months of data, I can trigger a month at a time since Airplane makes it easy for me to see the status of previous jobs.
Thank you to the Airplane team!