Processing Local YAML Files With Sidekiq

This post will demonstrate how to have a Sidekiq job read a local YAML file and then take that information and add the information to a database.

Source code can be found here

Prerequisites

Read my previous blog post "Getting Started With Sidekiq In Rails 7".

Getting started

It is a prerequisite that you get the repo up and running by following the steps outlined in the post "Getting Started With Sidekiq In Rails 7".

After you have a running Rails setup with Sidekiq, we need to add a last few things.

# Create a folder and file to house our YAML templates
$ mkdir -p templates/yaml
$ touch templates/yaml/basic.yml

# Create a Person model
$ rails g model Person name:string age:integer

# Create a Jobs controller
$ bin/rails g controller jobs create

# Create a Template job
$ bin/rails g sidekiq:job template

# Run migrations
$ bin/rails db:migrate

At this stage, we are ready to start updating our controllers and jobs.

Updating our jobs controller

In the file app/controllers/jobs_controller.rb:

class JobsController < ApplicationController
  def create
    if params[:template_group].present? && params[:template_name].present?
      TemplateJob.perform_async(params[:template_group], params[:template_name])
      render json: { message: 'Accepted' }, status: :accepted
    else
      render json: { error: 'Missing template_group or template_name' }, status: :bad_request
    end
  end
end

Here, we effectively check that we have the required params and if so, we run the job.

Note: A better job could be done here to ensure the folder exists at this stage, but we'll assume it does for this post.

Next, we need to write up our job.

Updating our template job

In the file app/sidekiq/template_job.rb:

require 'yaml'

class TemplateJob
  include Sidekiq::Job

  def perform(template_group, template_name)
    # Do something
    p "TemplateJob started with args #{template_group} #{template_name}"

    template = YAML.safe_load(File.read(File.join(Rails.root, 'templates', template_group, "#{template_name}.yaml")))

    p "Template name: #{template['name']}"
    p "Template description: #{template['description']}"

    template['people'].each do |person|
      person = Person.new(name: person['name'], age: person['age']).save
      p "Person #{person['name']} could not be saved" if person.nil?
    end

    # Will display current time, milliseconds included
    p "TemplateJob #{Time.now.strftime('%F - %H:%M:%S.%L')}"
  end
end

Here, we are using the yaml module as a helper for us to safely load the YAML file.

The template_group and template_name variables are passed into the job and used to denote the name of the YAML file.

Once loaded, we assume there is a people property which is iterated over and each person is added to the database.

Before we test our job, we need to make some updates to our configuration.

Adding in the YAML content

In templates/yaml/basic.yaml, populate the YAML with the following:

name: Basic
description: Basic YAML template to insert people
people:
  - name: John Doe
    age: 30
  - name: Jane Doe
    age: 25
  - name: Joe Smith
    age: 32

We will be using the people key to iterate over the people and add them to the database.

Setting our routes

In config/routes.rb:

Rails.application.routes.draw do
  resources :jobs, only: [:create]
end

Updating our application config

In config/application.rb:

require_relative "boot"

require "rails/all"

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

module DemoRailsYaml
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 6.1

    config.action_controller.default_protect_from_forgery = false if ENV['RAILS_ENV'] == 'development'
  end
end

We turn default_protect_from_forgery for this current example where I am using the HTTPie CLI to test out the job.

Testing our job

If you followed the configuration from the prerequisite correctly, then you can run bin/dev to start both the Rails server and the Sidekiq server.

From here, I can use HTTPie to test out our job:

$ http POST localhost:5000/jobs template_group=yaml template_name=basic
HTTP/1.1 202 Accepted
Cache-Control: no-store, must-revalidate, private, max-age=0
Content-Type: application/json; charset=utf-8
Referrer-Policy: strict-origin-when-cross-origin
Set-Cookie: __profilin=p%3Dt; path=/; HttpOnly; SameSite=Lax
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-MiniProfiler-Ids: w5qbp38e6xytn2a1g5kt,d6mr3lsmk8e4i8i4o6ad,ve2sy6fpwhycw1bhi8m,pk199otp1pemditsz172,zij0sg8ruw8c350s4zrd,2fu1c028w4h8j1fg8eqn,sql5jajpv0iu7zoe9z2n,lqtxoxkso0p3s9udy24i,p74qirfir3xmhhiyo31,l2h57dj2hn4g,tokv1ogolv1vcj0fxnmd,szathvf0vaxo6glg9ruw,61hri8h3fo74jhwsolez,8mt3s6max4sdgylleub4,g43fqs7faec3hts96g7e,iwfy35nvml9fbsaqzqcp,qk6mrujh9gjdg291hfjg
X-MiniProfiler-Original-Cache-Control: no-cache
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: f411f617-48eb-45e4-87d8-b48976cd121d
X-Runtime: 0.045609
X-XSS-Protection: 1; mode=block

{
    "message": "Accepted"
}

Once the job is accepted, it will be queued and run by Sidekiq.

After the job completes, we can check the results in the Rails console:

$ rails c
irb(main):001:0> Person.all
   (1.1ms)  SELECT sqlite_version(*)
  Person Load (0.2ms)  SELECT "people".* FROM "people"
=>
[#<Person:0x00007f7c89e220b8
  id: 1,
  name: "John Doe",
  age: 30,
  created_at: Mon, 04 Apr 2022 10:33:34.031843000 UTC +00:00,
  updated_at: Mon, 04 Apr 2022 10:33:34.031843000 UTC +00:00>,
 #<Person:0x00007f7c89e51b10
  id: 2,
  name: "Jane Doe",
  age: 25,
  created_at: Mon, 04 Apr 2022 10:33:34.036244000 UTC +00:00,
  updated_at: Mon, 04 Apr 2022 10:33:34.036244000 UTC +00:00>,
 #<Person:0x00007f7c89e51a48
  id: 3,
  name: "Joe Smith",
  age: 32,
  created_at: Mon, 04 Apr 2022 10:33:34.039872000 UTC +00:00,
  updated_at: Mon, 04 Apr 2022 10:33:34.039872000 UTC +00:00>]

Success!

Summary

Today's post demonstrated how to read a local YAML file from a Sidekiq job and use the data from the file to populate a particular model in the database with some entries.