Back to home

Processing Local YAML Files With Sidekiq main image

Processing Local YAML Files With Sidekiq

This post will demonstrate how to have a Sidekiq job read a local YAML file and then take that information and add the information to a database.

Source code can be found here

Prerequisites

  1. Read my previous blog post "Getting Started With Sidekiq In Rails 7".

Getting started

It is a prerequisite that you get the repo up and running by following the steps outlined in the post "Getting Started With Sidekiq In Rails 7".

After you have a running Rails setup with Sidekiq, we need to add a last few things.

# Create a folder and file to house our YAML templates $ mkdir -p templates/yaml $ touch templates/yaml/basic.yml # Create a Person model $ rails g model Person name:string age:integer # Create a Jobs controller $ bin/rails g controller jobs create # Create a Template job $ bin/rails g sidekiq:job template # Run migrations $ bin/rails db:migrate

At this stage, we are ready to start updating our controllers and jobs.

Updating our jobs controller

In the file app/controllers/jobs_controller.rb:

class JobsController < ApplicationController def create if params[:template_group].present? && params[:template_name].present? TemplateJob.perform_async(params[:template_group], params[:template_name]) render json: { message: 'Accepted' }, status: :accepted else render json: { error: 'Missing template_group or template_name' }, status: :bad_request end end end

Here, we effectively check that we have the required params and if so, we run the job.

Note: A better job could be done here to ensure the folder exists at this stage, but we'll assume it does for this post.

Next, we need to write up our job.

Updating our template job

In the file app/sidekiq/template_job.rb:

require 'yaml' class TemplateJob include Sidekiq::Job def perform(template_group, template_name) # Do something p "TemplateJob started with args #{template_group} #{template_name}" template = YAML.safe_load(File.read(File.join(Rails.root, 'templates', template_group, "#{template_name}.yaml"))) p "Template name: #{template['name']}" p "Template description: #{template['description']}" template['people'].each do |person| person = Person.new(name: person['name'], age: person['age']).save p "Person #{person['name']} could not be saved" if person.nil? end # Will display current time, milliseconds included p "TemplateJob #{Time.now.strftime('%F - %H:%M:%S.%L')}" end end

Here, we are using the yaml module as a helper for us to safely load the YAML file.

The template_group and template_name variables are passed into the job and used to denote the name of the YAML file.

Once loaded, we assume there is a people property which is iterated over and each person is added to the database.

Before we test our job, we need to make some updates to our configuration.

Adding in the YAML content

In templates/yaml/basic.yaml, populate the YAML with the following:

name: Basic description: Basic YAML template to insert people people: - name: John Doe age: 30 - name: Jane Doe age: 25 - name: Joe Smith age: 32

We will be using the people key to iterate over the people and add them to the database.

Setting our routes

In config/routes.rb:

Rails.application.routes.draw do resources :jobs, only: [:create] end

Updating our application config

In config/application.rb:

require_relative "boot" require "rails/all" # Require the gems listed in Gemfile, including any gems # you've limited to :test, :development, or :production. Bundler.require(*Rails.groups) module DemoRailsYaml class Application < Rails::Application # Initialize configuration defaults for originally generated Rails version. config.load_defaults 6.1 config.action_controller.default_protect_from_forgery = false if ENV['RAILS_ENV'] == 'development' end end

We turn default_protect_from_forgery for this current example where I am using the HTTPie CLI to test out the job.

Testing our job

If you followed the configuration from the prerequisite correctly, then you can run bin/dev to start both the Rails server and the Sidekiq server.

From here, I can use HTTPie to test out our job:

$ http POST localhost:5000/jobs template_group=yaml template_name=basic HTTP/1.1 202 Accepted Cache-Control: no-store, must-revalidate, private, max-age=0 Content-Type: application/json; charset=utf-8 Referrer-Policy: strict-origin-when-cross-origin Set-Cookie: __profilin=p%3Dt; path=/; HttpOnly; SameSite=Lax Transfer-Encoding: chunked X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: SAMEORIGIN X-MiniProfiler-Ids: w5qbp38e6xytn2a1g5kt,d6mr3lsmk8e4i8i4o6ad,ve2sy6fpwhycw1bhi8m,pk199otp1pemditsz172,zij0sg8ruw8c350s4zrd,2fu1c028w4h8j1fg8eqn,sql5jajpv0iu7zoe9z2n,lqtxoxkso0p3s9udy24i,p74qirfir3xmhhiyo31,l2h57dj2hn4g,tokv1ogolv1vcj0fxnmd,szathvf0vaxo6glg9ruw,61hri8h3fo74jhwsolez,8mt3s6max4sdgylleub4,g43fqs7faec3hts96g7e,iwfy35nvml9fbsaqzqcp,qk6mrujh9gjdg291hfjg X-MiniProfiler-Original-Cache-Control: no-cache X-Permitted-Cross-Domain-Policies: none X-Request-Id: f411f617-48eb-45e4-87d8-b48976cd121d X-Runtime: 0.045609 X-XSS-Protection: 1; mode=block { "message": "Accepted" }

Once the job is accepted, it will be queued and run by Sidekiq.

After the job completes, we can check the results in the Rails console:

$ rails c irb(main):001:0> Person.all (1.1ms) SELECT sqlite_version(*) Person Load (0.2ms) SELECT "people".* FROM "people" => [#<Person:0x00007f7c89e220b8 id: 1, name: "John Doe", age: 30, created_at: Mon, 04 Apr 2022 10:33:34.031843000 UTC +00:00, updated_at: Mon, 04 Apr 2022 10:33:34.031843000 UTC +00:00>, #<Person:0x00007f7c89e51b10 id: 2, name: "Jane Doe", age: 25, created_at: Mon, 04 Apr 2022 10:33:34.036244000 UTC +00:00, updated_at: Mon, 04 Apr 2022 10:33:34.036244000 UTC +00:00>, #<Person:0x00007f7c89e51a48 id: 3, name: "Joe Smith", age: 32, created_at: Mon, 04 Apr 2022 10:33:34.039872000 UTC +00:00, updated_at: Mon, 04 Apr 2022 10:33:34.039872000 UTC +00:00>]

Success!

Summary

Today's post demonstrated how to read a local YAML file from a Sidekiq job and use the data from the file to populate a particular model in the database with some entries.

Resources and further reading

Photo credit: pawel_czerwinski

Personal image

Dennis O'Keeffe

@dennisokeeffe92
  • Melbourne, Australia

Hi, I am a professional Software Engineer. Formerly of Culture Amp, UsabilityHub, Present Company and NightGuru.
I am currently working on workingoutloud.dev, Den Dribbles and LandPad .

1,200+ PEOPLE ALREADY JOINED ❤️️

Get fresh posts + news direct to your inbox.

No spam. We only send you relevant content.