Executing 432 million scheduled background jobs per day

We just created a performance test of the Magic Scheduler, and although it’s using a version of Magic that is not yet released, the numbers were impressive to say the least.

We were able to use Hyperlambda to create and execute 4.000 scheduled background tasks, and wrap everything up in roughly 2 seconds, on a 5 year old MacBook Air.

Afterwards we only created the jobs, as fire and forget jobs, for then to execute them, and wait for all jobs to finish. Our results gave us a throughput number of roughly 5.000 jobs per second.

Of course, what your job actually does is important, and our jobs were pretty empty to be honest with you, simply performing an interlocked increment of a shared integer value – But this tells us something about the throughput of the Magic Scheduler, which obviously is pretty amazing compared to other similar tools. Watch how we tested below.

Throughput of 432 million scheduled jobs per day

You can find the code below. First the C# code.

using System.Threading;
using magic.node;
using magic.node.extensions;
using magic.signals.contracts;

namespace magic.lambda.slots
{
    [Slot(Name = "test.reset-job-count")]
    public class JobCount : ISlot
    {
        internal static int _count;
        internal static ManualResetEvent _waiter;
        internal static int _maxJobs;
        public void Signal(ISignaler signaler, Node input)
        {
            _count = 0;
            _waiter?.Dispose();
            _waiter = new ManualResetEvent(false);
            _maxJobs = input.GetEx<int>();
        }
    }

    [Slot(Name = "test.increase-job-count")]
    public class IncreaseJobCount : ISlot
    {
        public void Signal(ISignaler signaler, Node input)
        {
            if (Interlocked.Increment(ref JobCount._count) >= JobCount._maxJobs)
                JobCount._waiter.Set();
        }
    }

    [Slot(Name = "test.wait-for-jobs")]
    public class WaitForJobs : ISlot
    {
        public void Signal(ISignaler signaler, Node input)
        {
            JobCount._waiter.WaitOne();
        }
    }
}

Then you can find our Hyperlambda code file below.

/*
 * How many jobs we want to create.
 */
.max:int:4000

/*
 * Resetting job count.
 */
test.reset-job-count:x:@.max

/*
 * Creating a bunch of jobs.
 */
.no:int:0
while
   lt
      get-value:x:@.no
      get-value:x:@.max
   .lambda

      /*
       * Creating a unique name for each job,
       * and creating our job afterwards.
       */
      strings.concat
         .:some-task-
         get-value:x:@.no
      scheduler.tasks.create:x:@strings.concat
         immediate
         persisted:bool:false
         .lambda
            test.increase-job-count

      /*
       * Incrementing job count.
       */
      set-value:x:@.no
         math.increment:x:@.no

/*
 * Waiting for all jobs to finish.
 */
test.wait-for-jobs

I want to emphasise, that this test was conducted with a branch of Magic that has not yet been release. But the above features will be released in the 5.5.6 version of Magic, coming up in a couple of days.

Even without that amount of throughput, Magic is still pretty awesome. Feel free to download it here.

Published by servergardens

Create .Net Core CRUD APIs wrapping your SQL database in seconds