Sidekiq is a great framework for asynchronous processing for Ruby on Rails applications. It uses Redis to store jobs. We pay for the Enterprise version as it provides additional features. It allows us to spread load, even with bursts over time, so that our infrastructure could be used efficiently. Our major bottleneck is the PostgreSQL databases we use to compute & store our analytics data.<\/p>\n\n\n\n
Background processing is handy as you don\u2019t have to be always scaled up to handle the peak load on demand, when most of the time you need much less capacity. But at the same time, you also have to be careful not to break important non-functional requirements. One of them is that accounts should not be starved while another account is running a large processing batch.<\/p>\n\n\n\n
How does our application scale?<\/strong><\/h2>\n\n\n\nBefore we dive into Sidekiq, it\u2019s important to explain the relevant context of our application. To scale we\u2019ve been using database sharding since the application\u2019s inception:<\/p>\n\n\n\n
A database shard, or simply a shard, is a horizontal partition of data in a database or search engine. Each shard is held on a separate database server instance, to spread load.<\/p>\n\n\n\n
\u2014 Shard (database architecture)<\/a>, Wikipedia<\/p>\n\n\n\nIn our case we split data to separate shards by account. And as our user base grows, we just add a new database shard from time to time.<\/p>\n\n\n
\n![\"what's](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Shard-selection.png\")
<\/a><\/figure>\n<\/div>\n\n\nThe load should spread evenly as long as the accounts behave randomly without huge excesses. This is not always the case, though, and sometimes the main limitation of this approach becomes apparent \u2014 one shard has a fixed capacity and you can\u2019t give an account the processing capacity of more than one shard. There are remedies like vertical scaling of database (ie. upgrade the instance type on AWS), but they\u2019re expensive and optimally we\u2019d like the system to self-regulate.<\/p>\n\n\n\n
Sidekiq cluster<\/strong><\/h2>\n\n\n\nBefore we start talking about the queues let me explain how we had the infrastructure set up. That way you will have a better idea about the components we\u2019re going to talk about. First let\u2019s have a look at how you can set up a Sidekiq cluster in theory:<\/p>\n\n\n\n\n![\"Sidekiq](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Sidekiq-Cluster.png\")
<\/a><\/figure>\n<\/figure>\n\n\n\nThe smallest grain of processing capacity in this overview is the thread<\/strong>. It gives you processing capability for one job<\/strong> at a time. A job has a job class<\/strong> and arguments. These are the very basics of Sidekiq. You can think about the thread as the consumer part in the distributed<\/em> producer-consumer<\/em> pattern.<\/p>\n\n\n\nHow do you determine which jobs to assign to threads? You configure this for each process<\/strong>. It has a list of queues<\/strong> to consume from (eg. Q1, Q2\u2026). Now, you can use weighted queues<\/a>, but we opted for strict priority as it gives better performance and it\u2019s easier to reason about.<\/p>\n\n\n\nYou can have multiple processes per instance<\/strong>, which will typically live somewhere in the cloud, for example as an AWS EC2 instance. Perhaps you want to set up a certain arrangement of configurations for such an instance – let\u2019s call it a role<\/strong>. You can then deploy multiple copies of such instances with the same role to scale. All of this together we call a Sidekiq cluster<\/strong>.<\/p>\n\n\n\nOriginal setup<\/strong><\/h2>\n\n\n\nWe had multiple queues that were named by what the jobs there were approximately doing. But also a lot of them were named after their teams or subsystems managed by those teams. This unsurprisingly follows Conway’s law<\/a>:<\/p>\n\n\n\n\nAny organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.<\/p>\n<\/blockquote>\n\n\n\n
We had some queues that implied low or high priority, but also a very highly prioritized default<\/code> queue \u2014 so if you just forgot to name a specific queue, your job class got really lucky. It now had the highest priority in the app, congratulations! We also had a mixture of roles in the cluster, which shuffled these queues in different setups, which looked somewhat random. That often led to unexpected side effects, like a dedicated capacity for low priority queues that ran jobs even during high load.<\/p>\n\n\n\n![\"queues](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Queues.png\")
<\/a><\/figure>\n<\/div>\n\n\nHow did we arrive to such a situation? Over the years different developers added new jobs to implement features and always tweaked the queues just a little bit to make their one feature work. However after many such iterations, while the system didn\u2019t have any guiding design principle, it became a very complicated tangle of queues, where nobody understood all the consequences.<\/p>\n\n\n\n
Another problem was scalability. Each instance that we had was running several processes with a mixture of queues, several virtual central processing units<\/a> (vCPUs) and 20 GB of RAM. To consume some queue faster, you had to add expensive instances and we had little idea how efficiently we were using them. The catch is that Ruby processes running on CRuby can leverage just one vCPU disregarding number of internal threads due to Global Interpreter Lock<\/a>.<\/p>\n\n\n\nImagine you have 6 vCPUs on an instance, but only 2 processes are consuming queues with jobs:<\/p>\n\n\n
\n![\"utilization](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/CPU-utilization.png\")
<\/a><\/figure>\n<\/div>\n\n\nYou\u2019ll see around 33 % utilization, because the other 4 cores have nothing to do. If you don\u2019t know the above-mentioned technical details, you might be left wondering how come the queues are not being consumed while the instance is underutilized. But in reality the ones that can do something, are overwhelmed.<\/p>\n\n\n\n
Typical incident scenario<\/strong><\/h2>\n\n\n\nTo illustrate better our increasing motivation for the new design, let\u2019s imagine a typical incident. You come to work and you see a queue is growing for a couple of hours linearly. No signs of the situation improving. You also see warnings that one shard is very busy, but other shards are not:<\/p>\n\n\n
\n![\"typical](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Typical-incident.png\")
<\/a><\/figure>\n<\/div>\n\n\nAdditionally, when you look at pgbouncer connection pool statistics, you see a flat line near the maximum for the overwhelmed shard. You can see client connections waiting to get anything done on the database and when they get there, the queries run excessively long. This means the jobs also run excessively long. Meanwhile all Sidekiq threads that can process the growing queue are busy, while others are sitting idle. All of this points to the fact that our design is not distributing load evenly across shards, so that jobs that could easily get processed on underutilized shards wait behind jobs that struggle on the saturated DB shard.<\/p>\n\n\n\n
Introducing: Kubernetes<\/strong><\/h2>\n\n\n\nPreviously we migrated to AWS<\/a>, and with that we also introduced Kubernetes into our infrastructure stack. With this step we gained new possibilities of infrastructure as code (IaC) and autoscaling. However, to leverage the cloud resources fully and perhaps to save some budget we had to first research a bit and adjust the system.<\/p>\n\n\n\nThe original setup was not ready for autoscaling – the instances were too big and it was not clear what metric to pick for scaling. Even the jobs themselves were not ready as they expected the instances to run continually for 24 hours between deployments. So, we knew we had to get smaller instances and optimize some of the jobs, but what queues should we use?<\/p>\n\n\n\n
Learning<\/strong><\/h2>\n\n\n\nBefore designing the changes, I already realized that throwing jobs into random queues doesn\u2019t scale. I also noticed we\u2019re lacking a meaningful way to distribute the load generated by our asynchronous processing on the database shards. This became obvious through addressing repeated incidents when a particular database shard got saturated either on CPU or allocated I\/O operations per second<\/a> (IOPS). Meanwhile other shards were not loaded, even though we had jobs for those shards sitting for very long time in the queues.<\/p>\n\n\n\n![\"](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Untitled-1024x640.png\")
<\/a><\/figure>\n<\/div>\n\n\nA colleague recommended to me the book & workshop Sidekiq in Practice<\/a> by Nate Berkopec. It was a very insightful reading that inspired me to rethink the system from ground up.<\/p>\n\n\n\nThinking about users<\/strong><\/h2>\n\n\n\nUntil now we talked about different system parameters, but we stayed completely silent about the final consumers of the analytics \u2014 the users. The ultimate goal of all of our processing is to be ready to show something to the user \u2014 be it a notification, a chart or some numbers in a table. Our users need those to gain insights about their business. And different parts need to be delivered within different timelines. For this we leverage the SLA-SLO-SLI framework<\/a>:<\/p>\n\n\n\n![\"SLA,](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/SLA-SLO-SLI.png\")
<\/a><\/figure>\n<\/div>\n\n\nIt is important to realize two facts:<\/p>\n\n\n\n
\n- Your SLOs must be measurable and that means you must be able to come up with matching SLIs. Otherwise you won\u2019t be able to evaluate impact of changes and you won\u2019t know if they are for the better or worse.<\/li>\n\n\n\n
- Fulfilling individual SLOs might not correspond to fulfilling the overall SLA. Think about a user waiting for an action to finish \u2014 but the action is composed of multiple jobs. Individual jobs might run fast, but maybe they interact in such a way that the overall action takes long time to finish. That\u2019s why it\u2019s also a good idea to have end-to-end measurements.<\/li>\n<\/ul>\n\n\n\n
Finally, the new design<\/strong><\/h2>\n\n\n\nIf you managed to read this far, I have good news! We now have enough background to introduce the new design that solves our issues and adds some perks as a bonus.<\/p>\n\n\n
\n![\"Finally,](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Queues-in-diagram-1.png\")
<\/a><\/figure>\n<\/div>\n\n\nThe diagram might seem complex \u2014 but in fact it\u2019s nothing more than a couple of standard Sidekiq first-in-first-out<\/em> (FIFO) queues (represented by rectangles). Each row represents queues for jobs that connect to one database shard. Each column represents a priority level, or in other words acceptable delay, or the SLO.<\/p>\n\n\n\nAs a consequence of this setup each job must know 2 things when being pushed (created) \u2014 its importance for the user and the shard:<\/p>\n\n\n\n
\n- The shard<\/strong> is easy, because it is determined by the account. Jobs operating on multiple shards can (and should) always be split into N independent jobs running on a single shard. By removing interdependence between shards, you achieve isolation of queues. Then if an incident happens, the impact to your system will be limited and you have achieved partial availability, which is better than full unavailability.<\/li>\n\n\n\n
- The priority<\/strong> is trickier, because now the author\/maintainer of a job class must think about how it will impact SLA. But also other questions, like how many such jobs might be pushed in a burst, what latency characteristics do the jobs have and so forth. To make this long-term maintainable, we created a documentation-heavy class with a sort of a questionnaire that enables developers to evaluate themselves, whether the job class fulfills criteria for higher priority queues.<\/li>\n<\/ul>\n\n\n\n
Recently, we have also finished sharding our ingestion database<\/a> to remove a single point of failure (SPOF) and as you can see now, it ties nicely in this design, distributing the load over the shards.<\/p>\n\n\n\nMeasurement (SLI)<\/strong><\/h2>\n\n\n\nTo match the job delay SLO per queue, we need to measure it and compare somehow. We collect the delay of the oldest job in a queue and send it to DataDog to generate insightful dashboards:<\/p>\n\n\n
\n![\"dashboards](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Schermafdruk-van-2023-07-14-15-16-43-1024x509.png\")
<\/a><\/figure>\n<\/div>\n\n\nOn this screenshot you can see two of our shards and their 5 queues each. Highest priority on the left, longest acceptable latency on the right. I picked this representation, because for each SLI it shows current value, but also the history and range of the value over a given period (an hour for example). That gives you a good idea of how the situation is developing.<\/p>\n\n\n\n
When the latency breaches the SLO, the color changes to orange<\/em>. And as you can see in this example, the shards are indeed isolated, allowing us to maintain SLOs elsewhere, even though shard 05 had issues with a large burst of jobs that we had to manually throttle. On the infrastructure side they also scale separately (see the next section).<\/p>\n\n\n\nI really like how this conveys information about the whole system inside one page with the possibility to drill down to details. If you\u2019re looking for inspiration on how to make great readable charts, then The Visual Display of Quantitative Information<\/a> by Edward Tufte is a must-read.<\/p>\n\n\n\nBut eye candy is not the only purpose and even we at ChartMogul don\u2019t have the whole day to stare at charts. We still want to be aware ASAP when something goes wrong, so when a delay reaches red<\/em> (twice as expected), we trigger an on-call alarm. We could only do this once we became confident enough about fulfilling the SLO. That required a lot of optimization and refactoring of the jobs that now have to live up to higher standards.<\/p>\n\n\n\nBut in reality this is not the only dashboard. We still need more that actually measure concrete complex operations from start to end throughout the different services & jobs. Those give better answers on such questions as how long does it take to process a webhook from receiving to showing a notification to the user<\/em>.<\/p>\n\n\n\nAutoscaling<\/strong><\/h2>\n\n\n\nAs I\u2019ve already mentioned, we automatically scale the queue processing capacity, but how do we do it precisely? We leverage Horizontal Pod Autoscaling<\/a> (HPA) that controls the number of pods inside an autoscaling group and prometheus<\/a> that provides the latency metrics scraped using Sidekiq API.<\/p>\n\n\n\nWe cannot scale the database easily, so there are limits on how many connections can be active through pgbouncer<\/code> pools. But we can scale the processing capacity of Sidekiq, so that we pay for less Sidekiq threads and less node instances on AWS. Our current topology looks like this:<\/p>\n\n\n\n![\"autoscaling\"](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Autoscaling.png\")
<\/a><\/figure>\n<\/div>\n\n\nThe number of containers in each shard group goes up and down as the demand by the accounts there changes. Meanwhile the number of nodes changes as well according to how many containers overall are required. There are some delays on both levels for practical reasons. For example to reduce disruption you need to tweak the autoscaling policy and introduce a graceful shutdown period.<\/p>\n\n\n\n
Now, why did we choose the minimal containers? Each of our containers has just 1 vCPU and a limited amount of memory. This gives us greater flexibility, because we can shift processing capacity granularly, ie. by small increments. In other words, we don\u2019t have to add or remove big virtual machines all the time. It also gives us transparent infrastructure metrics as we can directly observe the utilization of the vCPUs.<\/p>\n\n\n\n
Threads and memory<\/strong><\/h2>\n\n\n\nOriginally we had 20 threads per process. However, using DataDog APM Traces & Container monitoring we found out that the vCPUs were often close to 100 % during peak throughput and jobs took longer than necessary with suspicious gaps in traces. Those gaps were in fact the waiting of the ruby thread to get CPU time. So nowadays we settled on just 4 threads as it seems to be the best for our ratio of I\/O wait to computation. Sidekiq\u2019s default used to be higher, but it is now just 5<\/a>.<\/p>\n\n\n\nA bigger container with multiple vCPUs, processes and large memory does have the advantage that occasional memory-hungry jobs still manage to finish without an out-of-memory kill\u2026 as long as they don\u2019t throw a party on one container at the same time, that is. However, we opted to rather limit the impact of such jobs and either fix them or send them to a special quarantine<\/em> queue with extra memory, but less capacity. This means the normal pods can stay lean and optimized for 99.9% of the jobs.<\/p>\n\n\n\nWhile the following long-term distribution doesn\u2019t tell the whole story, it shows the balance we struck between perfect CPU utilization and minimal latency of jobs caused by CPU saturation:<\/p>\n\n\n
\n![\"CPU](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Untitled-1.png\")
<\/a><\/figure>\n<\/div>\n\n\nLong-running jobs<\/strong><\/h2>\n\n\n\nSidekiq has a self-protective mechanism against crashing jobs called poison pills<\/em>. If jobs are detected to be present during multiple crashes, they\u2019re not retried and marked directly as dead<\/em>. We encountered this already with out of memory crashes (OOM). Such jobs can be usually optimized to prevent such errors. But it can also detect some jobs as false positives. And after we migrated to an autoscaled architecture, we realized this was happening to us, here\u2019s why:<\/p>\n\n\n\nIf you have autoscaling instances in the cloud, your jobs can\u2019t rely on them existing \u201cforever\u201d. So it\u2019s best if you can split the workload such that the jobs finish quickly. Let\u2019s say our goal is 5 minutes. Then you can also set your graceful shutdown to respect this.<\/p>\n\n\n\n
However, what if your job runs for hours<\/em> and it\u2019s hard to refactor? Well, we have put those in a special queue and on a container role that wasn\u2019t autoscaled and we thought the job was done. But we still saw that the jobs were randomly killed, even without the graceful shutdown!<\/p>\n\n\n\nWhat we didn\u2019t realize at first was that we put it together with other autoscaled containers on the same node group in Kubernetes, so sometimes it decided it was a good idea to reshuffle the long-running pods and unfortunately, right now Kubernetes doesn\u2019t respect graceful period for this. In the end, we moved this job into a separate node group and also started refactoring the job.<\/p>\n\n\n\n
One aspect that we could still improve in our design is perhaps not putting the 1 minute through 24 hours SLO queues on the same roles, as they\u2019re mixing widely different latency expectations. So far we circumvented this problem by requiring 5 minutes latency in the 99th percentile for all jobs. But you can easily imagine with a thought experiment a large burst of 5-minute jobs would cause issues in the 1-minute queue quickly, we\u2019ll discuss how we tackle this with rate limiting later on.<\/p>\n\n\n\n
What about fair scheduling?<\/strong><\/h2>\n\n\n\nOne of the struggle you encounter on a multi-tenant application with shared resources is the competition for these resources. Optimally, you want your application to feel fast for an account that needs to compute a lot of data, however you have to be careful not to starve other accounts:<\/p>\n\n\n\n
\nresource starvation<\/strong> is a problem encountered in concurrent computing where a process is perpetually denied necessary resources to process its work.<\/p>\n<\/blockquote>\n\n\n\n\u2014 Starvation (computer science)<\/a>, Wikipedia<\/p>\n\n\n\nIt\u2019s an interesting problem, so let me go a bit deeper into it. We attempted to solve this puzzle once and for all in 2018 with a custom module<\/a> for Sidekiq that introduced prioritized queues, it worked like this (green = jobs added, red = jobs removed):<\/p>\n\n\n\n![\"prioritized](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Prioritized-queues.png\")
<\/a><\/figure>\n<\/div>\n\n\nWhen inserting new jobs, they got into a subqueue per account ID. We kept track of how many jobs there were for an account. New jobs then got a higher score. We used Redis command ZPOPMIN<\/a> to process the jobs with lowest score first. This works! Job done?<\/p>\n\n\n\nUnfortunately, no. On the face of it, this algorithm is very fair to all accounts (at least in regards to number of jobs). However it has pitfalls. First let\u2019s see the example where the queue stops working as FIFO:<\/p>\n\n\n
In our case we split data to separate shards by account. And as our user base grows, we just add a new database shard from time to time.<\/p>\n\n\n
<\/a><\/figure>\n<\/div>\n\n\nThe load should spread evenly as long as the accounts behave randomly without huge excesses. This is not always the case, though, and sometimes the main limitation of this approach becomes apparent \u2014 one shard has a fixed capacity and you can\u2019t give an account the processing capacity of more than one shard. There are remedies like vertical scaling of database (ie. upgrade the instance type on AWS), but they\u2019re expensive and optimally we\u2019d like the system to self-regulate.<\/p>\n\n\n\n
Sidekiq cluster<\/strong><\/h2>\n\n\n\nBefore we start talking about the queues let me explain how we had the infrastructure set up. That way you will have a better idea about the components we\u2019re going to talk about. First let\u2019s have a look at how you can set up a Sidekiq cluster in theory:<\/p>\n\n\n\n\n<\/a><\/figure>\n<\/figure>\n\n\n\nThe smallest grain of processing capacity in this overview is the thread<\/strong>. It gives you processing capability for one job<\/strong> at a time. A job has a job class<\/strong> and arguments. These are the very basics of Sidekiq. You can think about the thread as the consumer part in the distributed<\/em> producer-consumer<\/em> pattern.<\/p>\n\n\n\nHow do you determine which jobs to assign to threads? You configure this for each process<\/strong>. It has a list of queues<\/strong> to consume from (eg. Q1, Q2\u2026). Now, you can use weighted queues<\/a>, but we opted for strict priority as it gives better performance and it\u2019s easier to reason about.<\/p>\n\n\n\nYou can have multiple processes per instance<\/strong>, which will typically live somewhere in the cloud, for example as an AWS EC2 instance. Perhaps you want to set up a certain arrangement of configurations for such an instance – let\u2019s call it a role<\/strong>. You can then deploy multiple copies of such instances with the same role to scale. All of this together we call a Sidekiq cluster<\/strong>.<\/p>\n\n\n\nOriginal setup<\/strong><\/h2>\n\n\n\nWe had multiple queues that were named by what the jobs there were approximately doing. But also a lot of them were named after their teams or subsystems managed by those teams. This unsurprisingly follows Conway’s law<\/a>:<\/p>\n\n\n\n\nAny organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.<\/p>\n<\/blockquote>\n\n\n\n
We had some queues that implied low or high priority, but also a very highly prioritized default<\/code> queue \u2014 so if you just forgot to name a specific queue, your job class got really lucky. It now had the highest priority in the app, congratulations! We also had a mixture of roles in the cluster, which shuffled these queues in different setups, which looked somewhat random. That often led to unexpected side effects, like a dedicated capacity for low priority queues that ran jobs even during high load.<\/p>\n\n\n\n<\/a><\/figure>\n<\/div>\n\n\nHow did we arrive to such a situation? Over the years different developers added new jobs to implement features and always tweaked the queues just a little bit to make their one feature work. However after many such iterations, while the system didn\u2019t have any guiding design principle, it became a very complicated tangle of queues, where nobody understood all the consequences.<\/p>\n\n\n\n
Another problem was scalability. Each instance that we had was running several processes with a mixture of queues, several virtual central processing units<\/a> (vCPUs) and 20 GB of RAM. To consume some queue faster, you had to add expensive instances and we had little idea how efficiently we were using them. The catch is that Ruby processes running on CRuby can leverage just one vCPU disregarding number of internal threads due to Global Interpreter Lock<\/a>.<\/p>\n\n\n\nImagine you have 6 vCPUs on an instance, but only 2 processes are consuming queues with jobs:<\/p>\n\n\n
\n<\/a><\/figure>\n<\/div>\n\n\nYou\u2019ll see around 33 % utilization, because the other 4 cores have nothing to do. If you don\u2019t know the above-mentioned technical details, you might be left wondering how come the queues are not being consumed while the instance is underutilized. But in reality the ones that can do something, are overwhelmed.<\/p>\n\n\n\n
Typical incident scenario<\/strong><\/h2>\n\n\n\nTo illustrate better our increasing motivation for the new design, let\u2019s imagine a typical incident. You come to work and you see a queue is growing for a couple of hours linearly. No signs of the situation improving. You also see warnings that one shard is very busy, but other shards are not:<\/p>\n\n\n
\n<\/a><\/figure>\n<\/div>\n\n\nAdditionally, when you look at pgbouncer connection pool statistics, you see a flat line near the maximum for the overwhelmed shard. You can see client connections waiting to get anything done on the database and when they get there, the queries run excessively long. This means the jobs also run excessively long. Meanwhile all Sidekiq threads that can process the growing queue are busy, while others are sitting idle. All of this points to the fact that our design is not distributing load evenly across shards, so that jobs that could easily get processed on underutilized shards wait behind jobs that struggle on the saturated DB shard.<\/p>\n\n\n\n
Introducing: Kubernetes<\/strong><\/h2>\n\n\n\nPreviously we migrated to AWS<\/a>, and with that we also introduced Kubernetes into our infrastructure stack. With this step we gained new possibilities of infrastructure as code (IaC) and autoscaling. However, to leverage the cloud resources fully and perhaps to save some budget we had to first research a bit and adjust the system.<\/p>\n\n\n\nThe original setup was not ready for autoscaling – the instances were too big and it was not clear what metric to pick for scaling. Even the jobs themselves were not ready as they expected the instances to run continually for 24 hours between deployments. So, we knew we had to get smaller instances and optimize some of the jobs, but what queues should we use?<\/p>\n\n\n\n
Learning<\/strong><\/h2>\n\n\n\nBefore designing the changes, I already realized that throwing jobs into random queues doesn\u2019t scale. I also noticed we\u2019re lacking a meaningful way to distribute the load generated by our asynchronous processing on the database shards. This became obvious through addressing repeated incidents when a particular database shard got saturated either on CPU or allocated I\/O operations per second<\/a> (IOPS). Meanwhile other shards were not loaded, even though we had jobs for those shards sitting for very long time in the queues.<\/p>\n\n\n\n<\/a><\/figure>\n<\/div>\n\n\nA colleague recommended to me the book & workshop Sidekiq in Practice<\/a> by Nate Berkopec. It was a very insightful reading that inspired me to rethink the system from ground up.<\/p>\n\n\n\nThinking about users<\/strong><\/h2>\n\n\n\nUntil now we talked about different system parameters, but we stayed completely silent about the final consumers of the analytics \u2014 the users. The ultimate goal of all of our processing is to be ready to show something to the user \u2014 be it a notification, a chart or some numbers in a table. Our users need those to gain insights about their business. And different parts need to be delivered within different timelines. For this we leverage the SLA-SLO-SLI framework<\/a>:<\/p>\n\n\n\n<\/a><\/figure>\n<\/div>\n\n\nIt is important to realize two facts:<\/p>\n\n\n\n
\n- Your SLOs must be measurable and that means you must be able to come up with matching SLIs. Otherwise you won\u2019t be able to evaluate impact of changes and you won\u2019t know if they are for the better or worse.<\/li>\n\n\n\n
- Fulfilling individual SLOs might not correspond to fulfilling the overall SLA. Think about a user waiting for an action to finish \u2014 but the action is composed of multiple jobs. Individual jobs might run fast, but maybe they interact in such a way that the overall action takes long time to finish. That\u2019s why it\u2019s also a good idea to have end-to-end measurements.<\/li>\n<\/ul>\n\n\n\n
Finally, the new design<\/strong><\/h2>\n\n\n\nIf you managed to read this far, I have good news! We now have enough background to introduce the new design that solves our issues and adds some perks as a bonus.<\/p>\n\n\n
\n<\/a><\/figure>\n<\/div>\n\n\nThe diagram might seem complex \u2014 but in fact it\u2019s nothing more than a couple of standard Sidekiq first-in-first-out<\/em> (FIFO) queues (represented by rectangles). Each row represents queues for jobs that connect to one database shard. Each column represents a priority level, or in other words acceptable delay, or the SLO.<\/p>\n\n\n\nAs a consequence of this setup each job must know 2 things when being pushed (created) \u2014 its importance for the user and the shard:<\/p>\n\n\n\n
\n- The shard<\/strong> is easy, because it is determined by the account. Jobs operating on multiple shards can (and should) always be split into N independent jobs running on a single shard. By removing interdependence between shards, you achieve isolation of queues. Then if an incident happens, the impact to your system will be limited and you have achieved partial availability, which is better than full unavailability.<\/li>\n\n\n\n
- The priority<\/strong> is trickier, because now the author\/maintainer of a job class must think about how it will impact SLA. But also other questions, like how many such jobs might be pushed in a burst, what latency characteristics do the jobs have and so forth. To make this long-term maintainable, we created a documentation-heavy class with a sort of a questionnaire that enables developers to evaluate themselves, whether the job class fulfills criteria for higher priority queues.<\/li>\n<\/ul>\n\n\n\n
Recently, we have also finished sharding our ingestion database<\/a> to remove a single point of failure (SPOF) and as you can see now, it ties nicely in this design, distributing the load over the shards.<\/p>\n\n\n\nMeasurement (SLI)<\/strong><\/h2>\n\n\n\nTo match the job delay SLO per queue, we need to measure it and compare somehow. We collect the delay of the oldest job in a queue and send it to DataDog to generate insightful dashboards:<\/p>\n\n\n
\n<\/a><\/figure>\n<\/div>\n\n\nOn this screenshot you can see two of our shards and their 5 queues each. Highest priority on the left, longest acceptable latency on the right. I picked this representation, because for each SLI it shows current value, but also the history and range of the value over a given period (an hour for example). That gives you a good idea of how the situation is developing.<\/p>\n\n\n\n
When the latency breaches the SLO, the color changes to orange<\/em>. And as you can see in this example, the shards are indeed isolated, allowing us to maintain SLOs elsewhere, even though shard 05 had issues with a large burst of jobs that we had to manually throttle. On the infrastructure side they also scale separately (see the next section).<\/p>\n\n\n\nI really like how this conveys information about the whole system inside one page with the possibility to drill down to details. If you\u2019re looking for inspiration on how to make great readable charts, then The Visual Display of Quantitative Information<\/a> by Edward Tufte is a must-read.<\/p>\n\n\n\nBut eye candy is not the only purpose and even we at ChartMogul don\u2019t have the whole day to stare at charts. We still want to be aware ASAP when something goes wrong, so when a delay reaches red<\/em> (twice as expected), we trigger an on-call alarm. We could only do this once we became confident enough about fulfilling the SLO. That required a lot of optimization and refactoring of the jobs that now have to live up to higher standards.<\/p>\n\n\n\nBut in reality this is not the only dashboard. We still need more that actually measure concrete complex operations from start to end throughout the different services & jobs. Those give better answers on such questions as how long does it take to process a webhook from receiving to showing a notification to the user<\/em>.<\/p>\n\n\n\nAutoscaling<\/strong><\/h2>\n\n\n\nAs I\u2019ve already mentioned, we automatically scale the queue processing capacity, but how do we do it precisely? We leverage Horizontal Pod Autoscaling<\/a> (HPA) that controls the number of pods inside an autoscaling group and prometheus<\/a> that provides the latency metrics scraped using Sidekiq API.<\/p>\n\n\n\nWe cannot scale the database easily, so there are limits on how many connections can be active through pgbouncer<\/code> pools. But we can scale the processing capacity of Sidekiq, so that we pay for less Sidekiq threads and less node instances on AWS. Our current topology looks like this:<\/p>\n\n\n\n<\/a><\/figure>\n<\/div>\n\n\nThe number of containers in each shard group goes up and down as the demand by the accounts there changes. Meanwhile the number of nodes changes as well according to how many containers overall are required. There are some delays on both levels for practical reasons. For example to reduce disruption you need to tweak the autoscaling policy and introduce a graceful shutdown period.<\/p>\n\n\n\n
Now, why did we choose the minimal containers? Each of our containers has just 1 vCPU and a limited amount of memory. This gives us greater flexibility, because we can shift processing capacity granularly, ie. by small increments. In other words, we don\u2019t have to add or remove big virtual machines all the time. It also gives us transparent infrastructure metrics as we can directly observe the utilization of the vCPUs.<\/p>\n\n\n\n
Threads and memory<\/strong><\/h2>\n\n\n\nOriginally we had 20 threads per process. However, using DataDog APM Traces & Container monitoring we found out that the vCPUs were often close to 100 % during peak throughput and jobs took longer than necessary with suspicious gaps in traces. Those gaps were in fact the waiting of the ruby thread to get CPU time. So nowadays we settled on just 4 threads as it seems to be the best for our ratio of I\/O wait to computation. Sidekiq\u2019s default used to be higher, but it is now just 5<\/a>.<\/p>\n\n\n\nA bigger container with multiple vCPUs, processes and large memory does have the advantage that occasional memory-hungry jobs still manage to finish without an out-of-memory kill\u2026 as long as they don\u2019t throw a party on one container at the same time, that is. However, we opted to rather limit the impact of such jobs and either fix them or send them to a special quarantine<\/em> queue with extra memory, but less capacity. This means the normal pods can stay lean and optimized for 99.9% of the jobs.<\/p>\n\n\n\nWhile the following long-term distribution doesn\u2019t tell the whole story, it shows the balance we struck between perfect CPU utilization and minimal latency of jobs caused by CPU saturation:<\/p>\n\n\n
\n<\/a><\/figure>\n<\/div>\n\n\nLong-running jobs<\/strong><\/h2>\n\n\n\nSidekiq has a self-protective mechanism against crashing jobs called poison pills<\/em>. If jobs are detected to be present during multiple crashes, they\u2019re not retried and marked directly as dead<\/em>. We encountered this already with out of memory crashes (OOM). Such jobs can be usually optimized to prevent such errors. But it can also detect some jobs as false positives. And after we migrated to an autoscaled architecture, we realized this was happening to us, here\u2019s why:<\/p>\n\n\n\nIf you have autoscaling instances in the cloud, your jobs can\u2019t rely on them existing \u201cforever\u201d. So it\u2019s best if you can split the workload such that the jobs finish quickly. Let\u2019s say our goal is 5 minutes. Then you can also set your graceful shutdown to respect this.<\/p>\n\n\n\n
However, what if your job runs for hours<\/em> and it\u2019s hard to refactor? Well, we have put those in a special queue and on a container role that wasn\u2019t autoscaled and we thought the job was done. But we still saw that the jobs were randomly killed, even without the graceful shutdown!<\/p>\n\n\n\nWhat we didn\u2019t realize at first was that we put it together with other autoscaled containers on the same node group in Kubernetes, so sometimes it decided it was a good idea to reshuffle the long-running pods and unfortunately, right now Kubernetes doesn\u2019t respect graceful period for this. In the end, we moved this job into a separate node group and also started refactoring the job.<\/p>\n\n\n\n
One aspect that we could still improve in our design is perhaps not putting the 1 minute through 24 hours SLO queues on the same roles, as they\u2019re mixing widely different latency expectations. So far we circumvented this problem by requiring 5 minutes latency in the 99th percentile for all jobs. But you can easily imagine with a thought experiment a large burst of 5-minute jobs would cause issues in the 1-minute queue quickly, we\u2019ll discuss how we tackle this with rate limiting later on.<\/p>\n\n\n\n
What about fair scheduling?<\/strong><\/h2>\n\n\n\nOne of the struggle you encounter on a multi-tenant application with shared resources is the competition for these resources. Optimally, you want your application to feel fast for an account that needs to compute a lot of data, however you have to be careful not to starve other accounts:<\/p>\n\n\n\n
\nresource starvation<\/strong> is a problem encountered in concurrent computing where a process is perpetually denied necessary resources to process its work.<\/p>\n<\/blockquote>\n\n\n\n\u2014 Starvation (computer science)<\/a>, Wikipedia<\/p>\n\n\n\nIt\u2019s an interesting problem, so let me go a bit deeper into it. We attempted to solve this puzzle once and for all in 2018 with a custom module<\/a> for Sidekiq that introduced prioritized queues, it worked like this (green = jobs added, red = jobs removed):<\/p>\n\n\n\n<\/a><\/figure>\n<\/div>\n\n\nWhen inserting new jobs, they got into a subqueue per account ID. We kept track of how many jobs there were for an account. New jobs then got a higher score. We used Redis command ZPOPMIN<\/a> to process the jobs with lowest score first. This works! Job done?<\/p>\n\n\n\nUnfortunately, no. On the face of it, this algorithm is very fair to all accounts (at least in regards to number of jobs). However it has pitfalls. First let\u2019s see the example where the queue stops working as FIFO:<\/p>\n\n\n
<\/a><\/figure>\n<\/figure>\n\n\n\nThe smallest grain of processing capacity in this overview is the thread<\/strong>. It gives you processing capability for one job<\/strong> at a time. A job has a job class<\/strong> and arguments. These are the very basics of Sidekiq. You can think about the thread as the consumer part in the distributed<\/em> producer-consumer<\/em> pattern.<\/p>\n\n\n\n How do you determine which jobs to assign to threads? You configure this for each process<\/strong>. It has a list of queues<\/strong> to consume from (eg. Q1, Q2\u2026). Now, you can use weighted queues<\/a>, but we opted for strict priority as it gives better performance and it\u2019s easier to reason about.<\/p>\n\n\n\n You can have multiple processes per instance<\/strong>, which will typically live somewhere in the cloud, for example as an AWS EC2 instance. Perhaps you want to set up a certain arrangement of configurations for such an instance – let\u2019s call it a role<\/strong>. You can then deploy multiple copies of such instances with the same role to scale. All of this together we call a Sidekiq cluster<\/strong>.<\/p>\n\n\n\n We had multiple queues that were named by what the jobs there were approximately doing. But also a lot of them were named after their teams or subsystems managed by those teams. This unsurprisingly follows Conway’s law<\/a>:<\/p>\n\n\n\n Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.<\/p>\n<\/blockquote>\n\n\n\n We had some queues that implied low or high priority, but also a very highly prioritized How did we arrive to such a situation? Over the years different developers added new jobs to implement features and always tweaked the queues just a little bit to make their one feature work. However after many such iterations, while the system didn\u2019t have any guiding design principle, it became a very complicated tangle of queues, where nobody understood all the consequences.<\/p>\n\n\n\n Another problem was scalability. Each instance that we had was running several processes with a mixture of queues, several virtual central processing units<\/a> (vCPUs) and 20 GB of RAM. To consume some queue faster, you had to add expensive instances and we had little idea how efficiently we were using them. The catch is that Ruby processes running on CRuby can leverage just one vCPU disregarding number of internal threads due to Global Interpreter Lock<\/a>.<\/p>\n\n\n\n Imagine you have 6 vCPUs on an instance, but only 2 processes are consuming queues with jobs:<\/p>\n\n\n You\u2019ll see around 33 % utilization, because the other 4 cores have nothing to do. If you don\u2019t know the above-mentioned technical details, you might be left wondering how come the queues are not being consumed while the instance is underutilized. But in reality the ones that can do something, are overwhelmed.<\/p>\n\n\n\n To illustrate better our increasing motivation for the new design, let\u2019s imagine a typical incident. You come to work and you see a queue is growing for a couple of hours linearly. No signs of the situation improving. You also see warnings that one shard is very busy, but other shards are not:<\/p>\n\n\n Additionally, when you look at pgbouncer connection pool statistics, you see a flat line near the maximum for the overwhelmed shard. You can see client connections waiting to get anything done on the database and when they get there, the queries run excessively long. This means the jobs also run excessively long. Meanwhile all Sidekiq threads that can process the growing queue are busy, while others are sitting idle. All of this points to the fact that our design is not distributing load evenly across shards, so that jobs that could easily get processed on underutilized shards wait behind jobs that struggle on the saturated DB shard.<\/p>\n\n\n\n Previously we migrated to AWS<\/a>, and with that we also introduced Kubernetes into our infrastructure stack. With this step we gained new possibilities of infrastructure as code (IaC) and autoscaling. However, to leverage the cloud resources fully and perhaps to save some budget we had to first research a bit and adjust the system.<\/p>\n\n\n\n The original setup was not ready for autoscaling – the instances were too big and it was not clear what metric to pick for scaling. Even the jobs themselves were not ready as they expected the instances to run continually for 24 hours between deployments. So, we knew we had to get smaller instances and optimize some of the jobs, but what queues should we use?<\/p>\n\n\n\n Before designing the changes, I already realized that throwing jobs into random queues doesn\u2019t scale. I also noticed we\u2019re lacking a meaningful way to distribute the load generated by our asynchronous processing on the database shards. This became obvious through addressing repeated incidents when a particular database shard got saturated either on CPU or allocated I\/O operations per second<\/a> (IOPS). Meanwhile other shards were not loaded, even though we had jobs for those shards sitting for very long time in the queues.<\/p>\n\n\n A colleague recommended to me the book & workshop Sidekiq in Practice<\/a> by Nate Berkopec. It was a very insightful reading that inspired me to rethink the system from ground up.<\/p>\n\n\n\n Until now we talked about different system parameters, but we stayed completely silent about the final consumers of the analytics \u2014 the users. The ultimate goal of all of our processing is to be ready to show something to the user \u2014 be it a notification, a chart or some numbers in a table. Our users need those to gain insights about their business. And different parts need to be delivered within different timelines. For this we leverage the SLA-SLO-SLI framework<\/a>:<\/p>\n\n\n It is important to realize two facts:<\/p>\n\n\n\n If you managed to read this far, I have good news! We now have enough background to introduce the new design that solves our issues and adds some perks as a bonus.<\/p>\n\n\n The diagram might seem complex \u2014 but in fact it\u2019s nothing more than a couple of standard Sidekiq first-in-first-out<\/em> (FIFO) queues (represented by rectangles). Each row represents queues for jobs that connect to one database shard. Each column represents a priority level, or in other words acceptable delay, or the SLO.<\/p>\n\n\n\n As a consequence of this setup each job must know 2 things when being pushed (created) \u2014 its importance for the user and the shard:<\/p>\n\n\n\n Recently, we have also finished sharding our ingestion database<\/a> to remove a single point of failure (SPOF) and as you can see now, it ties nicely in this design, distributing the load over the shards.<\/p>\n\n\n\n To match the job delay SLO per queue, we need to measure it and compare somehow. We collect the delay of the oldest job in a queue and send it to DataDog to generate insightful dashboards:<\/p>\n\n\n On this screenshot you can see two of our shards and their 5 queues each. Highest priority on the left, longest acceptable latency on the right. I picked this representation, because for each SLI it shows current value, but also the history and range of the value over a given period (an hour for example). That gives you a good idea of how the situation is developing.<\/p>\n\n\n\n When the latency breaches the SLO, the color changes to orange<\/em>. And as you can see in this example, the shards are indeed isolated, allowing us to maintain SLOs elsewhere, even though shard 05 had issues with a large burst of jobs that we had to manually throttle. On the infrastructure side they also scale separately (see the next section).<\/p>\n\n\n\n I really like how this conveys information about the whole system inside one page with the possibility to drill down to details. If you\u2019re looking for inspiration on how to make great readable charts, then The Visual Display of Quantitative Information<\/a> by Edward Tufte is a must-read.<\/p>\n\n\n\n But eye candy is not the only purpose and even we at ChartMogul don\u2019t have the whole day to stare at charts. We still want to be aware ASAP when something goes wrong, so when a delay reaches red<\/em> (twice as expected), we trigger an on-call alarm. We could only do this once we became confident enough about fulfilling the SLO. That required a lot of optimization and refactoring of the jobs that now have to live up to higher standards.<\/p>\n\n\n\n But in reality this is not the only dashboard. We still need more that actually measure concrete complex operations from start to end throughout the different services & jobs. Those give better answers on such questions as how long does it take to process a webhook from receiving to showing a notification to the user<\/em>.<\/p>\n\n\n\n As I\u2019ve already mentioned, we automatically scale the queue processing capacity, but how do we do it precisely? We leverage Horizontal Pod Autoscaling<\/a> (HPA) that controls the number of pods inside an autoscaling group and prometheus<\/a> that provides the latency metrics scraped using Sidekiq API.<\/p>\n\n\n\n We cannot scale the database easily, so there are limits on how many connections can be active through The number of containers in each shard group goes up and down as the demand by the accounts there changes. Meanwhile the number of nodes changes as well according to how many containers overall are required. There are some delays on both levels for practical reasons. For example to reduce disruption you need to tweak the autoscaling policy and introduce a graceful shutdown period.<\/p>\n\n\n\n Now, why did we choose the minimal containers? Each of our containers has just 1 vCPU and a limited amount of memory. This gives us greater flexibility, because we can shift processing capacity granularly, ie. by small increments. In other words, we don\u2019t have to add or remove big virtual machines all the time. It also gives us transparent infrastructure metrics as we can directly observe the utilization of the vCPUs.<\/p>\n\n\n\n Originally we had 20 threads per process. However, using DataDog APM Traces & Container monitoring we found out that the vCPUs were often close to 100 % during peak throughput and jobs took longer than necessary with suspicious gaps in traces. Those gaps were in fact the waiting of the ruby thread to get CPU time. So nowadays we settled on just 4 threads as it seems to be the best for our ratio of I\/O wait to computation. Sidekiq\u2019s default used to be higher, but it is now just 5<\/a>.<\/p>\n\n\n\n A bigger container with multiple vCPUs, processes and large memory does have the advantage that occasional memory-hungry jobs still manage to finish without an out-of-memory kill\u2026 as long as they don\u2019t throw a party on one container at the same time, that is. However, we opted to rather limit the impact of such jobs and either fix them or send them to a special quarantine<\/em> queue with extra memory, but less capacity. This means the normal pods can stay lean and optimized for 99.9% of the jobs.<\/p>\n\n\n\n While the following long-term distribution doesn\u2019t tell the whole story, it shows the balance we struck between perfect CPU utilization and minimal latency of jobs caused by CPU saturation:<\/p>\n\n\n Sidekiq has a self-protective mechanism against crashing jobs called poison pills<\/em>. If jobs are detected to be present during multiple crashes, they\u2019re not retried and marked directly as dead<\/em>. We encountered this already with out of memory crashes (OOM). Such jobs can be usually optimized to prevent such errors. But it can also detect some jobs as false positives. And after we migrated to an autoscaled architecture, we realized this was happening to us, here\u2019s why:<\/p>\n\n\n\n If you have autoscaling instances in the cloud, your jobs can\u2019t rely on them existing \u201cforever\u201d. So it\u2019s best if you can split the workload such that the jobs finish quickly. Let\u2019s say our goal is 5 minutes. Then you can also set your graceful shutdown to respect this.<\/p>\n\n\n\n However, what if your job runs for hours<\/em> and it\u2019s hard to refactor? Well, we have put those in a special queue and on a container role that wasn\u2019t autoscaled and we thought the job was done. But we still saw that the jobs were randomly killed, even without the graceful shutdown!<\/p>\n\n\n\n What we didn\u2019t realize at first was that we put it together with other autoscaled containers on the same node group in Kubernetes, so sometimes it decided it was a good idea to reshuffle the long-running pods and unfortunately, right now Kubernetes doesn\u2019t respect graceful period for this. In the end, we moved this job into a separate node group and also started refactoring the job.<\/p>\n\n\n\n One aspect that we could still improve in our design is perhaps not putting the 1 minute through 24 hours SLO queues on the same roles, as they\u2019re mixing widely different latency expectations. So far we circumvented this problem by requiring 5 minutes latency in the 99th percentile for all jobs. But you can easily imagine with a thought experiment a large burst of 5-minute jobs would cause issues in the 1-minute queue quickly, we\u2019ll discuss how we tackle this with rate limiting later on.<\/p>\n\n\n\n One of the struggle you encounter on a multi-tenant application with shared resources is the competition for these resources. Optimally, you want your application to feel fast for an account that needs to compute a lot of data, however you have to be careful not to starve other accounts:<\/p>\n\n\n\n resource starvation<\/strong> is a problem encountered in concurrent computing where a process is perpetually denied necessary resources to process its work.<\/p>\n<\/blockquote>\n\n\n\n \u2014 Starvation (computer science)<\/a>, Wikipedia<\/p>\n\n\n\n It\u2019s an interesting problem, so let me go a bit deeper into it. We attempted to solve this puzzle once and for all in 2018 with a custom module<\/a> for Sidekiq that introduced prioritized queues, it worked like this (green = jobs added, red = jobs removed):<\/p>\n\n\n When inserting new jobs, they got into a subqueue per account ID. We kept track of how many jobs there were for an account. New jobs then got a higher score. We used Redis command ZPOPMIN<\/a> to process the jobs with lowest score first. This works! Job done?<\/p>\n\n\n\n Unfortunately, no. On the face of it, this algorithm is very fair to all accounts (at least in regards to number of jobs). However it has pitfalls. First let\u2019s see the example where the queue stops working as FIFO:<\/p>\n\n\nOriginal setup<\/strong><\/h2>\n\n\n\n
\n
default<\/code> queue \u2014 so if you just forgot to name a specific queue, your job class got really lucky. It now had the highest priority in the app, congratulations! We also had a mixture of roles in the cluster, which shuffled these queues in different setups, which looked somewhat random. That often led to unexpected side effects, like a dedicated capacity for low priority queues that ran jobs even during high load.<\/p>\n\n\n<\/a><\/figure>\n<\/div>\n\n\n<\/a><\/figure>\n<\/div>\n\n\nTypical incident scenario<\/strong><\/h2>\n\n\n\n
<\/a><\/figure>\n<\/div>\n\n\nIntroducing: Kubernetes<\/strong><\/h2>\n\n\n\n
Learning<\/strong><\/h2>\n\n\n\n
<\/a><\/figure>\n<\/div>\n\n\nThinking about users<\/strong><\/h2>\n\n\n\n
<\/a><\/figure>\n<\/div>\n\n\n\n
Finally, the new design<\/strong><\/h2>\n\n\n\n
<\/a><\/figure>\n<\/div>\n\n\n\n
Measurement (SLI)<\/strong><\/h2>\n\n\n\n
<\/a><\/figure>\n<\/div>\n\n\nAutoscaling<\/strong><\/h2>\n\n\n\n
pgbouncer<\/code> pools. But we can scale the processing capacity of Sidekiq, so that we pay for less Sidekiq threads and less node instances on AWS. Our current topology looks like this:<\/p>\n\n\n<\/a><\/figure>\n<\/div>\n\n\nThreads and memory<\/strong><\/h2>\n\n\n\n
<\/a><\/figure>\n<\/div>\n\n\nLong-running jobs<\/strong><\/h2>\n\n\n\n
What about fair scheduling?<\/strong><\/h2>\n\n\n\n
\n
<\/a><\/figure>\n<\/div>\n\n\n
![\"breaking](\"https:\/\/chartmogul.com\/blog\/wp-content\/uploads\/2024\/08\/Not-FIFO.png\")
<\/a><\/figure>\n<\/div>\n\n\n