Consul is HashiCorp’s service networking solution for naming service. In this post, we will show how to use consul to design serverless Jenkins clusters.
For small or Enterprise teams. there are many DevOps tools to choose.
|Small Teams||Enterprise Teams||Price|
|SAAS||CircleCI, Azure DevOps, Github Action||Depends on the security level.||Pay as pipeline time|
|Self-hosted||single free Jenkins instance is enough.||Buy enterprise license of SAAS/Jenkins, or develop from OSS version of Jenkins.||Infrastructure/Software License/customization|
This post is for readers
- Who are interested in Jenkins’s details.
- Who are in a large team looking for a self-hosting open source solution, and have the ability and time to customize the Jenkins.
- This post contains high-level thoughts only, the implementation may cost 2～5 man-months, and there is no open-source product available.
Liebig’s law of Jenkins
For historical reasons, Jenkins uses a memory-based solution to maintain the running job status.
|Configuration||XML file based||Not too bad|
|Queue||ArrayList||linear complexity, may be lost when offline.|
|Agent||ConcurrentSkipListMap||log(n) complexity, may slow when too much.|
|Running Jobs||Bind to agent||real-time scheduling/log stress.|
Existing multiple masters solutions
To overcome the problem, there are some solutions already.
- Binlog-like solution: Using
SCM Sync configuration pluginor Shared NFS to replicate settings to another master. However, it is just a hot standby failover solution, only one Jenkins instance is working.
- Global lock solution: if you want a High Availability solution to share context to other masters, you need to implement a centralized lock(Database/RAFT) to notify changes to the others before persist, that means you may need a team to modify and maintain the core source code of Jenkins.
- Gearman based solution: This is the real multiple masters’ solution, the masters are ‘Jobs’ in Gearman. However the repository is unmaintained for near 5 years, and it’s cli is more like an Ansible solution.
What is serverless?
Serverless is stateless. In Jenkins, that means your Jenkins instance is only a jenkinsfile runner, which has nothing to bind with the local database(in JENKINS_HOME).
What’s Jenkins-X serverless solution?
The jenkins-X solution is based on kubernetes
- You need to buy, install, and maintain the kubernetes clusters.
- Too complex and no general purpose, Using CD only with kubernates/git.
- It’s hard to customize own steps.
- JVM memory saves, but k8s creates new problems.
My RCU(Read-copy-update) Solution
Here is my RCU solution, every masters are independence, what you need is get eventually consistency with low latency.
Using all opensource components.
- Create and send raw XML to the loadbalancer
- Inside XML, the payload can be Jenkinsfile, YAML, JSON, and other DSL.
- ELK/KsqlDB: ETL and visualization.
Modification of jenkins.war and other plugins are not required. PAAS based naming services(k8s/nomad) are also not required.
Just create a jenkins plugin(not open sourced)
PeriodicWorkto register self to the Consul
hudson.model.listeners.RunListenerto send back job results(RUNNING/SUCCESS/FAIL).
- Use shared library to intercepting the high-level DSL
- Use consul for naming service and health check.
- Use Fabio for load balancing.
- Use node deregister when releasing or upgrating.
Why not create and destroy Jenkins container instance per job?
- JVM cold starts is too slow when you have lots of jobs/plugins(30s or more).
- The main cost is infra(machines/energy) when self-hosting, and Jenkins masters always have a full workload.
Pros and cons
|Single Jenkins||Paid Solutions||Jenkins based RCU Solution|
|Costs||Free, no guarantees||License fees||Customization costs and time|
|Parallels jobs||Limited||By license type||10K Jobs+|
|HA||No||Yes||Yes(Flight jobs still down)|
|Custom DSL||with shared library||YAML||Same as Jenkins|
|Tenant||No||Yes||Customized by controller(Spring)|
|Authorization||Simple||Yes||Customized by controller(RBAC/LDAP…)|
So, what you will get depends on how much you pay. This post proposes a new way to implement mutilple masters, but requires a lot of works.