Jenkins multiple masters using Consul
2019年11月26日に投稿

Consul is HashiCorp’s service networking solution for naming service. In this post, we will show how to use consul to design serverless Jenkins clusters.

For small or Enterprise teams. there are many DevOps tools to choose.

Small TeamsEnterprise TeamsPrice
SAASCircleCI, Azure DevOps, Github ActionDepends on the security level.Pay as pipeline time
Self-hostedsingle free Jenkins instance is enough.Buy enterprise license of SAAS/Jenkins, or develop from OSS version of Jenkins.Infrastructure/Software License/customization

This post is for readers

  • Who are interested in Jenkins’s details.
  • Who are in a large team looking for a self-hosting open source solution, and have the ability and time to customize the Jenkins.
  • This post contains high-level thoughts only, the implementation may cost 2~5 man-months, and there is no open-source product available.

Liebig’s law of Jenkins

For historical reasons, Jenkins uses a memory-based solution to maintain the running job status.

Running statusInternalProblems
ConfigurationXML file basedNot too bad
QueueArrayListlinear complexity, may be lost when offline.
AgentConcurrentSkipListMaplog(n) complexity, may slow when too much.
Running JobsBind to agentreal-time scheduling/log stress.

Existing multiple masters solutions

To overcome the problem, there are some solutions already.

  • Binlog-like solution: Using SCM Sync configuration plugin or Shared NFS to replicate settings to another master. However, it is just a hot standby failover solution, only one Jenkins instance is working.
  • Global lock solution: if you want a High Availability solution to share context to other masters, you need to implement a centralized lock(Database/RAFT) to notify changes to the others before persist, that means you may need a team to modify and maintain the core source code of Jenkins.
  • Gearman based solution: This is the real multiple masters’ solution, the masters are ‘Jobs’ in Gearman. However the repository is unmaintained for near 5 years, and it’s cli is more like an Ansible solution.

Serverless

What is serverless?

Serverless is stateless. In Jenkins, that means your Jenkins instance is only a jenkinsfile runner, which has nothing to bind with the local database(in JENKINS_HOME).

What’s Jenkins-X serverless solution?

The jenkins-X solution is based on kubernetes

  • You need to buy, install, and maintain the kubernetes clusters.
  • Too complex and no general purpose, Using CD only with kubernates/git.
  • It’s hard to customize own steps.
  • JVM memory saves, but k8s creates new problems.

My RCU(Read-copy-update) Solution

Here is my RCU solution, every masters are independence, what you need is get eventually consistency with low latency.

Networking

Using all opensource components.

Controller

Client

  • Create and send raw XML to the loadbalancer
  • Inside XML, the payload can be Jenkinsfile, YAML, JSON, and other DSL.

Data lake

  • ELK/KsqlDB: ETL and visualization.

Jenkins

Modification of jenkins.war and other plugins are not required. PAAS based naming services(k8s/nomad) are also not required.

Just create a jenkins plugin(not open sourced)

  • Use PeriodicWork to register self to the Consul
  • Use hudson.model.listeners.RunListener to send back job results(RUNNING/SUCCESS/FAIL).
  • Use shared library to intercepting the high-level DSL

Consul

  • Use consul for naming service and health check.
  • Use Fabio for load balancing.
  • Use node deregister when releasing or upgrating.

Why not create and destroy Jenkins container instance per job?

  • JVM cold starts is too slow when you have lots of jobs/plugins(30s or more).
  • The main cost is infra(machines/energy) when self-hosting, and Jenkins masters always have a full workload.

Conclusion

Pros and cons

Single JenkinsPaid SolutionsJenkins based RCU Solution
CostsFree, no guaranteesLicense feesCustomization costs and time
Parallels jobsLimitedBy license type10K Jobs+
HANoYesYes(Flight jobs still down)
Custom DSLwith shared libraryYAMLSame as Jenkins
TenantNoYesCustomized by controller(Spring)
AuthorizationSimpleYesCustomized by controller(RBAC/LDAP…)

So, what you will get depends on how much you pay. This post proposes a new way to implement mutilple masters, but requires a lot of works.