Reasons to split large Jenkins

From time to time discussions start in (Jenkins Users) group on managing several Jenkins instances simultaneously. And each time there is a suggestion to merge these instances to a single one and leverage views and access control to separate domains.

Well, the single build server instance is easier to manage - to some extent.

It can provide more resources and computation power to its jobs.

But there are several strong reasons not to merge build servers.

• Administrative domain separation. Co-location of stuff belonging to different teams may be just prohibited.
• Performance. Smaller Jenkins servers boot faster. Also Jenkins has quite large memory footprint - 0.5 GB heap size is a minimum requirement for a modest installation. More jobs - more builds, more builds - larger heap. For multi GBytes heap GC delays may become an issue.
• There are also plugin-related issues. Such as Subversion credentials one - SVN plugin cannot log in with different credentials to a single authentication domain.

But if someone decides to run multiple build servers they should be installed at different (virtual) hosts. It is difficult to imagine a case when co-location of several Jenkins masters can be justified.

2 ways to distributed Jenkins

One of the key Jenkins features - build cluster at little to no cost. Ability to run jobs at Linux, Windows, Mac at the same time, ability to scale far above a single computing box. Moreover this ability is implemented in a very easy way to maintain and use.

Default method to build a distributed Jenkins is master-centered. I.e.

• slave node is configured at the master;
• master controls slave lifestyle - starts it (via SSH or DCOM) and stops.

There is a number of plugins supporting this scenario. Libvirt Slaves Plugin, VirtualBox Plugin, vSphere Cloud Plugin to control VM provisions, Slave Setup Plugin to configure the slave node. In an extreme case slaves are deployed to a public cloud (Amazon EC2 Plugin, Delta Cloud API plugin, JClouds Plugin).

Another approach - to assembly a cluster from slave side. Jenkins agents are started by someone at slave hosts, then they connect to a master and form a distributed server. For now only Swarm Plugin plugin does that.

Sure it is possible to start a slave agent simply via JNLP. But the corresponding slave node is to be configured at the master. So this case does not fit.

None of these methods is the best. They have different use cases.

• Master-controlled approach leverages centralized resource provisioning. And the center is Jenkins. It supposes that either slave hosts are dedicated for the Jenkins server or there is plenty of resources.
• Slave-initiated way works when resource control is outside Jenkins, when several Jenkins servers coincide in a computer poll, when some spare resources are to be utilized. Especially useful for heavy testing tasks such as Selenium tests.

In my opinion the centralized configuration is for build engineers servicing single project team. For a multi-project build farm the distributed control suits better.

Missed features in Jenkins?

Sure Jenkins is the leading continuous integration server software. Competition with it is as difficult as with Linux in general purpose server platform area. Jenkins is for general purpose too and it is the cause of some functionality omissions:

• Weak support for pipelines. I.e. build-test-QA-release, staged deployment.
• No support for parallel coding - feature branches, server side integration, pre-commit builds.
• Difficult mass configuration - shared SCM configuration, credentials, job configuration inheritance and parametrization.
• Weak traceability of user operations, no support for audit.

Well, we all implementing that yourself. There are a lot of helper plugins, but such solutions are patchy and copy-paste based.

I would argue that these functionality omissions are not really disadvantages of Jenkins. It is too general tool for such options. It would be right to treat Jenkins as a foundation for higher level solutions. A platform, see a note by Kohsuke himself (Kohsuke Kawaguchi: Writing programs that drive Jenkins). Because Jenkins is

• quite simple;
• really flexible;
• easily extensible;
• having a set of versatile APIs.

It is difficult to guess how many such uber-applications are being developed and used internally. We have one too. Next publicly available solutions should emerge - such as CloudBees Jenkins as a service, GitHub Janky. Hope these solutions will fuel Jenkins developers financially given a good balance between open source platform and custom applications will be maintained.