Quality...

Once upon a time I worked as a production manager for a jewelry manufacture. Sure the production quality was a key there. The issue was much more interesting because jewelry is not mechanics or electronics. It is a piece of beauty. And its quality cannot be (fully) expressed in numbers.

The company I was working for had a pretty high quality attitude. So I had enough occasions to explain to a worker that her work wouldn't pass. And why it wouldn't. And often the explanation failed just because (see above) the quality could not be fully expressed in measurable numbers. So finally I usually asked: "Would you wear such piece of adornment?" (most of jewelry workers are women). In most cases no further discussion was required. Because every woman has a "built in" sense of beauty.

Now I'm back in my beloved software industry. This time not as a developer but as a Continuous Integration man. I establish a lot of measures that should be passed before a code is considered being "done". But I still did not find such a final question that should distinguish a good code from another stuff. In a high and unequivocal way that should be clear for developers, testers and users.

"Will you use this software for yourself?" - "Hm, it suits for me."
"Will you let this software to control your heart simulator?" - "It won't control my heart simulator."

There is a few of software companies owning and using their products and enjoying rapid development cycle - such as online services and AntiVirus developers. Their engineers can gain a fame of success and a shame of failure. An excellent example is Facebook. Each Facebook engineer owns his new feature till it is published to production world wide. Moreover to much extent it is an engineer decision to push a feature to production. And starting from it the feature is his born baby.

Software development for sale separates an engineer from the code. In a case of outsourcing a subcontractor separation becomes a gulf.

So the question is: What are those Ten Commandments, that Dao, that sense of beauty to force a software developer to understand the difference between a good code and... a different thing?

Reasons to split large Jenkins

From time to time discussions start in (Jenkins Users) group on managing several Jenkins instances simultaneously. And each time there is a suggestion to merge these instances to a single one and leverage views and access control to separate domains.

Well, the single build server instance is easier to manage - to some extent.

It can provide more resources and computation power to its jobs.

But there are several strong reasons not to merge build servers.

• Administrative domain separation. Co-location of stuff belonging to different teams may be just prohibited.
• Performance. Smaller Jenkins servers boot faster. Also Jenkins has quite large memory footprint - 0.5 GB heap size is a minimum requirement for a modest installation. More jobs - more builds, more builds - larger heap. For multi GBytes heap GC delays may become an issue.
• There are also plugin-related issues. Such as Subversion credentials one - SVN plugin cannot log in with different credentials to a single authentication domain.

But if someone decides to run multiple build servers they should be installed at different (virtual) hosts. It is difficult to imagine a case when co-location of several Jenkins masters can be justified.

2 ways to distributed Jenkins

One of the key Jenkins features - build cluster at little to no cost. Ability to run jobs at Linux, Windows, Mac at the same time, ability to scale far above a single computing box. Moreover this ability is implemented in a very easy way to maintain and use.

Default method to build a distributed Jenkins is master-centered. I.e.

• slave node is configured at the master;
• master controls slave lifestyle - starts it (via SSH or DCOM) and stops.

There is a number of plugins supporting this scenario. Libvirt Slaves Plugin, VirtualBox Plugin, vSphere Cloud Plugin to control VM provisions, Slave Setup Plugin to configure the slave node. In an extreme case slaves are deployed to a public cloud (Amazon EC2 Plugin, Delta Cloud API plugin, JClouds Plugin).

Another approach - to assembly a cluster from slave side. Jenkins agents are started by someone at slave hosts, then they connect to a master and form a distributed server. For now only Swarm Plugin plugin does that.

Sure it is possible to start a slave agent simply via JNLP. But the corresponding slave node is to be configured at the master. So this case does not fit.

None of these methods is the best. They have different use cases.

• Master-controlled approach leverages centralized resource provisioning. And the center is Jenkins. It supposes that either slave hosts are dedicated for the Jenkins server or there is plenty of resources.
• Slave-initiated way works when resource control is outside Jenkins, when several Jenkins servers coincide in a computer poll, when some spare resources are to be utilized. Especially useful for heavy testing tasks such as Selenium tests.

In my opinion the centralized configuration is for build engineers servicing single project team. For a multi-project build farm the distributed control suits better.

Missed features in Jenkins?

Sure Jenkins is the leading continuous integration server software. Competition with it is as difficult as with Linux in general purpose server platform area. Jenkins is for general purpose too and it is the cause of some functionality omissions:

• Weak support for pipelines. I.e. build-test-QA-release, staged deployment.
• No support for parallel coding - feature branches, server side integration, pre-commit builds.
• Difficult mass configuration - shared SCM configuration, credentials, job configuration inheritance and parametrization.
• Weak traceability of user operations, no support for audit.

Well, we all implementing that yourself. There are a lot of helper plugins, but such solutions are patchy and copy-paste based.

I would argue that these functionality omissions are not really disadvantages of Jenkins. It is too general tool for such options. It would be right to treat Jenkins as a foundation for higher level solutions. A platform, see a note by Kohsuke himself (Kohsuke Kawaguchi: Writing programs that drive Jenkins). Because Jenkins is

• quite simple;
• really flexible;
• easily extensible;
• having a set of versatile APIs.

It is difficult to guess how many such uber-applications are being developed and used internally. We have one too. Next publicly available solutions should emerge - such as CloudBees Jenkins as a service, GitHub Janky. Hope these solutions will fuel Jenkins developers financially given a good balance between open source platform and custom applications will be maintained.

Code validation advocacy

It was said many times that static code validation sucks, validation warnings are false and senseless.

Well, it is a common mood despite numerous words to support code validation. So just my 2 cents.

What code validation tools do

So what is the code validation about - starting from most simple topics. Some are not covered - such as API analysis.

1. Copy-paste
   

   Detects chunks of duplicated code. It is bad, obviously.

2. Formatting
   

   Checks if the source code follow certain formatting rules.
   Well, these rules may differ from Sun, Microsoft or Google ones. That doesn't matter. The only things that matters is that the rules must be the same. For the whole project. For all projects around. Soon you will get used to them. And the project code will be like a book printed with a good font but not a hand-written notes on a napkin.
   And we don't write in Brainfuck, right?

3. Bad words
   

   Checks for certain typical words in comments - such as TODO, FIXME, DEBUG, etc.
   Imagine - you are a Big Company and your contractor sent you a piece of code. In the middle of the code:

   /* TODO: implement it according to the spec */

   What will you say? "WTF, what did they send?!"
   Put TODOs to your code in development. But clean them out prior to submission to a repository. Not just delete the comment but solve its cause. Leave the code in a stable state. If there is a problem - create a task in an issue tracker.
   Finally we are coders. Our product - those text files that we submit to the repository. Those lines of text brings us our money. They must shine.

4. Documentation
   

   Checks that all relevant code items have inline documentation.
   Sure good inline documentation is the must. Lack of proper inline docs leads to misuse, time waste and re-invention the wheel. And don't bother to choose what should be documented and what should not. You are not a prophet and cannot predict how and when this code will be used. So just document everything.
   Checks for legal stuff - such as license headers. Remember - nowadays is the era of lawyers. So don't feed the trolls.

5. Anti-patterns
   

   Looks for coding patterns that are either bad practice or directly dangerous.
   Most developers can argue that in this specific case it won't cause problems. Nobody can predict all execution flows. Nobody can imagine all possible uses in the future. All these anti-patterns may cause problems. So take for granted that if they can cause problems they will - according to the Murphy law.

How do code validation tools affect developers work?

• Difficulties?
  

  Everyone has habits. Everyone has own taste for formatting. OK, IDE will do it for you.
  Everyone has own code tricks. That is good sometimes - for sole coding. For team work it is not acceptable. We are speaking the code to our colleagues and need to understand and to be understood.
  It takes precious time. But bugs takes more. Bugs by you, by users of your code. Maintenance takes even more.

• Profit!
  

  It really helps to avoid some bugs.
  It makes more transparent and easier to understand for teammates - so developers can switch faster and accustom better with a new code. They will use the existing functionality better and make less bugs.
  It makes code look more clear and professional. That means - more valuable. That means higher customer satisfaction.
  The more professional is code look, the less tension it will cause with the customer-side reviewers - they are seldom benevolent.

What should we do?

Just follow the code validation rules. Always. But switch brain on and do not invent constructions to just shut the validation rule up.

It is not difficult, is it?