Automate Jenkins job creation with DSL

28 02 2016

Jenkins job creation in Web UI is simple but still – manual creation takes time. Depending on your coding and testing practices you may find it acceptable or burdensome and costly.

Jenkins DSL allows you to define Jenkins job in groovy script file, put it on VCS and allow Jenkins to create and execute jobs based on it. The another advantage is that you get your jobs versioned along with codebase.

So… is that hard? Having Jenkins configured to build your code it requires few steps to build jobs from DSL:

1. Install Jenkins Job DSL plugin

Simple thing to begin: go to your Jenkins’ Management page, find the Job DSL plugin and install it (https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin).

2.Create the Seed Job

That is the job that will create other jobs. Do it like any other job creation: Select the ‘New Item‘, give it a name (in my case ‘dsl-demo’) and select type of ‘Build a free-style software project‘ (or ‘Freestyle project’).

Remember to configure the Code Repository url and credentials where this job will do the checkout and find the DSL script files.

3. Configure DSL scripts files location

Then in your job configuration add build step named: ‘Process Job DSLs’ (provided by Job DSL plugin). I am going to use the script files located in the filesystem (Jenkins’ workspace), versioned in my git repo. All of them will be located under the Jenkins/jobs/ directory in my VCS with the .groovy extension. So the path in my config is:

Jenkins/jobs/*.groovy

Where I have one file as for now, named:

Jenkins/jobs/demoJob.groovy

Remember to follow naming conventions for files: script names may only contain letters, digits and underscores, but may not start with a digit

4. DSL Script file content

First of all to simply make Jenkins create the job it is sufficient to have such content in DSL script:

job('simpliest-job-ever') {
    steps {
        shell('echo hello from your automated job!')
    }
}

Here I am creating the job with a name ‘simpliest-job-ever’ with one step inside. The step is using shell plugin and executing the echo command.

5. Run Seed Job

Now is the time to push the DSL file to the git server and trigger the Seed Job execution. The effect found in Jenkins console after running my Seed job was:

Processing DSL script demoJob.groovy
Added items:
    GeneratedJob{name='simpliest-job-ever'}

So my job was created from DSL file.

6. Run newly created Job

Now, coming back to the Jenkins’ job list the new job actually is there. I did not set the triggers on it yet so I have to fire it manually.

Here is the result of executing my newly generated job, printed in Jenkins console:

Started by user LooksOK!
Building in workspace /var/lib/jenkins/jobs/simpliest-job-ever/workspace
+ echo hello from your automated 'job!'
hello from your automated job!
Finished: SUCCESS

Lovely, isn’t it? :>

Enhance your DSL script

Such a basic configuration and script file prooves that the whole thing works. Now is the time to enhance the scripts and make them creating the fully useful jobs. The plugins supported in DSL and their full documentation is here: https://github.com/jenkinsci/job-dsl-plugin/wiki/Job-DSL-Commands.

A list of most useful plugins is:

    • SCM to connect your Job with code repository
scm {
  git('git@github.com:yacekmm/looksok.git')
}
    • Triggers to setup the schedule on your job execution
triggers {
  scm('*/15 * * * *')
}
    • maven to execute maven tasks
maven('-e clean test')
    • shell to execute shell commands
shell('echo hello from your automated job!')
  • and more…

Thanks for reading, give it a try!

 

Advertisements




Migrate Mercurial hg repository to git keeping the whole history

5 12 2015

Although hg and git share the same idea of versioning files, git provides much more tools to help teams to maintain code, releases and fixes. That’s why I decided to migrate project repository from hg to git. The method I describe preserves the whole hg history, branches and tags.

Migration with fast-export tool

Fast-export is tool that processes each hg commit and migrates it to git. Tool has the python version that you can run on windows, but it didn’t work for me so i used .sh script on linux on VirtualBox. Steps to migrate are:

  1. clone the fast-export git repo
  2. init your target git repo in path of your choice
  3. run the hg-fast-export.sh script and wait until it finished (migrating 4k commits takes around 20 minutes)
  4. start using your new git repo (checkout HEAD, add remote, push)

In other words execute those commands:

git clone git://repo.or.cz/fast-export.git
git init your_git_repo
cd your_git_repo
~/fast-export/hg-fast-export.sh -r /path_to_old_mercurial_repo
git checkout HEAD

Which will be master branch?

As a default behaviour your default mercurial branch will become master in git.

Filenames containing special characters

If you have filenames with special regional characters (like ą, ó, €) make sure to rename them in your hg repo before migration since, depending on your OS, they may migrate to even more strange chars and you may have problems deleting them from your new git repo.

Nested .git repos in your mercurial codebase

If you have commited the whole git repo and versioned it on mercurial you will not be able to push to the remote git repo after migration. It will fail with error that your new git repo already contains nested .git repos:

remote: error: object a9ee490ac00987835de30bdbc851da5e8d45d28b:contains '.git'

You need to clean up any .git folders from your hg before running migration scripts. Fortunately there is a tool to do it: BFG repo cleaner. Download the bfg.jar and issue the command to delete all .git files from hg repo:

java -jar bfg.jar --delete-folders .git --delete-files .git --no-blob-protection my-repo.git

After the migration and solving all problems described above my new git repo works like a charm!





‘Git Happens’ by Jessica Kerr

10 10 2015

A perfect git introduction video for git beginners:

The most important git concepts are explained. That will help you to understand the way how merge, rebase or branching works.

Definetely a must watch – even if you are already a git user not understanding what actually is happening under the hood. That’s not a magic – it is well defined flow and idea behind it








%d bloggers like this: