Accessing Private Go Packages in CI/CD Pipelines

Go's default package repository is Git. This means any Go code you publish to a Git repository such as Bitbucket, Github, or GitLab is a module. When you use the Go command: go get package, under the hood Go package manager, Go Modules, runs a git clone command to pull down the package. Now, this is pretty straightforward when dealing with public repositories. But when dealing with a private repository, the server that needs to pull the Go package would need the right permissions. If you have access rights to the repository where the Go package is published, you may be able to access it easily from your local machine with your personal credentials using your username and passwords or with SSH if this has been set up. It's a different story when the package needs to be accessed by an automation tool or server like your CI/CD server.

Many pipeline servers once set up correctly have access to the repository they need to pull code from. However, the pipeline access may not allow access to private Go packages in the same repository when used as dependencies in the code that needs to be built in the pipeline. One option is to use access tokens.

SSH keys make it less cumbersome. For the purpose of brevity, I will focus on setting these up for your bitbucket pipelines. However, the same principle applies to most platforms.

Using SSH Keys in Bitbucket Pipelines

Although Bitbucket Pipelines is an integrated Bitbucket cloud service, it does not have automatic access to your Bitbucket Go packages when it needs to download dependencies and build your code. Here are the steps to give your bitbucket pipeline access:

1. From the target pipeline repository, go to Repository Settings and under Pipelines, select SSH keys. You can choose 'Generate keys' or 'Use my own keys'. If you choose the first option, this will generate a Private key that’s secured so you can’t access it - well, you won't need to. This is stored in the Bitbucket pipeline server. What you’ll need is the public key, so copy that.
   
2. Now go to the source repository where the private Go package needs to be pulled from. Go to Repository settings and select Access keys under General.
   
3. On the Access keys page, click Add key, give your key a name that identifies the (machine) user, in this case, your bitbucket pipeline. Then paste in the pipeline SSH public key that you copied in step 1 from your target repository.

Now the access permissions are set up for your pipeline to run, you just need to set Go Modules environment variable to your repository organisation name and set the git config for the pipeline server as follows:

Building a Docker image in the Pipeline

Whilst the pipeline host now has access rights to the private repository, building a docker image in the pipeline would fail because docker doesn’t yet have the right permissions. It needs to identify itself with the SSH key the pipeline server has set up with the private repository in order to access the private Go package.

All that’s needed as part of your Dockerfile is to add commands for docker to access the SSH keys on the pipeline host by adding the following steps before running your go build command:

NOTE: Please DO use a multi-stage build if adding SSH keys to Docker so that it does not end up as part of your docker image for security reasons. (See multi-stage Docker here).

The Dockerfile snippet above creates an environment variable SSH_PRIVATE_KEY for the private key on the pipeline host machine in the build stage of the Dockerfile. The value for the variable is expected to be supplied to docker during build time with the flag --build-arg. The snippet also creates an SSH directory where the private key from the host is added. This way, when docker tries to pull from the private repository, it will have access as it essentially has the same identity as the host machine that has been registered with the private repository. Bitbucket Pipeline SSH config directory is located at: /opt/atlassian/pipelines/agent/ssh/id_rsa and can be safely accessed and assigned to a variable. For example:

I personally had a hard time wrapping my head around how authentication works for Bitbucket Pipelines. So this is an attempt to document it somewhere. I may write up one for Github for, you know...series 😀