Introduction
Packages are reusable software modules that help users take advantage of work done by other developers. Packages can be related to operating systems, programming languages, or even frameworks. For example, Debian-based operating systems use the .deb format and the Java programming language uses packages in .jar format. Package repositories are locations where packages are hosted. These are generally managed by the developer community or privately by the enterprises that use them.
Package managers are responsible for fetching and managing the installation of packages. They act as the bridge between the system that needs the packages and the place they’re hosted. Debian’s APT and Python’s PIP are examples of package managers. In modern CI/CD-based software development, it is imperative that your CI/CD is integrated with package managers for seamless build process and rapid deployment. This post explains how to accomplish this integration.
What is the role of package managers in CI/CD?
Continuous integration and continuous deployment help modern organizations deliver quality software at speed. Continuous integration is about having a fully integrated product after every successful merge request. In other words, the entire code base is fully integrated and expected to work at any point in time. This is accomplished by having automated regression tests connected to the build process. In a continuous integration pipeline, a merge request is always submitted with complete regression results, eliminating the possibility of any issues once it is integrated.
Continuous deployment takes this to the next step, where immediately after the pull request is merged and integrated, a build of the integrated package is taken and deployed to production and staging instances. This helps in rapidly making the new feature set available to the market.
Package managers play critical roles in continuous integration and delivery processes in multiple ways. During the continuous integration process, the code has to be built into an executable to run the automated tests. This requires fetching the required packages from package repositories and bundling them into one executable.
The executables delivered this way may not always be applications that are directly usable by customers. They could also be packages that are meant to be used by developers for building customer-facing applications. In such cases, the delivery pipeline will be delivering them to a package repository from where package managers can pick them up for installation.
In delivery strategies like these, the packages need to be configured in such a way that package managers can search and install them. This means your delivery pipeline must adhere to the packaging format of specific package managers. On another note, if you are looking for a package repository that can handle packages for multiple programming languages and frameworks, check out Packagecloud.
To summarize, package managers are critical parts of your CI/CD pipeline in two ways. For continuous integration and building the executable, package managers are imperative. After the packages are built, they need to be deployed to either a package repository or the runtime environment. Package managers play an important role in this distribution process too. You will see how all this is accomplished in the coming sections.
Integrating Package Managers with CI/CD pipelines
Let us now shift our focus on how we can integrate package managers with CI/CD pipelines. We will try to understand this with the help of Maven - a build and package management tool for Java. Since Maven does almost everything related to building, packaging, and deploying in Java, it has numerous identities. Beyond helping manage packages, Maven also provides a repository where Java packages can be hosted. In this example, we will use Maven with Github Actions.
Github Actions is a continuous integration tool that can integrate with various package managers and repositories. Maven is one among them. Github Actions can initiate a build and deployment process when specific development sequence events happen. For example, you can configure it to trigger this process every time a pull request is merged.
To accomplish this, github-actions.yml has to be configured as below.
name: Java CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up JDK 11
uses: actions/setup-java@v2
with:
java-version: '11'
distribution: 'adopt'
- name: Build with Maven
run: mvn --batch-mode --update-snapshots verify
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Maven Central Repository
uses: actions/setup-java@v2
with:
java-version: '11'
distribution: 'adopt'
server-id: ossrh
server-username: MAVEN_USERNAME
server-password: MAVEN_PASSWORD
- name: Publish package
run: mvn --batch-mode deploy
env:
MAVEN_USERNAME: ${{ secrets.OSSRH_USERNAME }}
MAVEN_PASSWORD: ${{ secrets.OSSRH_TOKEN }}
In the above yml snippet, you can see that there is a build section that uses a Maven command to build the artifacts. There is also a publish section that deploys the artifacts to the Maven public repository. Please note that the credentials for your public repository must be present in the same file.
The next part of the puzzle is to configure the pom.xml in your file so that the executable gets built successfully using external dependencies and gets sent to the Maven package repository. Pom.xml is the file that represents Maven’s project object model to define the configuration for how a project should be built and deployed.
A typical pom.xml configuration that builds the project from the Maven repository and pushes it to a repository which will look like the below section when completed.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.package.yourapp</groupId>
<artifactId>app_name</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>demo</name>
<description>Project for distribution demot</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<distributionManagement>
<repository>
<id>central</id>
<name>83d43b5afeb5-releases</name>
<url>${env.MAVEN_REPO_URL}/libs-release-local</url>
</repository>
</distributionManagement>
</project>
The dependency section lists all external dependencies that are required to build the project. You can also find the distribution management section which mentions the URL of the repository to which the executable should be distributed. Ensure that the settings.xml file in your project includes the credentials for accessing this repository.
And that is how package managers are integrated with a CI/CD pipeline. However, if you are a fast-paced development organization with exposure to multiple technologies, you would typically have to implement the same system for multiple repositories other than Maven. Another option is to use a repository like Packagecloud that supports multiple languages and frameworks. You can check out Packagecloud here. Let us now see how the same process can be done using Packagecloud based repositories.
How Packagecloud can work with CI/CD pipelines
Packagecloud repositories work smoothly with the most commonly used continuous integration and delivery tools. It supports CircleCI, TravisCI, Github Actions, Jenkins, and BuildKite to name a few. The biggest advantages of Packagecloud are that it can handle almost all popular package formats, and that there is no need for separate repositories. Having the ability to use a single package repository for all your build and deploy needs goes a long way in the clean setup of your CI/CD pipeline.
Let us now look at how packages can be pushed to Packagecloud using Github Actions. The below example shows how you can push an RPM package to Packagecloud.
name: Push to packagecloud
on: [push]
jobs:
Push-to-packagecloud:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v2
- name: Upload package PackageCloud.io
uses: danielmundi/upload-packagecloud@v1
with:
PACKAGE-NAME: packagecloud-test-1.1-1.x86_64.rpm
PACKAGECLOUD-USERNAME: username
PACKAGECLOUD-REPO: repository
PACKAGECLOUD-DISTRIB: fedora/34
PACKAGECLOUD-TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
You can see that instead of the Maven repository details the Github Actions configuration now has the Packagecloud details. Packagecloud tokens can be obtained from your Packagecloud admin console.
In case you are not using a separate CI tool and want to publish to Packagecloud directly from your build tool, you can do so by including the Packagecloud plugin for that build tool. For example, in the case of Maven, all you need to do is add the below snippet to your pom.xml.
<build>
<extensions>
<extension>
<groupId>io.packagecloud.maven.wagon</groupId>
<artifactId>maven-packagecloud-wagon</artifactId>
<version>0.0.6</version>
</extension>
</extensions>
</build>
This will install the Packagecloud wagon for Maven and facilitate publishing to Packagecloud repositories. You should also have the correct Packagecloud URL in the distribution section of your pom.xml as below.
<distributionManagement>
<repository>
<id>packagecloud-myrepo</id>
<url>packagecloud+https://packagecloud.io/my-user/myrepo</url>
</repository>
<snapshotRepository>
<id>packagecloud-myrepo</id>
<url>packagecloud+https://packagecloud.io/my-user/myrepo</url>
</snapshotRepository>
</distributionManagement>
Similarly, you can integrate Packagecloud with most other build tools like Gradle, SBT, Lien, etc. Furthermore, instead of Github Actions, you could also use other CI tools like TravisCI, Circle CI, Jenkins, etc.
Conclusion
Integrating package managers into your CI/CD setup is a critical step in accomplishing rapid delivery with quality and reliability. In most engineering organizations using more than one technology and language at the same time, using a package repository that can handle only a single language can cause problematic bottlenecks. This is where Packagecloud can help. It can handle most of the common package formats and ensure an elegant CI/CD setup.
