Ktor Webserver on Docker Swarm, with CI/CD
My Ampere Server is finally finished. I actually have a powerful server now. Well technically, I've always had the Ryzen 9900x "server", but firstly it's running Windows, and secondly it's kind of designed to be my cloud gaming server, not for actual "work" 😄. MH:Wilds only further confirmed that this rig is going to just be gaming.
Alright, back to topic.
The plan is simple. I prefer the JVM stack and JAVA/Kotlin as my primary language of choice for the backend. It's just what I'm familiar with, and I know how powerful and easy the Ktor library is to work with. So I'm going all-in on the Intellij ecosystem, (also because I'm already paying for an Intellij Ultimate license).
This means that my development environment will consist of:
- Intellij as my IDE
- Self-hosted Jetbrains Teamcity as a CI/CD pipeline
- Utilize both my Windows PC and the ARM server as build nodes
- Portainer as a deployment target through the use of its exposed APIs (I assume this should work?)
- Private docker image registry
Setting up a remote development environment was really easy. On my laptop, I opened IntelliJ and started a Remote Development project using the SSH Connection, which automatically installed IntelliJ and dependencies on the server.
Next I installed the ProxyAI plugin to configure a coding assistant to use my local server running Gemma 3.
After a little bit of trial and error, the configuration was complete. I'm not a huge fan of coding assistants, but I am now at peace that my LLM server (ehem, MH:Wilds machine) is integrated to one more thing.
Next was setting up a local docker image registry. I put together a compose file based on some examples online to deploy as a swarm stack:
services:
registry:
image: registry:latest
environment:
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
REGISTRY_AUTH_HTPASSWD_PATH: /.password
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
deploy:
placement:
constraints:
- node.labels.portainer == true
volumes:
- ./mount/registry/.password:/.password
- ./mount/registry/data:/data
ports:
- 5000:5000I was able to verify the registry:
$ docker login ampere:5000
$ docker pull ubuntu
$ docker tag ubuntu ampere:5000/ubuntu
$ docker push localhost:5000/ubuntuNext was setting up Teamcity, which will be my CI/CD mechanism.
I was trying very hard to get teamcity to run through Docker, carefully following their installation instructions multiple times, trying both in Swarm mode and non-swarm mode, but the startup web page would just not load, no matter what configuration I tried. I just absolutely could not get it to work.
Instead I went to install bare-metal, which I was fine with because my Ampere machine was designed to be a dedicated teamcity server anyways.
And while following their last few steps, I was getting some weird error about Java not found, when I clearly had both 17 and 21 installed.
teamcity@ampere:/opt$ TeamCity/bin/runAll.sh start
Spawning TeamCity restarter in separate process
TeamCity restarter running with PID 95183
Starting TeamCity build agent...
Java executable of version 1.8 is not found:
- Java executable is not found under the specified directories: '', '', '/opt/TeamCity/buildAgent/bin/../jre', '/opt/TeamCity/buildAgent/bin/../../jre'
- Neither the JAVA_HOME nor the JRE_HOME environment variable is defined
- Java executable is not found in the default locations
- Java executable is not found in the directories listed in the PATH environment variable
Please make sure either JAVA_HOME or JRE_HOME environment variable is defined and is pointing to the root directory of the valid Java (JRE) installation
Please note that all Java versions starting from 12 were skipped because stable operation on these Java versions is not guaranteed
Environment variable FJ_DEBUG can be set to enable debug output
Java not found. Cannot start TeamCity agent. Please ensure JDK or JRE is installed and JAVA_HOME environment variable points to it.Well dang it, nothing is easy, huh?
One thing I noticed was Java executable of version 1.8 is not found:. I knew this couldn't be right because all around the industry (including my company), Java 8 is being deprecated and there's no way Jetbrains would be requiring JAVA 8 when they are supposed to be the new hip JAVA experts.
Then I realized I didn't properly read the instructions and just copy/pasted verbatim:
# Switch to the /opt directory, where TeamCity will be unpacked
root@ubuntu:/# cd /opt
# Download the tar.gz file using wget
root@ubuntu:/opt# wget https://download.jetbrains.com/teamcity/TeamCity-2022.10.1.tar.gzAnd here, I saw that the documentation's example code was written to run a 2022 version of TeamCity. I went ahead and cleaned up the workspace, and then downloaded the latest version from their site.
wget https://download.jetbrains.com/teamcity/TeamCity-2025.03.tar.gzAnd finally I was able to get it to run:
teamcity@ampere:/opt$ TeamCity/bin/runAll.sh start
Spawning TeamCity restarter in separate process
TeamCity restarter running with PID 101653
Starting TeamCity build agent...
Java executable is found: '/usr/lib/jvm/java-17-amazon-corretto/bin/java'
Starting TeamCity Build Agent Launcher...
Agent home directory is /opt/TeamCity/buildAgent
Done [102123], see log at /opt/TeamCity/buildAgent/logs/teamcity-agent.log
And we're in!
To test out the end to end setup, I first created a new Ktor webserver sample package through the IntelliJ wizard.
And I've pushed the code as-is to a private GitHub repo. (Again, I have no real plans to use a self-hosted git repo)
Next, setup a connection from Teamcity to Github:
And now I created a new project off of the sample project.
I fired off a build, and build seems to have succeeded, but Artifacts were empty. Saw that I didn't configure the artifacts directory, which I went ahead and updated:
And with that, the artifacts are showing up.
As a quick test, I ran this jar directly on the build agent to make sure that it is able to run:
Next was to generate and deploy a docker image. To do that, I first added a configuration in build.gradle.kts, but do note that a Ktor webserver template generated by the Intellij wizard already provides mostly everything you need to get it to work.
ktor {
docker {
jreVersion.set(JavaVersion.VERSION_17)
localImageName.set("ktor-sample")
}
}Second I added a build step that calls the publishImageToLocalRegistry gradle task on the teamcity UI.
$ sudo docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
ktor-sample latest 04cbb27ed776 55 years ago 280MBAfter firing another build from TC, I was able to see that the ktor-sample image was loaded into the Local Registry by checking the build host.
Then to deploy this to my self-hosted registry, which is the one that is hooked up to Portainer, I added an additional build step:
And after rerunning the build (I also had to grant users root permissions to call docker), I was able to verify that the new step was able to push the image to a private repo.
Step 5/5: Docker Push (Docker)
02:51:09 Starting: /bin/sh -c docker push ****/ktor-sample
02:51:09 in directory: /opt/TeamCity/buildAgent/work/fef37c3f1011f649
02:51:09 Using default tag: latest
02:51:09 The push refers to repository [****/ktor-sample]
02:51:10 latest: digest: sha256:7750d51e8374bb50ae8c07282f511ab09ba3418b244a61c0c3a564359e382eb6 size: 2202
02:51:10 Process exited with code 0
02:51:10 Removing pushed images/tags from build agent: ****/ktor-sampleKtor and the default configuration was able to solve 90% of the workspace setup, and in TeamCity, I really only needed to configure how to build a linear pipeline, making this setup super simple.
One last thing to do is launching a container with an image of this dummy service, which for now will be done manually through Portainer.
And errr....
Checking the image, it seems like Ktor built an amd64 image which my Raspberry Pis cannot run:
"architecture":"amd64"Looking through the source, Ktor is using Jib to build the images. I'm not familiar with Jib, but I gave a shot of configuring Jib in my build.gradle in hopes that it would override whatever defaults based on their documentation.
GoogleContainerTools... redacted ...
jib {
from {
platforms {
platform {
architecture = "arm64"
os = "linux"
}
}
}
container {
creationTime = "USE_CURRENT_TIMESTAMP"
}
}
... redacted ...And after unpacking the tar and looking at the config file,
{"created":"2025-03-22T04:01:53.054950569Z","architecture":"arm64","os":"linux" ...Success! I really didn't think this would work, and I was ready to write custom jib integration. Now time to push the change and let the pipeline do its work, and hope for the best.
After restarting the stack in portainer, the service was finally deployed to the raspberry pi cluster, I can dare say that my dev environment end-to-end has been finally set up!
There will be some more digging to do through the documentation:
- Adding integration tests - this one seems actually quite simple to do, a docker-compose build step to run the service, then a gradle step to execute integ tests
- How to automate the portainer deployment. There is an API to start and stop a stack, which may be all I need.
- Artifact dependencies - Assuming I have two projects, one for frontend and one for backend, I need to be able to have the frontend generate assets that will be copied to be included in the super-jar/docker image.
- Once I set up a ZFS pool with high capacity disks, I will need to move the build artifact directory (for both the server and the agent) to utilize the ZFS pool rather than the OS volume.
I'll also be reading some more documentation on Teamcity so that I know what the hell I'm doing, but overall I'm pretty happy with the current state of the development environment.
