I am having a problem with my GitLab pipeline when trying to mount a docker volume inside a container. Before I explain the problem, first I will describe my whole setup, because I think that is very essential to understand the problem, because I think that this is the reason why I am having this problem.
Setup
Okay, so to start off, I have a kubernetes cluster. This cluster runs my gitlab/gitlab-ee:15.8.0-ee.0 image. I installed a GitLab runner in this cluster as well, so that I am able to run pipelines of course. Then the last thing I installed is a docker instance, because I saw that you can mount the docker.sock from your host machine to the gitlab pipeline, but this is not recommended, because the entire cluster relies on that docker.sock, so I have another instance of docker running and I am mounting that docker.sock for pipelines only. These 3 deployments are used by me to run GitLab pipelines.
The problem
I am happy with the way everything is setup, but I think I am still missing some configuration, because the mounting of docker volumes are not working properly in pipelines. I have this script to test this, which contains this code:
image: docker:20.10.16-dind
variables:
DOCKER_HOST: "tcp://docker-service:2375" # <-- Address to reach the docker instance from my cluster
DOCKER_COMPOSE_CMD: "docker-compose -f docker-compose-test.yml"
stages:
- test
test:
stage: test
script:
- $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
- $DOCKER_COMPOSE_CMD build
- $DOCKER_COMPOSE_CMD --env-file .env.pipeline up -d
- $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test sh -c "ls"
With the following docker-compose-test.yml:
version: '3.7'
services:
laravel-api-test:
build:
context: .
dockerfile: docker/development/Dockerfile
volumes:
- .:/var/www/html
environment:
- COMPOSER_MEMORY_LIMIT=-1
depends_on:
- database-test
database-test:
image: postgres:15.1-alpine
ports:
- ${DB_PORT}:5432
environment:
POSTGRES_DB: ${DB_DATABASE}
POSTGRES_PASSWORD: ${DB_PASSWORD_SECRET}
POSTGRES_USER: ${DB_USERNAME_SECRET}
redis-test:
image: redis:7.0.8
ports:
- ${REDIS_PORT}:6379
networks:
default:
name: application
Now what this pipeline does, it builds the docker containers and then starts them. Then it runs the ls command which prints out all the files in the working-dir of the container. However, this working-dir is empty. This is caused by the volume mount in the docker-compose-test.yml with this line:
volumes:
- .:/var/www/html
In the Dockerfile I also have this:
COPY . /var/www/html/
So when I remove the volume mount in docker-compose-test.yml, all files are there, so the copying does work for the Dockerfile, but not mounting it later on. I saw this thread and tried some of their solutions and tested it with their test script:
variables:
SHARED_PATH: /builds/shared/$CI_PROJECT_PATH
script:
- mkdir -p ${SHARED_PATH}
- touch ${SHARED_PATH}/test_file
- docker run -v ${SHARED_PATH}:/mnt ubuntu ls /mnt
But this still resulted in an empty /mnt dir, while the test_file should have been there. In the GitLab runner I added this line to the config:
volumes = ["/cache", "/builds:/builds"]
Unfortunately, this did not change anything. I am not sure, but my guess is that I need to access the /builds from my other docker instance, because I have a feeling that I am mounting the /builds from the host machine, which is not the docker I am using in my pipeline. If this is the case, I am not sure how to configure my Kubernetes cluster to use the other one. The weird thing is that when
I do cd /builds/projects/laravel-api (my repo is named laravel-api and its inside the projects group) and then ls in my pipeline, I do see my repository containing all the files. But when I try to mount that directory in my docker-compose-test.yml I still get an empty dir. So I mean this:
volumes:
- /builds/projects/laravel-api:/var/www/html
So every way of mounting volumes after builds are resulting in empty directories…
Wrap up
So to summarize the problem. Every form of mounting I do in my pipeline results eventually in an empty directory. When copying files from a Dockerfile only the directory does work, but that is not something I can work with.
I hope this covers the entire problem. Some help is really appreciated! If there are any questions about the setup or something like that, please ask I will respond ASAP!