Skip to content

GitLab

Job logs (FREE SELF)

Renamed from job traces to job logs in GitLab 12.5.

Job logs are sent by a runner while it's processing a job. You can see logs in job pages, pipelines, email notifications, etc.

Data flow

In general, there are two states for job logs: log and archived log. In the following table you can see the phases a log goes through:

Phase State Condition Data flow Stored path
1: patching log When a job is running Runner => Puma => file storage #{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log
2: overwriting log When a job is finished Runner => Puma => file storage #{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log
3: archiving archived log After a job is finished Sidekiq moves log to artifacts folder #{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log
4: uploading archived log After a log is archived Sidekiq moves archived log to object storage (if configured) #{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log

The ROOT_PATH varies per environment. For Omnibus GitLab it would be /var/opt/gitlab, and for installations from source it would be /home/git/gitlab.

Changing the job logs local location

To change the location where the job logs are stored, follow the steps below.

In Omnibus installations:

  1. Edit /etc/gitlab/gitlab.rb and add or amend the following line:

    gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds'

  2. Save the file and reconfigure GitLab for the changes to take effect.

Alternatively, if you have existing job logs you can follow these steps to move the logs to a new location without losing any data.

  1. Pause continuous integration data processing by updating this setting in /etc/gitlab/gitlab.rb. Jobs in progress are not affected, based on how data flow works.

    sidekiq['queue_selector'] = true
sidekiq['queue_groups'] = [
  "feature_category!=continuous_integration"
]

  2. Save the file and reconfigure GitLab for the changes to take effect.

  3. Set the new storage location in /etc/gitlab/gitlab.rb:

    gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds'

  4. Save the file and reconfigure GitLab for the changes to take effect.

  5. Use rsync to move job logs from the current location to the new location:

    sudo rsync -avzh --remove-source-files --ignore-existing --progress /var/opt/gitlab/gitlab-ci/builds/ /mnt/to/gitlab-ci/builds`

    Use --ignore-existing so you don't override new job logs with older versions of the same log.

  6. Resume continuous integration data processing by editing /etc/gitlab/gitlab.rb and removing the sidekiq setting you updated earlier.

  7. Save the file and reconfigure GitLab for the changes to take effect.

  8. Remove the old job logs storage location:

    sudo rm -rf /var/opt/gitlab/gitlab-ci/builds`

In installations from source:

  1. Edit /home/git/gitlab/config/gitlab.yml and add or amend the following lines:

    gitlab_ci:
  # The location where build logs are stored (default: builds/).
  # Relative paths are relative to Rails.root.
  builds_path: path/to/builds/

  2. Save the file and restart GitLab for the changes to take effect.

Uploading logs to object storage

Archived logs are considered as job artifacts. Therefore, when you set up the object storage integration, job logs are automatically migrated to it along with the other job artifacts.

See "Phase 4: uploading" in Data flow to learn about the process.

Prevent local disk usage

If you want to avoid any local disk usage for job logs, you can do so using one of the following options:

How to remove job logs

There isn't a way to automatically expire old job logs, but it's safe to remove them if they're taking up too much space. If you remove the logs manually, the job output in the UI is empty.

For example, to delete all job logs older than 60 days, run the following from a shell in your GitLab instance:

WARNING: This command permanently deletes the log files and is irreversible.

find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete

Incremental logging architecture

Job logs are sent from the GitLab Runner in chunks and cached temporarily on disk in /var/opt/gitlab/gitlab-ci/builds by Omnibus GitLab. After the job completes, a background job archives the job log. The log is moved to /var/opt/gitlab/gitlab-rails/shared/artifacts/ by default, or to object storage if configured.

In a scaled-out architecture with Rails and Sidekiq running on more than one server, these two locations on the filesystem have to be shared using NFS.

To eliminate both filesystem requirements:

  • Enable the incremental logging feature, which uses Redis instead of disk space for temporary caching of job logs.
  • Configure object storage for storing archived job logs.

Technical details

The data flow is the same as described in the data flow section with one change: the stored path of the first two phases is different. This incremental log architecture stores chunks of logs in Redis and a persistent store (object storage or database) instead of file storage. Redis is used as first-class storage, and it stores up-to 128KB of data. After the full chunk is sent, it is flushed to a persistent store, either object storage (temporary directory) or database. After a while, the data in Redis and a persistent store is archived to object storage.

The data are stored in the following Redis namespace: Gitlab::Redis::SharedState.

Here is the detailed data flow:

  1. The runner picks a job from GitLab
  2. The runner sends a piece of log to GitLab
  3. GitLab appends the data to Redis
  4. After the data in Redis reaches 128KB, the data is flushed to a persistent store (object storage or the database).
  5. The above steps are repeated until the job is finished.
  6. After the job is finished, GitLab schedules a Sidekiq worker to archive the log.
  7. The Sidekiq worker archives the log to object storage and cleans up the log in Redis and a persistent store (object storage or the database).

Limitations

Enable or disable incremental logging (FREE SELF)

Incremental logging is under development, but ready for production use. It is deployed behind a feature flag that is disabled by default. GitLab administrators with access to the GitLab Rails console can enable it.

Before you enable the feature flag:

To enable incremental logging:

Feature.enable(:ci_enable_live_trace)

Running jobs' logs continue to be written to disk, but new jobs use incremental logging.

To disable incremental logging:

Feature.disable(:ci_enable_live_trace)

Running jobs continue to use incremental logging, but new jobs write to the disk.