Automated GitLab Artifact Cleanup with AWS Lambda

Photo by Siarhei Palishchuk on Unsplash

GitLab CI/CD pipelines generate artifacts and build logs that accumulate over time, consuming valuable storage space. While these artifacts are crucial for debugging and deployment, keeping them indefinitely is neither practical nor cost-effective. Let’s explore how to automate the cleanup process using AWS Lambda and EventBridge.

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. Combined with EventBridge for scheduling and SNS for notifications, we can create a robust solution for maintaining our GitLab instance’s hygiene.

Let’s break down the solution into its core components and understand how they work together.

Core Components

Our cleanup solution consists of several key elements:

• AWS Lambda function for artifact cleanup
• GitLab API integration for accessing projects and jobs
• AWS SNS integration for notifications
• Environment variables for secure credential management
• Error handling and logging

Code Analysis

Let’s analyze the main components of our cleanup function:

1. Configuration and Setup:

const gitlabHost = 'https://gitlab.myinstance.com';
const token = process.env.GITLAB_TOKEN;
const date = new Date();
date.setMonth(date.getMonth() - 3); //Can be change

This section establishes our GitLab connection details and sets up a three-month threshold for artifact retention.

1. Main Cleanup Function:

async function deleteOldArtifacts() {
    try {
        let page = 1;
        while (true) {
            const projectsUrl = `${gitlabHost}/api/v4/projects?private_token=${token}&per_page=100&page=${page}`;
            const projectsResponse = await axios.get(projectsUrl);
            const projects = projectsResponse.data;
            if (projects.length === 0) break;
            for (const project of projects) {
                        await deleteArtifactsForJob(project.id);
                        await deleteArtifactsForProject(project.id);
                    }
                    page++;
                }
                sendSnsMessage('Old artifacts from all projects deleted successfully.');
            } catch (error) {
                sendSnsMessage(error.message);
            }
}

This function iterates through all GitLab projects, handling pagination, and processing each project’s artifacts.

1. SNS Notification:

async function sendSnsMessage(message) {
    const sns = new AWS.SNS();
    const topicArn = 'arn:aws:sns:eu-central-1:00000000000:arn'; 
    
    sns.publish({
        TopicArn: topicArn,
        Message: JSON.stringify(message)
    });
}

This component sends notifications about the cleanup process status.

Setting Up the Solution

1. Create a new Lambda function and set these environment variables:

• GITLAB_TOKEN
• AWS_A_KEY
• AWS_S_KEY

1. Configure the EventBridge rule with a cron expression to schedule the cleanup:

0 0 1 * ? *  // Runs monthly on the 1st

1. Set up an SNS topic and subscribe to receive notifications about the cleanup process.

Complete Code

Here’s the complete Lambda function code:

const axios = require('axios');
const AWS = require('aws-sdk');
require('dotenv').config();
const gitlabHost = 'https://gitlab.myinstance.com';
const token = process.env.GITLAB_TOKEN;
const date = new Date();
date.setMonth(date.getMonth() - 3);
const formattedThreeMonthsAgo = date.toISOString();
AWS.config.update({
    region: 'eu-central-1', 
    accessKeyId: process.env.AWS_A_KEY,
    secretAccessKey: process.env.AWS_S_KEY
});
async function deleteOldArtifacts() {
    try {
        let page = 1;
        while (true) {
            const projectsUrl = `${gitlabHost}/api/v4/projects?private_token=${token}&per_page=100&page=${page}`;
            const projectsResponse = await axios.get(projectsUrl);
            const projects = projectsResponse.data;
            if (projects.length === 0) break;
            for (const project of projects) {
                console.log(`Processing project ID: ${project.id}`);
                await deleteArtifactsForJob(project.id);
                await deleteArtifactsForProject(project.id);
            }
            page++;
        }
        sendSnsMessage('Old artifacts from all projects deleted successfully.');
    } catch (error) {
        console.log(error.message);
        sendSnsMessage(error.message);
    }
}
async function deleteArtifactsForProject(projectId) {
    try {
        const projectArtifactsUrl = `${gitlabHost}/api/v4/projects/${projectId}/artifacts?private_token=${token}`;
        await axios.delete(projectArtifactsUrl);   
    } catch (error) {
        console.error(`Error deleting artifacts for project ${projectId}:`, error.message);
    }
}
async function deleteArtifactsForJob(projectId) {
    try {
        const jobsUrl = `${gitlabHost}/api/v4/projects/${projectId}/jobs?private_token=${token}`;
        const jobsResponse = await axios.get(jobsUrl);
        const jobs = jobsResponse.data;
        for (const job of jobs) {
            try {
                if(typeof job.artifacts !== 'undefined' && job.artifacts.length > 0 && job.finished_at && job.finished_at < formattedThreeMonthsAgo) {
                    const deleteUrl = `${gitlabHost}/api/v4/projects/${projectId}/jobs/${job.id}/artifacts?private_token=${token}`;
                    await axios.delete(deleteUrl);
                    console.log(`Artifact ${job.id} deleted successfully.`);
                }
            } catch (error) {
                console.error(`Error deleting artifacts for job ${projectId}:`, error.message);
            }
        }
    } catch (error) {
        console.error(`Error deleting artifacts for job ${projectId}:`, error.message);
    }
}
async function sendSnsMessage(message) {
    const sns = new AWS.SNS();
    const topicArn = 'arn:aws:sns:eu-central-1:00000000000:arn'; 
    sns.publish({
    TopicArn: topicArn,
    Message: JSON.stringify(message)
    }, (err, data) => {
    if (err) {
        console.error('Error sending message:', err);
    } else {
        console.log('Message sent successfully:', data);
    }
    });
}
exports.handler = async (event) => {
    await deleteOldArtifacts();
    const response = {
      statusCode: 200,
      body: JSON.stringify('Done!'),
    };
    return response;
};

Benefits

• Automated cleanup reduces manual maintenance
• Configurable retention period
• Notification system for monitoring
• Cost-effective storage management
• Scalable solution for growing GitLab instances

Conclusion

This automated solution helps maintain a clean and efficient GitLab instance by regularly removing old artifacts. The combination of AWS Lambda, EventBridge, and SNS creates a reliable, hands-off maintenance system that can be easily adapted to your specific needs.

See you in the next article! 👻