Spring Boot + AWS S3 List Bucket Files

In this tutorial, we will develop AWS Simple Storage Service (S3) together with Spring Boot Rest API service to list all the files from AWS S3 Bucket.

What is S3?

Amazon Simple Storage Service (Amazon S3) is an object storage service that provides industry-leading scalability, data availability, security, and performance.
The service can be used as online backup and archiving of data and applications on Amazon Web Services (AWS).

AWS Core S3 Concepts

In 2006, S3 was one of the first services provided by AWS. Many features have been introduced since then, but the core principles of S3 remain Buckets and Objects.

• AWS Buckets

  Buckets are containers for objects that we choose to store. It is necessary to remember that S3 allows the bucket name to be globally unique.

• AWS Objects

  Objects are the actual items that we store in S3. They are marked by a key, which is a sequence of Unicode characters with a maximum length of 1,024 bytes in UTF-8 encoding.

Prerequisites

First Create Bucket on Amazon S3 and then Generate Credentials(accessKey and secretKey) to access AWS S3 bucket

Let's start developing AWS S3 + Spring Boot application.

Create Spring Boot Project

Refer to create Spring Boot S3 Project.

Maven Dependency

As seen in previous tutorial, add spring-cloud-starter-aws as given here.

Spring Cloud AWS Configuration

Add AWS configuration and security credentials in application.yml file, as seen in previous tutorial, refer application properties.

Add this logging.level.com.amazonaws.util.EC2MetadataUtils entry to get rid of EC2MetadataUtils exception.

fails to connect to service endpoint locally

Also we have shown here how to solve com.amazonaws.util.EC2MetadataUtils: Unable to retrieve the requested metadata (/latest/meta-data/instance-id). Failed to connect to service endpoint exception.

Amazon S3 Client Configuration

The main class for communicating with S3 is com.amazonaws.services.s3.AmazonS3.

To access the Amazon S3 web service, we must first establish a client connection. We will use BasicAWSCredentials AmazonS3 interface with AWS Access Key and AWS Secret Key and then configure the AmazonS3 client by passing this credential to AWSStaticCredentialsProvider as shown here.

Implement Service to Get the List of Files from AWS S3 Bucket

Create the S3BucketStorageService to implement the function to get the list of files present in AWS S3 Bucket.

package com.techgeeknext.springbootawss3.service;

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import java.util.ArrayList;
import java.util.List;

@Service
public class S3BucketStorageService {

    private Logger logger = LoggerFactory.getLogger(S3BucketStorageService.class);

    @Autowired
    private AmazonS3 amazonS3Client;

    @Value("${application.bucket.name}")
    private String bucketName;


    /**
     * Get all files from S3 bucket
     *
     * @return
     */
    public List<String> listFiles() {

        ListObjectsRequest listObjectsRequest =
                new ListObjectsRequest()
                        .withBucketName(bucketName);

        List<String> keys = new ArrayList<>();

        ObjectListing objects = amazonS3Client.listObjects(listObjectsRequest);

        while (true) {
            List<S3ObjectSummary> objectSummaries = objects.getObjectSummaries();
            if (objectSummaries.size() < 1) {
                break;
            }

            for (S3ObjectSummary item : objectSummaries) {
                if (!item.getKey().endsWith("/"))
                    keys.add(item.getKey());
            }

            objects = amazonS3Client.listNextBatchOfObjects(objects);
        }

        return keys;
    }

}

RestAPI - Get the List of Files from AWS S3 Bucket

Create the RestController class to get the list of files from AWS S3 bucket.

package com.techgeeknext.springbootawss3.controller;

import com.techgeeknext.springbootawss3.service.S3BucketStorageService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.util.List;

@RestController
public class S3BucketStorageController {
    @Autowired
    S3BucketStorageService service;

    @GetMapping("/list/files")
    public ResponseEntity<List<String>> getListOfFiles() {
        return new ResponseEntity<>(service.listFiles(), HttpStatus.OK);
    }
}

Test to get the list of file from AWS S3 Bucket

Now, run the Spring Boot application.

1. Upload File on AWS S3 Bucket

   Use POST method with url http://localhost:8081/file/upload, select file and provide filename.
2. Verify on AWS S3 Bucket.

3. List all Files from AWS S3 Bucket

   Use GET method with url http://localhost:8081/list/files.

Download Source Code

The full source code for this article can be found on below.
Download it here - Spring Cloud: AWS S3 Example