MePhoto of me

Josip Crnković

Uploading files directly from client to S3 without files ever touching your Laravel application

If your web application allows users to upload files, there are a few things to consider and be aware of before deploying file uploads to production. This article will assume you're using a storage service like S3 to store your user's files. There are many reasons why you should use it, and it is not the scope of this article. In this article, however, we will talk about how you can allow your users to upload files… but have the files never actually touch your application servers. Let's first see how many PHP and Laravel applications handle file uploads nowadays.

"Traditional" approach

The way PHP handles files is that once a file is sent as a part of the request, it's processed by PHP and then accessible via the $_FILES superglobal array. You can access the uploaded file from that array to persist it in the storage. This is how frameworks like Laravel handle file uploads. You get access to $request->file($name) and can use it to store the file in a directory on your "disk" of choice, for example: $request->file('document')->store('documents').

You even get support for uploading files to different filesystems. Whether you want to upload files to your local filesystem, S3, or something else — these frameworks have you covered. And there's nothing truly wrong with this approach, if your application is not huge and accepts small files, you can just go with the flow and use this. It's much simpler to code and test, after all.

However, there's a caveat with this approach. Whether you store files in local storage or on S3, every uploaded file touches your web server. I realized this was a bit counterintuitive for less experienced developers when they had a hard time grasping how the file is still processed by your web server, even though files ultimately end up in S3. Now ask yourself… what if you want to allow users to upload large files? All of these gigabyte-sized files will be landing on your server. We can improve that by having files never really touch your web server, but rather be uploaded directly from client to S3. It does require a bit of work, though.

Alternative approach

Storage services like S3 generally have a concept of "signed URLs". They are precisely that… signed URLs that you pre-generate and send to the user, and the user then uploads the file to that URL instead of your backend URL. This is the general idea, but it does require a few steps and some orchestration. As you can probably guess, we have to track whether the user has uploaded the file in a different way than "traditionally". Before we start, we can create a simple value object for our signed URL.

readonly class SignedUrl
{
    public function __construct(
        public string $key,
        public string $url,
        public array $headers,
    ) {}
}

In general, our logic (flow) for uploading files is going to be the following:

  1. The user selects the file to upload from their filesystem
  2. Our JavaScript client sends file metadata to our backend URL
  3. Our backend URL validates the file size and the content type, and generates the signed URL for this pending file
  4. The client receives the URL and sends the PUT request to the URL with the file attached to the request
  5. Upon upload, the client notifies our backend that the file was successfully uploaded
  6. Our backend creates all the necessary database records to be able to know about this file

Generating the signed URL

Below is the code snippet that generates the signed URL for uploading files of a given type with the given size on S3. I used S3Client from aws/aws-sdk-php library. You can see how our files have ACL (access control, visibility) set to private and the signed request expires after 5 minutes. Feel free to tune any of these values as you wish.

The main gist is creating an S3 "command" to "put object" into an S3 bucket (because S3 refers to files as "objects"). We'll specify the key that the file will be stored as, the bucket itself, and the size and type of the file. Then we'll use the createPresignedRequest method to create a signed URL for that putObject command. If you examine the URL, you'll see a giant mess. You can create pre-signed requests for viewing the file, and more.

class S3
{
    public function createUploadUrl(int $bytes, string $contentType): SignedUrl
    {
        $client = $this->client();

        $command = $client->getCommand('putObject', array_filter([
            'Bucket' => config('filesystems.disks.s3.bucket'),
            'Key' => $key = 'tmp/'.hash('sha256', (string) Str::uuid()),
            'ACL' => 'private',
            'ContentType' => $contentType,
            'ContentLength' => $bytes,
        ]));

        $request = $client->createPresignedRequest($command, '+5 minutes');

        $uri = $request->getUri();

        return new SignedUrl(
            key: $key,
            url: $uri->getScheme().'://'.$uri->getAuthority().$uri->getPath().'?'.$uri->getQuery(),
            headers: array_filter(array_merge($request->getHeaders(), [
                'Content-Type' => $contentType,
                'Cache-Control' => null,
                'Host' => null,
            ])),
        );
    }

    protected function client(): S3Client
    {
        return new S3Client([
            'region' => config('filesystems.disks.s3.region'),
            'version' => 'latest',
            'signature_version' => 'v4',
            'credentials' => [
                'key' => config('filesystems.disks.s3.key'),
                'secret' => config('filesystems.disks.s3.secret'),
            ],
        ]);
    }
}

Now that we have this logic, we can use it to generate a signed URL for the user on-demand, when the user asks to upload the document. We'll get back to step 2, but now let's hook that URL generator to our controller which is used in step 3. We'll have this concept of documents and the matching Eloquent model will be named Document.

public function store(Request $request, S3 $storage): JsonResponse
{
    $request->validate([
        'size' => 'required|int|max:'.Document::MAX_SIZE_BYTES,
        'content_type' => ['required', 'string', Rule::in(Document::validContentTypes())],
    ]);

    $signedUrl = $storage->createUploadUrl(
        bytes: $request->input('size'),
        contentType: $request->input('content_type'),
    );

    return response()->json([
        'key' => $signedUrl->key,
        'headers' => $signedUrl->headers,
        'url' => $signedUrl->url,
    ], 201);
}

At the current stage, we have the following: our users can hit our backend endpoint with the filesize and the content type of their file, and we'll first validate that the file they want to send is within our constraints, and then generate the signed URL for them to upload. You may be asking yourself… can't they just say their file is 10 bytes to pass the validation and then upload a 10GB file using the signed URL? Well, not quite. S3 takes care of that. The file uploaded must match the parameters used when creating the signed URL. Turns out, S3 no longer validates that the file size matches the pre-signed file when uploading, so you may need to do it yourself, either through S3 policies or manually removing larger files.

Notifying the backend when files upload

In step 5, I wrote that our backend must be notified when files are uploaded. This is logical; we'll upload files directly from the client, and our backend will not know whether the file has been uploaded or not. So after the file gets uploaded, we'll ping our backend with some data about the file.

Let's create a controller code that will:

  • accept file metadata and do the final validation
  • create the corresponding Document model for the file, that contains the final path where the file is located on S3
  • move the file from the temporary directory we used for these "pending" files to a permanent directory
public function store(Request $request): JsonResponse
{
    $request->validate([
        'key' => 'required|string',
        'size' => 'required|int|max:'.Document::MAX_SIZE_BYTES,
        'name' => 'required|string',
        'content_type' => ['required', 'string', Rule::in(Document::validContentTypes())],
    ]);

    $extension = Extension::guessFromContentType($request->input('content_type'));

    $name = hash('sha256', (string) Str::uuid()).'.'.$extension;

    $document = $request->user()->documents()->create([
        'name' => $request->input('name'),
        'extension' => $extension,
        'size' => $request->input('size'),
        'content_type' => $request->input('content_type'),
        'path' => 'documents/'.$name,
    ]);

    // Move the file from tmp directory...
    dispatch(function () use ($request, $name) {
        Storage::move($request->input('key'), 'documents/'.$name);
    });

    return response()->json([
        'document' => $document,
    ], 201);
}

Now, note that there are still ways to improve this code. Right now, users can just send an AJAX request to this endpoint and enter some random key, size, name, and content types, and our backend will still create a Document model, as we don't currently check that the file actually exists in S3. This is your place to shine and improvise. You can check whether the file exists on S3 as a part of the validation, you can create a temporary PendingDocument model and send its UUID when generating pre-signed URLs, and then the user would have to specify the UUID of the file that's uploaded, etc. The world is your oyster. Improvise and let me know on Twitter (X) on @jcrnkovic95 what approach you did, or if you have any questions.

Orchestrating uploads from client

Alright, now that we have our backend in place, we must orchestrate all of that on the client side. We know we're going to have to make 3 AJAX requests in the following order:

  • one to generate a signed URL
  • one to upload the file using the signed URL
  • one to notify our backend that the file was uploaded

Instead of talking too much, below is the basic code I used. It's still pretty primitive, meaning you should handle errors and show them to the user, handle upload progress bar, and other things… but that's all up to you. Note that it's written in TypeScript and uses Axios to make requests.

The gist is that we execute the following 3 requests sequentially and ultimately have the final document model created and sent to the frontend. We still catch all the potential exceptions and do any additional cleanup we migth need.

try {
    // Ask the backend for signed URL...
    const response = await axios.post<{
        url: string;
        key: string;
        headers: Record<string, string>;
    }>('/document-upload-url', {
        size: file.size,
        content_type: file.type,
    })
    
    // Upload the file to S3 using signed URL...
    await axios.put(response.data.url, file, {
        headers: response.data.headers,
        onUploadProgress: (e: ProgressEvent) => {
            console.log({
                progress: Math.round((e.loaded * 100) / e.total
            })
        },
    })
    
    // Notify the backend that the file was uploaded...
    const { data } = await axios.post<{
        document: Document;
    }>('/uploaded-documents', {
        key: response.data.key,
        size: file.size,
        name: file.name,
        content_type: file.type,
    })

    console.log("File Uploaded")
    console.log(data.document)
} catch (error) {
    console.log("Something went wrong trying to upload files")

    // cleanup...
}

With this approach, we do have more code to maintain, but we also have more control and don't have to worry about gigantic files just sitting there on our servers. I kinda like this approach, and I use a version of this every time I need to allow users to upload files from the client side.

If you want to allow users to view the files, I really don't need to tell you what to do — you have the path where the file is located on S3.