Uploading and Downloading CSV Files Using Amazon EFS in .NET Core: A Step-by-Step Guide

 

Working with files in distributed environments can be tricky, especially when you need to maintain high availability, scalability, and security. That’s where Amazon Elastic File System (EFS) comes into play, offering a fully managed, scalable file storage solution for use with AWS services and on-premise resources. In this blog, we’ll dive into how you can upload and download CSV files using EFS in a .NET Core application.

Let’s break it down step by step, and as always, we’ll make it interactive, engaging, and packed with useful code snippets!

Why Use Amazon EFS? 🤔

Amazon EFS is a great choice for:

• Elastic storage: It grows and shrinks as you add or remove files.
• Multi-AZ support: Accessible from multiple availability zones, perfect for distributed applications.
• Secure and reliable: EFS offers encryption and seamless integration with AWS IAM and VPCs.

For scenarios where you need to manage CSV files in a multi-instance .NET Core application, EFS can act as a shared storage layer, making it ideal for microservices, web apps, or data processing pipelines.

Never miss new posts by subscribe

Powered by

Step 1: Setting Up Amazon EFS for Your .NET Core Application 🔧

Before we dive into code, let’s configure Amazon EFS:

1.1 Create an EFS File System

• In your AWS console, navigate to EFS and create a new file system.
• Ensure that the file system is mounted in the same VPC as your EC2 instances or containers running the .NET Core application.

1.2 Mount the EFS on Your Instances

• Follow the steps in the AWS documentation to mount the EFS on your EC2 instances. Use the following command as an example:

sudo mount -t efs fs-xxxxxxxx:/ /mnt/efs

Ensure you have the required permissions and networking setup.

Step 2: CSV File Upload in .NET Core 📝

Now, let’s jump into the fun part — coding! We’ll use a .NET Core API to handle CSV uploads.

2.1 Model Class for Data Representation

First, we’ll create a model to represent our CSV data. For simplicity, let’s assume the CSV contains a list of Product:

public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
}

2.2 Upload Endpoint in the Controller

Next, create an API endpoint to upload the CSV file to the mounted EFS directory:

[ApiController]
[Route("api/[controller]")]
public class FileUploadController : ControllerBase
{
    private readonly string _efsDirectoryPath = "/mnt/efs/csv-files";  // EFS mount point
    
    [HttpPost("upload")]
    public async Task<IActionResult> UploadCsvFile(IFormFile file)
    {
        if (file == null || file.Length == 0)
            return BadRequest("No file uploaded.");

        var filePath = Path.Combine(_efsDirectoryPath, file.FileName);

        using (var stream = new FileStream(filePath, FileMode.Create))
        {
            await file.CopyToAsync(stream);
        }

        return Ok(new { FilePath = filePath });
    }
}

Breakdown:

• IFormFile: Used to capture the uploaded CSV file.
• FileStream: The file is saved to the EFS directory using a FileStream.
• /mnt/efs/csv-files: This is the EFS mount point. When uploading, the CSV file is stored in this shared directory.

Step 3: Download CSV from EFS 🚀

Now that we’ve uploaded files to EFS, let’s create an endpoint to download them.

3.1 Download Endpoint

[HttpGet("download/{fileName}")]
public IActionResult DownloadCsvFile(string fileName)
{
    var filePath = Path.Combine(_efsDirectoryPath, fileName);
    
    if (!System.IO.File.Exists(filePath))
        return NotFound("File not found.");

    var fileBytes = System.IO.File.ReadAllBytes(filePath);
    return File(fileBytes, "text/csv", fileName);
}

Breakdown:

• System.IO.File.ReadAllBytes(): Reads the CSV file from EFS into a byte array.
• File(): Returns the file to the client as a downloadable resource.

Step 4: Parsing and Handling CSV Data in .NET Core 🔍

You’ll need to install the CsvHelper package. You can do this via NuGet Package Manager or by running the following command in the terminal:

dotnet add package CsvHelper

4.2 Parsing CSV Data After Upload

Let’s enhance our file upload functionality to parse the CSV and perhaps save the data into a database after uploading the file to EFS.

using CsvHelper;
using System.Globalization;

[HttpPost("upload-and-parse")]
public async Task<IActionResult> UploadAndParseCsv(IFormFile file)
{
    if (file == null || file.Length == 0)
        return BadRequest("No file uploaded.");

    var filePath = Path.Combine(_efsDirectoryPath, file.FileName);

    // Save the file to EFS
    using (var stream = new FileStream(filePath, FileMode.Create))
    {
        await file.CopyToAsync(stream);
    }

    // Parse CSV file
    using (var reader = new StreamReader(filePath))
    using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture))
    {
        var products = csv.GetRecords<Product>().ToList();
        // Process the products list (e.g., save to DB)
    }

    return Ok(new { FilePath = filePath });
}

Breakdown:

• CsvReader: After saving the CSV file to EFS, we use the CsvReader from the CsvHelper library to parse the CSV file.
• GetRecords<Product>(): This automatically maps each row in the CSV file to a Product object.
• List of Products: You can now process the parsed products, such as saving them into a database or doing further business logic.

Step 5: Testing the API 🧪

At this point, you can test your API using tools like Postman or Swagger. Try uploading a CSV file and then downloading it, or parse it to check if everything works smoothly.

Here’s a simple example of what your CSV file might look like:

Id,Name,Price
1,Laptop,1200.00
2,Smartphone,800.00
3,Headphones,150.00

• Upload it: Hit your api/fileupload/upload endpoint and send the CSV file.
• Download it: After the file is uploaded, use the api/fileupload/download/{fileName} endpoint to retrieve it.
• Parse and process it: Use the api/fileupload/upload-and-parse endpoint to parse and process the uploaded CSV data.

Never miss new posts by subscribe

Powered by

Wrapping Up: Scaling and Future Considerations 🎯

Congratulations! You’ve now successfully learned how to handle CSV file uploads and downloads using Amazon EFS in a .NET Core application. By leveraging the scalability of AWS’s EFS and the powerful file processing capabilities of .NET Core, you can build resilient, scalable systems capable of handling large amounts