Skip to main content

Uploading and Downloading CSV Files Using Amazon EFS in .NET Core: A Step-by-Step Guide

 

Working with files in distributed environments can be tricky, especially when you need to maintain high availability, scalability, and security. That’s where Amazon Elastic File System (EFS) comes into play, offering a fully managed, scalable file storage solution for use with AWS services and on-premise resources. In this blog, we’ll dive into how you can upload and download CSV files using EFS in a .NET Core application.

Let’s break it down step by step, and as always, we’ll make it interactive, engaging, and packed with useful code snippets!

Why Use Amazon EFS? 🤔

Amazon EFS is a great choice for:

  • Elastic storage: It grows and shrinks as you add or remove files.
  • Multi-AZ support: Accessible from multiple availability zones, perfect for distributed applications.
  • Secure and reliable: EFS offers encryption and seamless integration with AWS IAM and VPCs.

For scenarios where you need to manage CSV files in a multi-instance .NET Core application, EFS can act as a shared storage layer, making it ideal for microservices, web apps, or data processing pipelines.

Step 1: Setting Up Amazon EFS for Your .NET Core Application 🔧

Before we dive into code, let’s configure Amazon EFS:

1.1 Create an EFS File System

  • In your AWS console, navigate to EFS and create a new file system.
  • Ensure that the file system is mounted in the same VPC as your EC2 instances or containers running the .NET Core application.

1.2 Mount the EFS on Your Instances

  • Follow the steps in the AWS documentation to mount the EFS on your EC2 instances. Use the following command as an example:

sudo mount -t efs fs-xxxxxxxx:/ /mnt/efs
Ensure you have the required permissions and networking setup.

Step 2: CSV File Upload in .NET Core 📝

Now, let’s jump into the fun part — coding! We’ll use a .NET Core API to handle CSV uploads.

2.1 Model Class for Data Representation

First, we’ll create a model to represent our CSV data. For simplicity, let’s assume the CSV contains a list of Product:

public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
}

2.2 Upload Endpoint in the Controller

Next, create an API endpoint to upload the CSV file to the mounted EFS directory:

[ApiController]
[Route("api/[controller]")]
public class FileUploadController : ControllerBase
{
    private readonly string _efsDirectoryPath = "/mnt/efs/csv-files";  // EFS mount point
    
    [HttpPost("upload")]
    public async Task<IActionResult> UploadCsvFile(IFormFile file)
    {
        if (file == null || file.Length == 0)
            return BadRequest("No file uploaded.");

        var filePath = Path.Combine(_efsDirectoryPath, file.FileName);

        using (var stream = new FileStream(filePath, FileMode.Create))
        {
            await file.CopyToAsync(stream);
        }

        return Ok(new { FilePath = filePath });
    }
}

Breakdown:

  • IFormFile: Used to capture the uploaded CSV file.
  • FileStream: The file is saved to the EFS directory using a FileStream.
  • /mnt/efs/csv-files: This is the EFS mount point. When uploading, the CSV file is stored in this shared directory.

Step 3: Download CSV from EFS 🚀

Now that we’ve uploaded files to EFS, let’s create an endpoint to download them.

3.1 Download Endpoint

[HttpGet("download/{fileName}")]
public IActionResult DownloadCsvFile(string fileName)
{
    var filePath = Path.Combine(_efsDirectoryPath, fileName);
    
    if (!System.IO.File.Exists(filePath))
        return NotFound("File not found.");

    var fileBytes = System.IO.File.ReadAllBytes(filePath);
    return File(fileBytes, "text/csv", fileName);
}

Breakdown:

  • System.IO.File.ReadAllBytes(): Reads the CSV file from EFS into a byte array.
  • File(): Returns the file to the client as a downloadable resource.

Step 4: Parsing and Handling CSV Data in .NET Core 🔍

You’ll need to install the CsvHelper package. You can do this via NuGet Package Manager or by running the following command in the terminal:

dotnet add package CsvHelper

4.2 Parsing CSV Data After Upload

Let’s enhance our file upload functionality to parse the CSV and perhaps save the data into a database after uploading the file to EFS.

using CsvHelper;
using System.Globalization;

[HttpPost("upload-and-parse")]
public async Task<IActionResult> UploadAndParseCsv(IFormFile file)
{
    if (file == null || file.Length == 0)
        return BadRequest("No file uploaded.");

    var filePath = Path.Combine(_efsDirectoryPath, file.FileName);

    // Save the file to EFS
    using (var stream = new FileStream(filePath, FileMode.Create))
    {
        await file.CopyToAsync(stream);
    }

    // Parse CSV file
    using (var reader = new StreamReader(filePath))
    using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture))
    {
        var products = csv.GetRecords<Product>().ToList();
        // Process the products list (e.g., save to DB)
    }

    return Ok(new { FilePath = filePath });
}

Breakdown:

  • CsvReader: After saving the CSV file to EFS, we use the CsvReader from the CsvHelper library to parse the CSV file.
  • GetRecords<Product>(): This automatically maps each row in the CSV file to a Product object.
  • List of Products: You can now process the parsed products, such as saving them into a database or doing further business logic.

Step 5: Testing the API 🧪

At this point, you can test your API using tools like Postman or Swagger. Try uploading a CSV file and then downloading it, or parse it to check if everything works smoothly.

Here’s a simple example of what your CSV file might look like:

Id,Name,Price
1,Laptop,1200.00
2,Smartphone,800.00
3,Headphones,150.00
  • Upload it: Hit your api/fileupload/upload endpoint and send the CSV file.
  • Download it: After the file is uploaded, use the api/fileupload/download/{fileName} endpoint to retrieve it.
  • Parse and process it: Use the api/fileupload/upload-and-parse endpoint to parse and process the uploaded CSV data.

Wrapping Up: Scaling and Future Considerations 🎯

Congratulations! You’ve now successfully learned how to handle CSV file uploads and downloads using Amazon EFS in a .NET Core application. By leveraging the scalability of AWS’s EFS and the powerful file processing capabilities of .NET Core, you can build resilient, scalable systems capable of handling large amounts

Labels:

Comments

Post a Comment

Popular posts from this blog

Implementing and Integrating RabbitMQ in .NET Core Application: Shopping Cart and Order API

RabbitMQ is a robust message broker that enables communication between services in a decoupled, reliable manner. In this guide, we’ll implement RabbitMQ in a .NET Core application to connect two microservices: Shopping Cart API (Producer) and Order API (Consumer). 1. Prerequisites Install RabbitMQ locally or on a server. Default Management UI: http://localhost:15672 Default Credentials: guest/guest Install the RabbitMQ.Client package for .NET: dotnet add package RabbitMQ.Client 2. Architecture Overview Shopping Cart API (Producer): Sends a message when a user places an order. RabbitMQ : Acts as the broker to hold the message. Order API (Consumer): Receives the message and processes the order. 3. RabbitMQ Producer: Shopping Cart API Step 1: Install RabbitMQ.Client Ensure the RabbitMQ client library is installed: dotnet add package RabbitMQ.Client Step 2: Create the Producer Service Add a RabbitMQProducer class to send messages. RabbitMQProducer.cs : using RabbitMQ.Client; usin...
Post a Comment
Read more

How Does My .NET Core Application Build Once and Run Everywhere?

One of the most powerful features of .NET Core is its cross-platform nature. Unlike the traditional .NET Framework, which was limited to Windows, .NET Core allows you to build your application once and run it on Windows , Linux , or macOS . This makes it an excellent choice for modern, scalable, and portable applications. In this blog, we’ll explore how .NET Core achieves this, the underlying architecture, and how you can leverage it to make your applications truly cross-platform. Key Features of .NET Core for Cross-Platform Development Platform Independence : .NET Core Runtime is available for multiple platforms (Windows, Linux, macOS). Applications can run seamlessly without platform-specific adjustments. Build Once, Run Anywhere : Compile your code once and deploy it on any OS with minimal effort. Self-Contained Deployment : .NET Core apps can include the runtime in the deployment package, making them independent of the host system's installed runtime. Standardized Libraries ...
Post a Comment
Read more

Clean Architecture: What It Is and How It Differs from Microservices

In the tech world, buzzwords like   Clean Architecture   and   Microservices   often dominate discussions about building scalable, maintainable applications. But what exactly is Clean Architecture? How does it compare to Microservices? And most importantly, is it more efficient? Let’s break it all down, from understanding the core principles of Clean Architecture to comparing it with Microservices. By the end of this blog, you’ll know when to use each and why Clean Architecture might just be the silent hero your projects need. What is Clean Architecture? Clean Architecture  is a design paradigm introduced by Robert C. Martin (Uncle Bob) in his book  Clean Architecture: A Craftsman’s Guide to Software Structure and Design . It’s an evolution of layered architecture, focusing on organizing code in a way that makes it  flexible ,  testable , and  easy to maintain . Core Principles of Clean Architecture Dependency Inversion : High-level modules s...
Post a Comment
Read more