You should know how to parse or read CSV files as a developer. Because it is one of the most common formats to store data. It is very flexible and also not dependent on any programming language.
We will use createReadStream() method from the fs module to create a readable stream and extract data as plain text from CSV files into smaller pieces. We will convert that text into a suitable data format like an array or object. We can do this manually or with external packages like csv-parse or fast-csv etc.
There are other methods in Node.js to extract data from a CSV file. We can also use the readFile() method from fs module instead of creating a readable stream. But it has some limitations.
In this article, I will discuss how you can read CSV files using Node.js to convert their data into meaningful information for your application.
Not only that, you will see different methods that you can use to parse data from a CSV file and their limitations. So that you can choose the right method for your requirements.
What is a CSV File?
CSV stands for comma-separated values. A CSV file is used to store data as plain text which is separated by commas. Most of the time, such files will store the tabular dataset in CSV format. Each row in the table will have a separate line inside this file. The comma will separate every column entry inside the row.
The main advantage of this format is we can keep a large amount of data inside a CSV file. Most of the applications allow exporting data in this format for this reason.
We can easily export large quantities from one application and then import that into another application. Therefore, it makes transferring information from one place to another.
That is why this file is used in spreadsheet applications and for statistical analysis.
How to Read CSV Files with Node.js
As you have learned what a CSV file is and the advantages of using it. Now you can guess why it is important to know how to read this file in your application.
By reading, you can allow your users to import data by uploading a CSV file to your Node.js application. You can also write and create CSV files.
Any users then will be able to export their data by downloading that file from the Node.js server very easily. This feature not only increases flexibility but also provides a better user experience.
These are the steps we will go through to parse information from a CSV file:
- Step 1: We will get the raw data from the file. We can extract it in 2 ways using
readFile()method and creating a read stream. - Step 2: We will convert the raw data into different formats like an array and object so that we can use them inside our application.
- Step 3: We will read CSV files using external packages. If you don't want to handle the data conversion manually then these packages can help you.
Let's see how we can extract data from a CSV file.
Read CSV Files Using readFile() Method
You can read the whole file at once using readFile() method from the fs module. This method will extract everything from a CSV file and keep it in the memory.
Once the reading completes, it will return the text from the file. I will use the readFile() method that returns a promise. Because async/await is much cleaner than the callback function.
const { readFile } = require('fs').promises;
(async () => {
const data = await readFile('./products.csv', {
encoding: 'utf8',
});
console.log(data);
})();
Output:
ID,Name,Price,Stock,Rating
1,T-Shirt,12.65,48,4.8
2,Hoodie,14.88,33,4.6
3,Sunglasses,54.49,17,4.7
4,Belt,17.3,23,4.3
5,Cap,16.96,18,4.5
6,Watches,79.65,11,5
7,Wallet,14.99,37,4.3
8,Shoes,33.9,26,4.6
9,Socks,14.4,41,4.5
10,Jeans,28.23,57,4.3
I have a products.csv file that I want to read the data from. I am using a self-invoking function in JavaScript because I will use async/await syntax. When I run this javascript file, this function will automatically be executed.
Inside this function, I am calling the readFile() method with the file path. This method also accepts some options as its second argument.
I am setting the encoding: "utf8" option. Otherwise, this method will return the data as a buffer, not as a text. When we set this option, it will convert the buffer into plain text and returns it.
In the output section, you can see the data extracted from the product.csv file.
The main limitation of this method is it reads the entire file before giving any data from the file. If the file size is large, it will take some time, and users have to wait to see any result.
For a large file, it will also take more memory of your application. Because everything will be stored in the memory first. Therefore, your application might become out of memory very quickly
To overcome this limitation, we can read any files by creating a readable stream in the Node.js application.
Create Read Stream to Parse CSV Files
When we create a read stream, it will read a file into smaller pieces. It doesn't store the entire file content in the memory. Instead, it parses a smaller portion of it and then returns that portion.
It repeats the same process until the entire file is completed. Therefore, it will consume less memory and we will start getting our data very quickly.
Not only for reading, but we can also follow the same process for writing a file. We can create a read and write stream in Node.js for handling files.
const { createReadStream } = require('fs');
(() => {
const readStream = createReadStream('./products.csv', {
encoding: 'utf8',
});
readStream.on('data', (chunk) => {
console.log(chunk);
});
readStream.on('error', (err) => {
console.log(err);
console.log('error found');
});
readStream.on('end', () => {
console.log('Finished reading');
});
})();
Output:
ID,Name,Price,Stock,Rating
1,T-Shirt,12.65,48,4.8
2,Hoodie,14.88,33,4.6
3,Sunglasses,54.49,17,4.7
4,Belt,17.3,23,4.3
5,Cap,16.96,18,4.5
6,Watches,79.65,11,5
7,Wallet,14.99,37,4.3
8,Shoes,33.9,26,4.6
9,Socks,14.4,41,4.5
10,Jeans,28.23,57,4.3
Finished reading
To create a readable stream, you have to use the createReadStream() method from the fs module. When you call this method with the file path, it will return the stream for that file.
This method also takes some options as its second argument where you need to set the encoding option so that you receive plain text instead of a buffer.
To get the data from the stream, we can listen to the data event. It also emits other events like error to handle unexpected errors and end event to do something when the stream completes reading the entire file.
When we listen to the data event, the callback function will give you access to a chunk of data. Every time the read stream parses a small amount of data and returns it, this callback function will execute.
At the end, when the stream completes reading the file completely, it will emit the end event. If there is an error in the process, the error event will be emitted.
Read CSV Files Line By Line
A read stream returns data in small pieces. But if you want to get data from CSV files line by line, you will use the readline module. It is a Node.js core module.
You still need to create a read stream because this module needs to be used with the readable streams.
const { createReadStream } = require('fs');
const readline = require('readline');
(() => {
const readStream = createReadStream('./products.csv', {
encoding: 'utf8',
});
// Reading line by line
const reader = readline.createInterface({ input: readStream });
reader.on('line', (line) => {
console.log('=====================');
console.log(line);
console.log('=====================');
});
readStream.on('error', (err) => {
console.log(err);
console.log('error found');
});
readStream.on('end', () => {
console.log('Finished reading');
});
})();
Output:
=====================
ID,Name,Price,Stock,Rating
=====================
=====================
1,T-Shirt,12.65,48,4.8
=====================
=====================
2,Hoodie,14.88,33,4.6
=====================
=====================
3,Sunglasses,54.49,17,4.7
=====================
=====================
4,Belt,17.3,23,4.3
=====================
=====================
5,Cap,16.96,18,4.5
=====================
=====================
6,Watches,79.65,11,5
=====================
=====================
7,Wallet,14.99,37,4.3
=====================
=====================
8,Shoes,33.9,26,4.6
=====================
=====================
9,Socks,14.4,41,4.5
=====================
=====================
10,Jeans,28.23,57,4.3
=====================
Finished reading
You need to follow the same method to create the stream. After that, you have to call the createInterface() method on the readline module. This method accepts a stream.
Therefore, I am passing the readStream on the method as its argument. I am storing the return value from this method in the reader variable.
Now to get back each line separately, you have to listen to the line event on the reader variable instead of the data event on the read stream.
Inside the callback function of the line event, you will have access to your data line by line from the CSV file. Everything else is the same.
You can still listen to other events on the readable stream to handle errors and do something when the reading is completed.
Also Read: Learn How to Get File Extensions in Node.js From Any Files
Convert CSV File Content into Array
When we get data from a CSV file, we get it as plain text. It is not that useful when we try to analyze or do something with it. That is why we need to convert that plain text into another data type.
In this section, I will show you how you can convert plain text into an array. From our product data, we will create an array for each product.
const { createReadStream } = require('fs');
(() => {
const readStream = createReadStream('./products.csv', {
encoding: 'utf8',
});
const products = [];
readStream.on('data', (chunk) => {
console.log('products received');
products.push(
...chunk.split(/\r\n/).map((line) => {
return line.split(',');
})
);
});
readStream.on('error', (err) => {
console.log(err);
console.log('error found');
});
readStream.on('end', () => {
console.log('Finished reading');
console.log(products);
});
})();
Output:
[
[ 'ID', 'Name', 'Price', 'Stock', 'Rating' ],
[ '1', 'T-Shirt', '12.65', '48', '4.8' ],
[ '2', 'Hoodie', '14.88', '33', '4.6' ],
[ '3', 'Sunglasses', '54.49', '17', '4.7' ],
[ '4', 'Belt', '17.3', '23', '4.3' ],
[ '5', 'Cap', '16.96', '18', '4.5' ],
[ '6', 'Watches', '79.65', '11', '5' ],
[ '7', 'Wallet', '14.99', '37', '4.3' ],
[ '8', 'Shoes', '33.9', '26', '4.6' ],
[ '9', 'Socks', '14.4', '41', '4.5' ],
[ '10', 'Jeans', '28.23', '57', '4.3' ]
]
I am getting data using a read stream. Every time I receive a chunk of data, I am converting them into arrays using split() and map() methods.
At first, I am getting an array by splitting the text by /\r\n/ where each item will be a single product. Then I am mapping over the array and convert each product string into an individual array by splitting it by comma (,).
- Step 1: Getting an array by splitting the text using
split(/\r\n/)method. Every item inside this array will be the string of a product.
[
'ID,Name,Price,Stock,Rating',
'1,T-Shirt,12.65,48,4.8',
'2,Hoodie,14.88,33,4.6',
'3,Sunglasses,54.49,17,4.7',
'4,Belt,17.3,23,4.3',
...
]
- Step 2: Mapping over this array with the
map()method to get the individual item. Then split each of them by the comma (,).
[
[ 'ID', 'Name', 'Price', 'Stock', 'Rating' ],
[ '1', 'T-Shirt', '12.65', '48', '4.8' ],
[ '2', 'Hoodie', '14.88', '33', '4.6' ],
[ '3', 'Sunglasses', '54.49', '17', '4.7' ],
[ '4', 'Belt', '17.3', '23', '4.3' ],
...
]
- Step 3: Push this array into the
productsarray by using the spread operator in JavaScript.
Finally, I am pushing all the product arrays into the products array by using the spread operator in JavaScript.
We can check the products array when the reading completes.
Convert CSV File Content into Object
Other than an array, we can convert plain text received from the CSV file into an array of objects. As you can see, the first line in the plain text is the labels of the products like ID, Name, Price, Stock, and Rating.
We can generate product objects by using these labels as the object properties.
const { createReadStream } = require('fs');
(() => {
const readStream = createReadStream('./products.csv', {
encoding: 'utf8',
});
let products = [];
readStream.on('data', (chunk) => {
products = [
...chunk.split(/\r\n/).map((link) => {
return link.split(',');
}),
];
});
readStream.on('error', (err) => {
console.log(err);
console.log('error found');
});
readStream.on('end', () => {
console.log('Finished reading');
const objectArray = [];
products.forEach((product, index) => {
if (index === 0) {
return;
}
const keys = products[0];
// [ 'ID', 'Name', 'Price', 'Stock', 'Rating' ]
const productObject = {};
keys.forEach((key, index) => {
productObject[key] = product[index];
});
objectArray.push(productObject);
});
console.log(objectArray);
});
})();
Output:
[
{
ID: '1',
Name: 'T-Shirt',
Price: '12.65',
Stock: '48',
Rating: '4.8'
},
{
ID: '2',
Name: 'Hoodie',
Price: '14.88',
Stock: '33',
Rating: '4.6'
},
{
ID: '3',
Name: 'Sunglasses',
Price: '54.49',
Stock: '17',
Rating: '4.7'
},
{ ID: '4', Name: 'Belt', Price: '17.3', Stock: '23', Rating: '4.3' },
{ ID: '5', Name: 'Cap', Price: '16.96', Stock: '18', Rating: '4.5' },
{
ID: '6',
Name: 'Watches',
Price: '79.65',
Stock: '11',
Rating: '5'
},
{
ID: '7',
Name: 'Wallet',
Price: '14.99',
Stock: '37',
Rating: '4.3'
},
{ ID: '8', Name: 'Shoes', Price: '33.9', Stock: '26', Rating: '4.6' },
{ ID: '9', Name: 'Socks', Price: '14.4', Stock: '41', Rating: '4.5' },
{
ID: '10',
Name: 'Jeans',
Price: '28.23',
Stock: '57',
Rating: '4.3'
}
]
To get an array of objects, first, we have to convert the text into an array of arrays. We have learned this in the previous section.
Now, when the read stream emits the end event, we have to convert that array of arrays into an array of objects.
- I am looping over the
productsarray usingforEach()method. Inside the callback function, I am accessing the individual product array and its index. - When the
indexis equal to 0, I am not doing anything because this array doesn't contain product information. The first array contains all the object keys and I am storing them in thekeysvariable. - Now I am generating an object for each product by looping over the
keysarray. Finally, I am pushing each object into theobjectArrayarray.
Also Read: Perfect Ways to Serve Static Files in Express and Node.js
Using csv-parse Package to Read CSV Files
When you try to read data from CSV files and convert them manually, you have to do many things. You can make this process simple by using an external package. There are many packages available to get the job done.
Among them, the csv-parse is a very popular package. You can install this package by running the following command:
npm install csv-parse
Now, import the fs and csv-parse packages in your JavaScript file.
const { parse } = require('csv-parse');
const { createReadStream } = require('fs');
Here, I am importing the parse and createReadStream method from those packages. Because I have to create a readable stream using the createReadStream() method.
You can call the pipe() method on the read stream. Inside this pipe() method, you need to execute the parse() method. You need to listen to the data event to access the data.
(() => {
const products = [];
const readStream = createReadStream('./products.csv', 'utf8');
readStream.pipe(parse()).on('data', (chunk) => {
products.push(chunk);
});
readStream.on('error', (err) => {
console.log('Error found');
});
readStream.on('end', () => {
console.log(products);
console.log('Finished reading using csv parse');
});
})();
Output:
[
[ 'ID', 'Name', 'Price', 'Stock', 'Rating' ],
[ '1', 'T-Shirt', '12.65', '48', '4.8' ],
[ '2', 'Hoodie', '14.88', '33', '4.6' ],
[ '3', 'Sunglasses', '54.49', '17', '4.7' ],
[ '4', 'Belt', '17.3', '23', '4.3' ],
[ '5', 'Cap', '16.96', '18', '4.5' ],
[ '6', 'Watches', '79.65', '11', '5' ],
[ '7', 'Wallet', '14.99', '37', '4.3' ],
[ '8', 'Shoes', '33.9', '26', '4.6' ],
[ '9', 'Socks', '14.4', '41', '4.5' ],
[ '10', 'Jeans', '28.23', '57', '4.3' ]
]
By default, you get an array of arrays. But you can customize our output by passing some options in the parse() method.
readStream
.pipe(
parse({
delimiter: ',',
columns: true,
trim: true,
})
)
.on('data', (chunk) => {
products.push(chunk);
});
Now I am getting my output as an array of objects. You can specify how your data is separated by the delimiter option. Our CSV file data is separated by commas. Therefore, I am using a comma (,) for this option.
The columns: true option is responsible to convert an array of arrays into an array of objects. The trim: true removes any extra space from the property values.
There are many options available in this package to make your output more customizable. This is what your code will look like:
const { parse } = require('csv-parse');
const { createReadStream } = require('fs');
(() => {
const products = [];
const readStream = createReadStream('./products.csv', 'utf8');
readStream
.pipe(
parse({
delimiter: ',',
columns: true,
trim: true,
})
)
.on('data', (chunk) => {
products.push(chunk);
});
readStream.on('error', (err) => {
console.log('Error found');
});
readStream.on('end', () => {
console.log(products);
console.log('Finished reading using csv parse');
});
})();
Output:
[
{
ID: '1',
Name: 'T-Shirt',
Price: '12.65',
Stock: '48',
Rating: '4.8'
},
{
ID: '2',
Name: 'Hoodie',
Price: '14.88',
Stock: '33',
Rating: '4.6'
},
{
ID: '3',
Name: 'Sunglasses',
Price: '54.49',
Stock: '17',
Rating: '4.7'
},
{ ID: '4', Name: 'Belt', Price: '17.3', Stock: '23', Rating: '4.3' },
{ ID: '5', Name: 'Cap', Price: '16.96', Stock: '18', Rating: '4.5' },
{
ID: '6',
Name: 'Watches',
Price: '79.65',
Stock: '11',
Rating: '5'
},
{
ID: '7',
Name: 'Wallet',
Price: '14.99',
Stock: '37',
Rating: '4.3'
},
{ ID: '8', Name: 'Shoes', Price: '33.9', Stock: '26', Rating: '4.6' },
{ ID: '9', Name: 'Socks', Price: '14.4', Stock: '41', Rating: '4.5' },
{
ID: '10',
Name: 'Jeans',
Price: '28.23',
Stock: '57',
Rating: '4.3'
}
]
Also Read: Best Setup to Use Typescript with Node.js and Express
Using csv-parser Package to Read CSV Files
The csv-parse package is very popular and has many options. But the main disadvantage of this package is its size. It is about 1.31 MB. On the other hand, the csv-parser package is only about 27.7 KB in size.
If you need a lightweight package to read CSV files in Node.js, you can use this package for your application. The csv-parser package is great in the case of formatting data.
It always returns an array of objects. You can format object properties and values individually.
Install this package with this command:
npm install csv-parser
Now, you can use this package in your project. You have to use it with the read stream. So, create a stream using the createReadStream() method.
const csvParser = require('csv-parser');
const { createReadStream } = require('fs');
(() => {
const products = [];
const readStream = createReadStream('./products.csv', 'utf8');
readStream
.pipe(
csvParser({
separator: ',',
mapHeaders: ({ header }) => {
return header.toLowerCase();
},
})
)
.on('data', (chunk) => {
products.push(chunk);
});
readStream.on('error', (err) => {
console.log('Error found');
});
readStream.on('end', () => {
console.log(products);
console.log('Finished reading using csv parser');
});
})();
Output:
[
{
id: '1',
name: 'T-Shirt',
price: '12.65',
stock: '48',
rating: '4.8'
},
{
id: '2',
name: 'Hoodie',
price: '14.88',
stock: '33',
rating: '4.6'
},
{
id: '3',
name: 'Sunglasses',
price: '54.49',
stock: '17',
rating: '4.7'
},
{ id: '4', name: 'Belt', price: '17.3', stock: '23', rating: '4.3' },
{ id: '5', name: 'Cap', price: '16.96', stock: '18', rating: '4.5' },
{
id: '6',
name: 'Watches',
price: '79.65',
stock: '11',
rating: '5'
},
{
id: '7',
name: 'Wallet',
price: '14.99',
stock: '37',
rating: '4.3'
},
{ id: '8', name: 'Shoes', price: '33.9', stock: '26', rating: '4.6' },
{ id: '9', name: 'Socks', price: '14.4', stock: '41', rating: '4.5' },
{
id: '10',
name: 'Jeans',
price: '28.23',
stock: '57',
rating: '4.3'
}
]
You can execute the csvParser() function inside the pipe() on the read stream. This function accepts an object with the options.
I am passing separator: ',' because my CSV data uses commas to separate the values. I am also using mapHeader() method to convert all the properties into lowercase. You can use mapValues() method to work with property values in the objects.
Using fast-csv Package to Read CSV Files
The fast-csv package is very small. This package is only 8.53 KB in size. It is smaller than the csv-parser package. It also works with the read stream and by default without any options returns an array of arrays.
First, install this package with this command:
npm install fast-csv
This package also has the parse() method that you need to call inside the pipe() method on the read stream. You can pass the customization options as an object in the parse() method.
You can receive an array of objects from this package by setting headers: true options.
const { parse } = require('fast-csv');
const { createReadStream } = require('fs');
(() => {
const products = [];
const readStream = createReadStream('./products.csv', 'utf8');
readStream
.pipe(
parse({
delimiter: ',',
headers: true,
trim: true,
})
)
.on('data', (chunk) => {
products.push(chunk);
});
readStream.on('error', (err) => {
console.log('Error found');
});
readStream.on('end', () => {
console.log(products);
console.log('Finished reading using fast-csv');
});
})();
Output:
[
{
ID: '1',
Name: 'T-Shirt',
Price: '12.65',
Stock: '48',
Rating: '4.8'
},
{
ID: '2',
Name: 'Hoodie',
Price: '14.88',
Stock: '33',
Rating: '4.6'
},
{
ID: '3',
Name: 'Sunglasses',
Price: '54.49',
Stock: '17',
Rating: '4.7'
},
{ ID: '4', Name: 'Belt', Price: '17.3', Stock: '23', Rating: '4.3' },
{ ID: '5', Name: 'Cap', Price: '16.96', Stock: '18', Rating: '4.5' },
{
ID: '6',
Name: 'Watches',
Price: '79.65',
Stock: '11',
Rating: '5'
},
{
ID: '7',
Name: 'Wallet',
Price: '14.99',
Stock: '37',
Rating: '4.3'
},
{ ID: '8', Name: 'Shoes', Price: '33.9', Stock: '26', Rating: '4.6' },
{ ID: '9', Name: 'Socks', Price: '14.4', Stock: '41', Rating: '4.5' },
{
ID: '10',
Name: 'Jeans',
Price: '28.23',
Stock: '57',
Rating: '4.3'
}
]
It works like the csv-parse package with limited features. But the main advantage is it is very small compared to the other packages.
If you don't need any advanced features in your project then you can definitely choose this package.
Conclusion
CSV file format is very effective in order to store a large amount of data and it is also easy to parse. You can use any programming language to extract data out of a CSV file. That's why companies are using this file format.
You can use the Node.js core fs module to read data from this file or there are many external packages available to make the reading process easy and simple.
After getting the data as plain text, you can convert it into any other data type like an array, object, etc according to the requirement of your application.
In this article, I have shown you how to read CSV files using the Node.js core module and other npm packages. I hope you will be able to use this knowledge in your upcoming projects.
