Recently
just make in depth research about DataTable. DataTable is easy to use variable
which allows user to pool data into it with consistence format, embed with
other properties that allow us to do sorting, search etc. Currently I am
researching on high performance application, which need to store large amount
of data into memory. Soon I hit a bottleneck with DataTable. When inserting
more than 5 million of records into DataTable, my desktop starts to struggle, even with 5G Ram So
I need to find an alternative.
The
main purpose of my tool is analyzing large
amount of data. It uses to compare ID and do heavy calculation on selected
field. After do some research, I found list can on par with DataTable’s needed
function, at the same time increase efficiency.
So
lets prove it!
DataTable
button click event
private void button3_Click(object
sender, EventArgs e)
{
Stopwatch
stopwatch = new Stopwatch();
stopwatch.Start();
DataTable
dt = new DataTable();
dt.Columns.Add("a");
dt.Columns.Add("b");
dt.Columns.Add("c");
dt.Columns.Add("d");
dt.Columns.Add("e");
long
i = 0;
while
(i < 1000000)
{
dt.Rows.Add("a" + i.ToString(),"b" + i.ToString(), "c" + i.ToString(), "d" + i.ToString(), "e" + i.ToString());
i++;
}
stopwatch.Stop();
MessageBox.Show(stopwatch.Elapsed.TotalMilliseconds.ToString());
}
List button click event
private void button4_Click(object
sender, EventArgs e)
{
Stopwatch
stopwatch = new Stopwatch();
stopwatch.Start();
List<List<lst>>
lists = new List<List<lst>>();
long
i = 0;
while
(i < 1000000)
{
List<lst> list = new
List<lst>();
lst
superlist = new lst
{
field1 = "a" + i.ToString(),
field2 = "b" + i.ToString(),
field3 = "c" + i.ToString(),
field4 = "d" + i.ToString(),
field5 = "e" + i.ToString()
};
lists.Add(list);
i++;
}
stopwatch.Stop();
MessageBox.Show(stopwatch.Elapsed.TotalMilliseconds.ToString());
}
class lst
{
public
string field1 { get;
set; }
public
string field2 { get;
set; }
public
string field3 { get;
set; }
public
string field4 { get;
set; }
public
string field5 { get;
set; }
}
Benchmark result:
datatable memory usage
list memory usage
time taken in milisecond, for DataTable
time taken in milisecond, for List
List is 4X faster then datatable
DataTable uses 6X more memory then list
Conclusion:
there are some performance tweak can be done on source code.
