Closed

dynamic CSV to SQL server import - automated using SSIS C#

Project Requirement:

I want to create a solution to load a dynamic csv file to SQL table. Following requirements need to be met:

1. The input CSV/TXT file could have any number of columns.

2. Column Delimiter of input file could be comma(,), pipe (|), tab (<tab>), basically it could be different for different file but will be standard in a single file

3. Row delimiter could be [CR][LF] or [LF] or [CR] etc

4. The solution should load all the contents of this file to a SQL server table. (sql version could be 2012/2014 etc)

5. The destination table will be created based on the column names in the header row of the input file

6. All the destination table columns should have sql data type nvarchar(max)

7. Destination table should have PAGE type compression in SQL server

8. Preferably the solution should be a SSIS package or a programatically generated and executed SSIS that loads any files into a table it creates.

9. Solution should just accept a path to the CSV/TXT file and should be able to figure out everything else automatically (number of columns, create table and then load)

10. I should be able to use this to load to SQL 2012, SQL 2014, SQL 2016

Performance requirements

1. The input files I have are very heavy 50GB, 70GB, 80GB etc, so the solution needed should be able to work with such files and also load the the data quickly

2. I already have a few solutions:

a) Load csv into a single column table and then parse the column names and data to a table will multiple columns, this process is too slow, so its not acceptable

b) use C# script to read header row from csv file, create a table and then load the data using C#. This process is also slow

So the requirement is to have the solution as fast as a BCP task in SSIS. I am able to build a static BCP where filename is provided, column structure is static, this works fast but need is to make this dynamic.

Copyright requirement:

1. Cannot use any thing that is patented to someone

2. Cannot used a paid library, SSIS addon, every thing should be developed in house

3. you can simply use C#, SSIS, script component etc.... microsoft provided features, so there is no question of using anything copyrighted

Documentation Requirements

1. Everything should provide should be documented, so it can be easily reproduced by me on a new installation of SQL/SSIS/Visual Studio etc

2. Code needs to be commented, so every line to code has an explanation

Skills: .NET, C# Programming, Microsoft SQL Server, Software Architecture, SQL

See more: import most recent csv file to sql server in ssis, ssis excel source dynamic filename, ssis script component read csv file, how to create dynamic excel destination in ssis, ssis import excel dynamic columns, import data from csv to sql server using c# in asp. net, c# import csv file to datatable, dynamic ssis package to import excel files into sql server database, import csv sql server 2000 using vb6, import csv sql server 2005 database, import csv sql server, import csv sql server 2005 express, import csv sql server express edition, import import csv sql server express 2005, import csv sql server 2005 dts

About the Employer:
( 0 reviews ) India

Project ID: #16256939

18 freelancers are bidding on average ₹9287 for this job

aaz256

Hi! Read your project proposal about fast import of CSV data to SQL Server. This is a really interesting and challenging task for me, I really want to do this. I have a 15 years experience on MS SQL database developmen More

₹8777 INR in 10 days
(6 Reviews)
4.6
dogradiwakar

I have 8 + years of experience of working on SSIS & SQL server based technologies. I have completed these typed of projects in the past .

₹13333 INR in 6 days
(4 Reviews)
4.6
₹7777 INR in 3 days
(2 Reviews)
4.0
vw7590795vw

Please review my profile. https://www.freelancer.com/u/vw7590795vw.html I have 8 years and 3 months experience in SQL and 5 years and 5 months experience in SSRS, SSIS. Please let me know more details and when w More

₹6500 INR in 5 days
(19 Reviews)
4.0
daisoftware123

Hi, I have gone through your REQUIREMENT. I can help you in this as I have much EXPERIENCE in Asp.net, C#, MVC, JSON Web services , ATOM Web services. Can we discuss the REQUIREMENT in detail? We work on dif More

₹10000 INR in 20 days
(3 Reviews)
3.0
bnrajumnc

A proposal has not yet been provided

₹13333 INR in 10 days
(9 Reviews)
3.5
sreenivas2903

I have ready package to load a file and import data , i will do for all files as well if you would like to take it . please let me know

₹8888 INR in 2 days
(2 Reviews)
1.1
₹7777 INR in 3 days
(0 Reviews)
0.0
₹11111 INR in 4 days
(0 Reviews)
0.0
₹9444 INR in 5 days
(0 Reviews)
0.0
leadsexport

A proposal has not yet been provided

₹12500 INR in 3 days
(0 Reviews)
0.0
ghadage70

Hi... It's Easy to do this work in normal form when we insert data from any type of file like (CSV,text,docx) to database....But when we insert 50GB of data it mean it carry lots of rows ... and lots of time... so don' More

₹8888 INR in 3 days
(0 Reviews)
0.0
tushargarg01

Certifications  IBM DB2 Academic Associate: DB2 Database and Application Fundamentals  Microsoft Certified Professional(MCP)  Microsoft Technology Associate(MTA): Networking Fundamentals  Training Certificate: More

₹7222 INR in 3 days
(0 Reviews)
0.0
rarruda1988

I am currently a Business Intelligence Consultant, where I work with ETL and also with reports and Dashboards. With ETL I work with SSIS, where to read the data of a text file to load in a table of SQL, to me it is qu More

₹8888 INR in 10 days
(0 Reviews)
0.0
dhamog

I have worked similar project where i have to load hedis measures. It is a healthcare quality measure project.

₹6666 INR in 3 days
(0 Reviews)
0.0
pankajk20

A proposal has not yet been provided

₹9288 INR in 5 days
(0 Reviews)
0.0
₹7777 INR in 3 days
(0 Reviews)
0.0
₹9000 INR in 3 days
(0 Reviews)
0.0