CSVFiletoarray reading large file

Technical support and scripting issues

Moderators: Dorian (MJT support), JRL

Post Reply
jsmyong55
Newbie
Posts: 18
Joined: Thu Aug 29, 2013 4:14 am

CSVFiletoarray reading large file

Post by jsmyong55 » Fri Jan 03, 2014 3:12 am

I am experiencing a problem where MS hangs when I read a huge CSV file containing more than 10k records. What is another alternative to using csvfiletoarray?

I have to read all records from the csv and then transfer it to the database. Would DB query be a better option? Or Readln (and removing the delimiters of course)?

User avatar
Marcus Tettmar
Site Admin
Posts: 7395
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Post by Marcus Tettmar » Fri Jan 03, 2014 11:50 am

DBQuery or use the XL commands to read one cell at a time.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

User avatar
Grovkillen
Automation Wizard
Posts: 1131
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: CSVFiletoarray reading large file

Post by Grovkillen » Tue Oct 18, 2016 12:22 pm

I have a large CSV-file (almost 3MB big, 18 000 rows, 12 columns) and I have problems importing this as an array when compiled my script as a EXE-file.

Is there something I should know about the CSVFileToArray regarding limits?
Let>ME=%Script%

Running: 15.0.27
version history

User avatar
Marcus Tettmar
Site Admin
Posts: 7395
Joined: Thu Sep 19, 2002 3:00 pm
Location: Dorset, UK
Contact:

Re: CSVFiletoarray reading large file

Post by Marcus Tettmar » Tue Oct 18, 2016 12:31 pm

What problems? Should be absolutely no difference between any script running as a .exe vs not.
Marcus Tettmar
http://mjtnet.com/blog/ | http://twitter.com/marcustettmar

Did you know we are now offering affordable monthly subscriptions for Macro Scheduler Standard?

User avatar
Grovkillen
Automation Wizard
Posts: 1131
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: CSVFiletoarray reading large file

Post by Grovkillen » Tue Oct 18, 2016 12:43 pm

It crashes the program, i.e. "not responding". I have managed to get it to run the complete cycle once in maybe 10-15 tries...

Image
Let>ME=%Script%

Running: 15.0.27
version history

User avatar
Grovkillen
Automation Wizard
Posts: 1131
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: CSVFiletoarray reading large file

Post by Grovkillen » Tue Oct 18, 2016 12:54 pm

I just realized that I use semicolon ";" as delimiter instead of comma ","

Could this be the problem (even though I get it to work in the editor)?
Let>ME=%Script%

Running: 15.0.27
version history

User avatar
Grovkillen
Automation Wizard
Posts: 1131
Joined: Fri Aug 10, 2012 2:38 pm
Location: Bräcke, Sweden
Contact:

Re: CSVFiletoarray reading large file

Post by Grovkillen » Wed Oct 19, 2016 5:40 am

I found out that a DelArray> just prior to the CSVFileToArray> was causing the problem. I just removed the DelArray> and it worked just fine. Sorry for flagging this as an issue.
Let>ME=%Script%

Running: 15.0.27
version history

Post Reply
Sign up to our newsletter for free automation tips, tricks & discounts