Best way to determine Number of records being processed.

[MGuyM]

I'm currently running a process that reads each line of a file and then performs some action on the data retrieved:

Code: Select all

Label>Start
  ReadLn>%strFileToProcess%.CSV,X,Line
  If>Line=##EOF##,End_Script
  Separate>Line,comma,LineValue

In my process, I'm incrementing X each time. What I'd like to know is how can I determine the total number of lines in my File so that I can produce a display window to show how many records of the total have been processed?

I've been reading upon on the different ways of looping, but I can't quite figure out if there is one that can determine the total number of records in the file (unless I read and loop through the entire file before I begin processing, which I think would be a duplicate effort, but may be the only way).

Any ideas?

Thanks!

[Marcus Tettmar]

ReadFile>filename,filedata
Separate>filedata,CRLF,lines
MessageModal>Number of lines: %lines_count%

[MGuyM]

Thanks Marcus!

Just had another thought.... Is there a Size Limitation to the Reading of a File? My File could have up to 100000 Lines in it and I just wonder if it would be too big?

Thanks!

[JRL]

I've used readfile to read files that are around 10 meg. Not that there is not a size limit on readfile> it probably has more to do with your computer's capability though and not a limit imposed by readfile>.

If Readfile>/Separate> is slow for you, see this thread, there are two more possibilities discussed.

[Me_again]

If you are dealing with very large files you may want to check whether the additional time needed for the extra step of scanning scanning the file to determine the number of lines is really worthwhile.

[pasdad1]

When dealing with VERY LARGE files, what I have done is to find out the size of the file in bytes (using the MS FileSize> command or DOS DIR command) and then pick a reasonable average line length to divide into this.

This gives a reasonably accurate guess as to the number of lines. The FileSize> or DOS DIR commands return INSTANTLY because they don't have to read the file first.... this info is already stored in the directory structure by the operating system.

On VERY LARGE files it usually doesn't matter that the guess by this method is off by a little, because you are usually only using this info to update a status bar or percentage done type of thing. Of course as you are actually processing the file (reading one line at a time in a loop) you are counting the lines actually processed, and comparing this to your estimated, computed total of lines.

[JRL]

Great idea pasdad1... I'll try it the next script I have needing line count. Being off a little won't hurt my feelings, usually I put the count in a message box as Line X of XXXX just so I have a feel for when I'll get my computer back.