Issue with Multiple While Commands?

[RNIB]

Apologies for the length of this but I've got myself completely confused and have gone 'code blind'. I just can't see what I'm doing wrong. I'm wondering if it's because I'm using lots of while commands which themselves are enclosed within a Repeat Until loop but really not sure.

I've got this macro which performs 10 different checks on the metadata of an HTML file. The idea is you point the macro at a drive containing thousands of folders each of which contains a separate HTML file and it works it way through them all performing each of these 10 checks to each html file in turn. The whole macro runs between a Repeat and Until command as it works through each folder.

The complete code is too lengthy to post (over 3000 lines) so just posting a section of it as it's really just variations of this throughout.

Each of the different checks is a separate '1st level' subroutine. Depending on the result of the Subroutine there are then three possible other '2nd level' Subroutines that run.

This is one such combination of a 1st level SRT and a single 2nd level SRT

Code: Select all

//1st Level SRT
SRT>nccFiles
Let>nccFC=0
While>strFCPos<>##EOF##
Label>FindFC
Let>nccFC=nccFC+1
ReadLn>%Path%\ncc.html,%nccFC%,strFCPos
Trim>strFCPos,strFCPos
MidStr>strFCPos,13,9,strFCPosTrim
If>{(%strFCPosTrim%="ncc:files") AND (%nccFCLineCount%=0)}
//When it finds the ncc:files line save the line number as nccFCLine1
Let>nccFCLine1=%nccFC%
//Increment nccFCLineCount to indicate that 1 instance of ncc:files has been found so far
Let>nccFCLineCount=1
//Read the line in the ncc file that has the 1st instance of ncc:files
ReadLn>%Path%\ncc.html,%nccFCLine1%,strFCValue1
LTrim>strFCValue1,strFCValue1
Let>FCpattern=ncc:files.+?content="\K[^"]+
RegEx>FCpattern,strFCValue1,0,FCMatches1,nm1,0
//If nothing had been entered into the metadata set FCMatches1_1 to 0 to identify it needs changing
If>nm1=0
Let>FCMatches1_1=0
Endif
Trim>FCMatches1_1,FCMatches1_1
If>FCMatches1_1=
Let>FCMatches1_1=0
Endif
Goto>FindFC
Endif
If>{(%strFCPosTrim%="ncc:files") AND (%nccFCLineCount%=1)}
//If it finds a 2nd line save the line number as nccFCLine2
Let>nccFCLine2=%nccFC%
//Increment nccFCLineCount to indicate that 2 instances of ncc:files have been found
Let>nccFCLineCount=2
//Read the line in the NCC that has the 2nd instance of ncc:files
ReadLn>%Path%\ncc.html,%nccFCLine2%,strFCValue2
LTrim>strFCValue2,strFCValue2
Let>FCpattern=ncc:files.+?content="\K[^"]+
RegEx>FCpattern,strFCValue2,0,FCMatches2,nm1,0
//If nothing had been entered into the metadata set FCMatches2_1 to 0 to identify it needs changing
If>nm1=0
Let>FCMatches2_1=0
Endif
Trim>FCMatches2_1,FCMatches2_1
If>FCMatches2_1=
Let>FCMatches2_1=0
Endif
Endif
EndWhile
END>nccFiles

//Work out which Level 2 SRT to use
//If the ncc:files line is missing entirely from the metadata
If>nccFCLineCount=0
GoSub>MissingNCCFiles
Endif
//If there is only 1 instance of ncc:files in the metadata
If>nccFCLineCount=1
GoSub>SingleNCCFiles
Endif
//If the ncc:files line appears twice in the metadata
If>nccFCLineCount=2
GoSub>MultipleNCCFiles
Endif

//Level 2 SRT
SRT>MissingNCCFiles
SetFocus>Notepad++*
//Open the GoTo command in Notepad++
Wait>1
Press Ctrl
Send>g
Release Ctrl
WaitWindowOpen>Go To...
Wait>0.2
//Line 25 will always be within the metadata area. It doesn't matter where the ncc:files line
Send>25
Wait>0.2
Press Enter
Wait>1
SetFocus>Notepad++*
Wait>0.5
Press Enter
Wait>0.2
//Add the metadata line with the correct file count value
Send><meta name="ncc:files" content="%strNames_Count%"/>
Let>NCCFileReport=Added Missing ncc:files metadata tag
END>MissingNCCFiles

So basically the Level 1 SRT looks to see if a specific tag exists in the HTML file and if it does, it captures the value. If the tag doesn't exist the Level 2 SRT adds it in. Equally, if the tag existed but was incorrectly formed, another Level 2 SRT corrects it. Whilst if the tag appears more than once, a third Level 2 SRT removes the duplicate and corrects the remaining if required.

The whole macro consists of 10 sets of Level 1 and Level 2 SRT's like this.

The macro works fine if I just run a single Level 1 SRT. It works fine whether it's adding a missing tag, correcting an existing or removing and correcting a duplicate.

It also works fine if I run all the Level 1 SRT's as long as none of them had to remove a duplicate. If they do, the line number count goes out of sequence and the macro starts adding/deleting the wrong lines.

For example, lets say that in the html file the metadata line <meta name="ncc:files" content="64"/> appears twice on lines 16 and 17 (but has the wrong value on line 16 - should be 70 and not 64). The next level 1 SRT then needs to make a change to another metadata tag on line 20.

What happens here is that the macro will delete line 17 and correct line 16 to have the content of 70 and not 64. However, the line that was originally on line 20 is now on line 19 but the next Level 1 SRT still tries to correct the tag on line 20.

I would completely understand this if all the level 1 SRT's were running at the same time but I thought I had them running one after the other:

Code: Select all

//Level 1 SRT
GoSub>nccFiles

//Level 2 SRTS
//If the ncc:files line is missing entirely from the metadata
If>nccFCLineCount=0
GoSub>MissingNCCFiles
Endif
//If there is only 1 instance of ncc:files in the metadata
If>nccFCLineCount=1
GoSub>SingleNCCFiles
Endif
//If the ncc:files line appears twice in the metadata
If>nccFCLineCount=2
GoSub>MultipleNCCFiles
Endif

//Level 1 SRT
GoSub>ncckByteSize

//Level 2 SRTS
//If the ncc:kByteSize line is missing entirely from the metadata
If>nccKBLineCount=0
GoSub>MissingncckByteSize
Endif
//If there is only 1 instance of ncc:kByteSize in the metadata
If>nccKBLineCount=1
GoSub>SinglencckByteSize
Endif
//If the ncc:kByteSize line appears twice in the metadata
If>nccKBLineCount=2
GoSub>MultiplencckByteSize
endif

So the Level 1 SRT should only read the html and get the line numbers after the previous Level 2 SRT's have completed shouldn't they?

I've also found that after it's checked the first html file in the first folder, when it moves to the second html file it just gets things completely wrong and just runs the various Missing level 2 SRT's even though the relevant tags aren't missing. This is despite the fact at the start of each loop I delete the contents of all the variables that are set in the various Level 1 SRT's so that everything starts afresh again.

I don't know if any of that made any sense whatsoever, but does anyone have any idea why this might be happening?

[RNIB]

Well I solved the problem of why subsequent SRT's were changing the wrong lines. I had a spelling mistake in the line that saves the html file each time. As a result subsequent SRT's weren't working on the updated file

What I still haven't worked out is that it now works fine with a single HTML file but when it loops to a second file, it breaks and it no longer seems to correctly read the HTML file in that it thinks that its missing all the things it checks for even though they are present.

[Dorian (MJT support)]

My first thought is to turn on logging and then carefully investigate the logfile. You should be able to see exactly how the script runs and what's going awry. Pay close attention to unexpected variables. It can be painstaking but being able to "look under the hood" in that way is invaluable.

[jonathangoss]

hi
I would echo what Dorian says there - even having your own custom log file - so you can track down exactly where your script is going off piste.
Like he says - a bit of a pain and slow to setup - but worth it for this type of troubleshooting on such a big macro
thx
Jon

[RNIB]

Well I'm slowly getting there. I've managed to find 3 errors so far and the REALLY annoying thing is that if I step through the macro it works PERFECTLY. However, if I run it, the first couple of SRT's run fine and then it just thinks it can't find any of the meta tags and just runs the SRT's that add them if they are missing.

I've tried slowing it down by using SK_Delay and adding in countless wait commands but it doesn't seem to make a difference.

I'm obviously being more dense than normal but for the life of me I can't seem to get logging to work. I've checked the two check boxes, selected a location, tried each of the log levels but nothing gets created.

[Dorian (MJT support)]

Hunch : Exit the editor and then run it.

[jonathangoss]

hi
I am also recommending you do your own logging
Note - you can end up with war and peace and Marcus who owns MJT Net has sometimes laughed out loud when he has seen my log files

e.g = something like this:
Day>the_day
Month>the_month
Year>the_year
let>LogFileNamelogfile=E:\ConsultantConnectERS\LogFile_ConsultantConnectERS_EdocumentsMover_DailyLogFile%the_year%%the_month%%the_day%.txt
TimeStamp>%LogFileNamelogfile%,---------START OF MACRO----------

you can do things like state when you enter and exit a sub routine via the TimeStamp command. Up to you how detailed you go with it.....
in your case - you can also trap if you found a web element and also sometimes I loop for a while to keep looking for the element (as network & website speed may be an issue) like this:
//find and hit login button
EdgeFindElements>session_id,xpath,XPathLoginButton,strElementID4
TimeStamp>%LogFileNamelogfile%,strElementID4_COUNT is %strElementID4_COUNT%
//loop to check and wait for it to be there
let>T=0
Label>LoginToESR
SetDialogProperty>Dialog2,Label1,Caption,Waiting for Login button to be there and timerCounter is %T%
wait>2
If>strElementID4_COUNT=0
Let>T=T+1
Wait>2
SetDialogProperty>Dialog1,Label1,Caption,Waiting for Login button to be there and timerCounter is %T%
goto>LoginToESR
Else>
//it is there so click it!
EdgeElementAction>session_id,strElementID4_1,click
EndIf>
//Also you could do a counter here - so that you don't wait forever and if required, do a nice exit...

Hope that helps
thanks
Jonathan

[Dorian (MJT support)]

Do you also use _Dump_Vars, Jon? That really is going the whole hog.

[RNIB]

I will take a look at trying to create a custom log file but I'm struggling a bit with understanding how to do this and what needs to be captured.

In the mean time I believe I've found the the area of code that is causing the problems, just not exactly which bit of code it is or why. I think it's to do with all my "level 1 SRT".

This is the code of one of them (they are all variations of the same code just with different variables). Excuse the heavy annotation but was trying to make it clear to myself what each bit was doing.

Code: Select all

SRT>dcIdentifier
Let>dcIdent=0
//EOF=End of File. This will search through the entire NCC.html until it reaches the end of the file
While>strDCIdentPos<>##EOF##
Label>FinddcIdent
Let>dcIdent=dcIdent+1
ReadLn>%Path%\ncc.html,%dcIdent%,strDCIdentPos
//Trim and Midstr trim off excess characters from the line of text in the metadata.
//so that when found, strDCDatePosTrim should just include the characters dc:indentifier
Trim>strDCIdentPos,strDCIdentPos
MidStr>strDCIdentPos,13,13,strDCIdentPosTrim
If>{(%strDCIdentPosTrim%="dc:identifier") AND (%dcIdentLineCount%=0)}
//When it finds the dc:indentifier line save the line number as dcIdentLine1 as it will continue counting to EOF afterwards
Let>dcIdentLine1=%dcIdent%
//Increment dcIdentLineCount to indicate that 1 instance of dc:identifier has been found so far
Let>dcIdentLineCount=1
//Read the line in the ncc file that has the 1st instance of dc:identifier and store the line as strdcIdentValue1
ReadLn>%Path%\ncc.html,%dcIdentLine1%,strDCIdentValue1
//Trim excess characters from the string and use Regular Expression to remove everything
//but the value between the " ". This then gives us the value of the first dc:identifier line
LTrim>strDCIdentValue1,strDCIdentValue1
Let>dcIdentpattern=dc:identifier.+?content="\K[^"]+
RegEx>dcIdentpattern,strDCIdentValue1,0,dcIdentMatches1,nm5,0
//If nothing had been entered into the metadata we need to set dcIdentMatches1_1 to 0 to identify it needs changing
If>nm5=0
Let>dcIdentMatches1_1=0
Endif
Trim>dcIdentMatches1_1,dcIdentMatches1_1
//If a space or spaces had been entered into the metadata we need to set dcIdentMatches1_1 to 0 to identify it needs changing
If>dcIdentMatches1_1=
Let>dcIdentMatches1_1=0
Endif
Goto>FinddcIdent
Endif
If>{(%strDCIdentPosTrim%="dc:identifier") AND (%dcIdentLineCount%=1)}
//When it finds a 2nd line save the line number as dcIdentLine2 as it will continue counting to EOF afterwards
Let>dcIdentLine2=%dcIdent%
//Increment nccFCLineCount to indicate that 2 instances of dc:identifier have been found
Let>dcIdentLineCount=2
//Read the line in the NCC that has the 2nd instance of dc:identifier and store the line as strdcIdentValue2
ReadLn>%Path%\ncc.html,%dcIdentLine2%,strdcIdentValue2
//Trim excess characters from the string and use Regular Expression to remove everything
//but the value between the " ". This then gives us the value of the second dc:identifier line
LTrim>strdcIdentValue2,strdcIdentValue2
Let>dcIdentpattern=dc:identifier.+?content="\K[^"]+
RegEx>dcIdentpattern,strdcIdentValue2,0,dcIdentMatches2,nm5,0
//If nothing had been entered into the metadata we need to set dcIdentMatches2_1 to 0 to identify it needs changing
If>nm5=0
Let>dcIdentMatches2_1=0
Endif
Trim>dcIdentMatches2_1,dcIdentMatches2_1
//If a space or spaces had been entered into the metadata we need to set dcIdentMatches2_1 to 0 to identify it needs changing
If>dcIdentMatches2_1=
Let>dcIdentMatches2_1=0
Endif
Endif
EndWhile
END>dcIdentifier

If I use Debugger to step through the entire macro, it works flawlessly. Therefore I was thinking that it must be a speed issue and the macro was running faster than the applications etc could keep up. So I added a whole bunch of wait commands in the "level 2 SRT's" wherever it was interacting with the Notepad application and saving files etc. This made no difference.

This led me to wonder if the issue was with the speed of the Level 1 SRT's. So I added a small wait command after each ReadLn. Initially I just used Wait>0.2. When I then ran the macro I found the whole thing ran incredibly slowly. There was around a 20 second (guestimated) delay between each step of the macro. If I remove the Wait commands, it runs at normal speed. I tried again this time adding a Wait>1 but the same thing. The speed wasn't noticeably slower than when I had used 0.2 but it was as slow. Again removing the wait commands fixes it. It also doesn't seem to matter which SRT I put the wait command in. If I just put them in 1 SRT the whole macro still runs too slow.

I don't understand how telling the macro to wait for 0.2 seconds after it reads a single line in a file can result in a 20 second delay between every step of the macro??? Obviously there must be something drastically wrong with my code but I've no idea what.

[Dorian (MJT support)]

I don't understand how telling the macro to wait for 0.2 seconds after it reads a single line in a file can result in a 20 second delay between every step of the macro??? Obviously there must be something drastically wrong with my code but I've no idea what.

Because you're reading the file line by line in a While/EndWhile loop. You're not adding one single 0.2 second wait, you're adding 0.2 seconds on every single loop - so for example if your file has 100 lines you're waiting 0.2 seconds 100 times, every time you have a 0.2 seconds wait after a ReadLn.

You would see this in the logfile.

Is it possible you can rewrite your script using ReadFile instead?

[RNIB]

Oh OF COURSE

I'll have a look at ReadFile. I've always avoided it because I was never sure how to then find specific content within it.