Find position of listing in search engines

Snowycat · Post by **Snowycat** » Thu Jun 01, 2006 1:11 pm

I am using the trial version of MacroScript WebRecorder and I wonder if it is possible to simulate the following....

I choose a search engine, say Yahoo, then search for a keyword or phrase. When the page of results appear, I use the 'Edit/Find on this page' feature to look for a specific URL in the results. Once found I note the position.

If not on the first page of results I go to next page and keep searching until URL found or the results come to an end. I usually set the number of results per page to 100.

I have managed to do some of the above with WebRecorder but I cannot work out if it is possible to stop the search when the URL is found, or how to repeat the process for the next page of results.

I guess someone has done this before if it is possible.

Thanks in advance.

Snowycat · Post by **Snowycat** » Wed Jun 07, 2006 11:09 am

Well I suppose it is not possible, judging by the nil response.

Ah well.

It's the main reason I wanted to buy.

Post by **Marcus Tettmar** » Wed Jun 07, 2006 5:31 pm

Hi,

It is possible. Sorry for the delay. For priority support don't forget you can contact us via http://www.mjtnet.com/contact.htm

It can be done a number of ways. One way is to use WebRecorder. You would need the ExtractTag function in a loop, checking each result for your URL. Another way would be to walk the Document Object Model with VBScript. Another way would be to control your browser at the user interface level but finding the location of the URL would be tricky that way. You could parse the source of the page but that would be long winded. Easier to use WebRecorder/ExtractTag or VBScript and work at a lower level with the DOM.

Snowycat · Post by **Snowycat** » Wed Jun 07, 2006 6:09 pm

OK Thanks Marcus.

That means I must buy.

I might need a bit more guidance though.

John Wakefield
Snowycat

Me_again · Post by **Me_again** » Wed Jun 07, 2006 7:03 pm

There are also SEO websites that do this for you, like http://www.mikes-marketing-tools.com/ranking-reports/, so rather than hit many search engines separately you might consider automating just one of those.

Post by **Marcus Tettmar** » Wed Jun 07, 2006 7:38 pm

Yeh that makes a lot more sense and will simplify the steps. All you'd need is a WebRecorder macro which enters the URL and key phrase into that Search Engine Rankings tool and then extracts the result. Less work as there is less to extract. You can then loop back to the start for another key phrase/URL combination. Could feed the script with key phrases/URLs from a text file or database, and output the results for each to a text file. Using the above ranking tool this would be quite simple.

Here's a script that just grabs the rankings for one URL/key phrase using this ranking tool:

Code: Select all

// Generated by MacroScript WebRecorder 1.70
// Recorded on Wednesday, June 7, 2006, at 08:19 PM
LibLoad>IEAuto.dll,hIE
If>hIE=0
  MessageModal>Could not load IEAuto.dll, make sure it is in the path or edit the LibLoad line.
  Goto>end_script
EndIf

//Move the mouse cursor out of harm's way to avoid causing mouseover events to interrupt
MouseMove>0,0
Let>delay=1

LibFunc>hIE,CreateIE,IE[0],0

LibFunc>hIE,Navigate,r,%IE[0]%,http://www.mikes-marketing-tools.com/ranking-reports/
LibFunc>hIE,WaitIE,r,%IE[0]%
Wait>delay

Label>StartQuery

Let>FrameName={""}
Let>FormName={"page"}
Let>FieldName={"url"}
Let>FieldValue={"http://www.mjtnet.com"}
LibFunc>hIE,FormFill,r,%IE[0]%,str:FrameName,str:FormName,str:FieldName,str:FieldValue,0

Let>FrameName={""}
Let>FormName={"page"}
Let>FieldName={"keywords"}
Let>FieldValue={"windows macro"}
LibFunc>hIE,FormFill,r,%IE[0]%,str:FrameName,str:FormName,str:FieldName,str:FieldValue,0

Let>FrameName={""}
Let>FormName={"page"}
Let>TagValue={""}
LibFunc>hIE,ClickTag,r,%IE[0]%,str:FrameName,str:FormName,INPUT,NAME,str:TagValue

libFunc>hIE,WaitNewIE,IE[1],0
LibFunc>hIE,WaitIE,r,%IE[1]%
Wait>delay

Let>TD11_SIZE=128
LibFunc>hIE,ExtractTag,r,%IE[1]%,,TD,11,0,TD11
MidStr>r_6,1,r,GOOGLE

Let>TD12_SIZE=128
LibFunc>hIE,ExtractTag,r,%IE[1]%,,TD,12,0,TD12
MidStr>r_6,1,r,YAHOO

Let>TD13_SIZE=128
LibFunc>hIE,ExtractTag,r,%IE[1]%,,TD,13,0,TD13
MidStr>r_6,1,r,MSN

Let>TD14_SIZE=128
LibFunc>hIE,ExtractTag,r,%IE[1]%,,TD,14,0,TD14
MidStr>r_6,1,r,AOL

Let>TD15_SIZE=128
LibFunc>hIE,ExtractTag,r,%IE[1]%,,TD,15,0,TD15
MidStr>r_6,1,r,ALTAVISTA

MessageModal>Results:%CRLF% Google: %GOOGLE% %CRLF% Yahoo: %YAHOO% %CRLF% MSN: %MSN% %CRLF% AOL: %AOL% %CRLF% Alta Vista: %ALTAVISTA%

//Close the results window
LibFunc>hIE,KillIE,r,IE[1]

//At this point you could loop back to StartQuery with new data

LibFree>hIE
Label>end_script

This just outputs a message box showing the results, but of course you could have the script write the results to a file or database, or whatever it is you want to do with them.

Snowycat · Post by **Snowycat** » Wed Jun 07, 2006 8:46 pm

It might sound like a perfect answer but my whole reason for doing it without using a piece of software is that search engines, especially Google, forbid using such tools.

If I enter a URL in a ranking report facility, Google records it and then the URL gets banned.

I just want to emulate me doing a search without software.

Post by **Marcus Tettmar** » Wed Jun 07, 2006 9:09 pm

This isn't software. It's a web site. Please read the above two or three posts again. It is suggesting using the following site:

http://www.mikes-marketing-tools.com/ranking-reports/

I should think if Google were to ban this it would have been banned a long time ago as you can go to this page and do a rank check easily. The script I posted makes use of this site. Not software.

Edit: In any case whether you choose to go this route or code a script that works against Google the techniques shown in my example script will still be valid and useful - the script I posted serves as an example of some techniques available (navigating to a page, completing a form, extracting elements from the page).

Snowycat · Post by **Snowycat** » Thu Jun 08, 2006 8:37 am

You say that it isn't software but a website. The website must use software to perfom a search though.

They use your URL, which is then vulnerable to detection and banning. They, of course, won't be banned because it is not their URL.

From Google's terms and conditions, you will see that you may not send automated queries of any sort to Google's system without express permission in advance from Google. Same applies to many other search engines.

Automated softwares usually send a user-agent header which Google can
easily detect. Google can also detect speed which is a dead give away. By this I mean, sending multiple queries of the same keywords (page 1-100,
101-200 and so on) in rapid succession is very easy for a search engine to detect.

The use of a macro, however, is not going to be detected because the speed of the search would be kept down to normal human keypress speed.

It will also not be necessary to give the full URL to search for, just a portion of it.

Of course I already know about automated search services, as I do SEO. The big problem is that using such a tool is risking my clients' websites from being penalised.

Thanks for your suggestions and solving the problem. A few quid in the post!

John

Post by **Marcus Tettmar** » Thu Jun 08, 2006 9:02 am

Google provide an API for programmers to do stuff like this and last time I checked you could get a Google API key which permitted you to query their system in this way.

Well it was just a thought to use a ranking site to cut down the steps, but you could still write a macro to drive Google's site directly.

Snowycat · Post by **Snowycat** » Thu Jun 08, 2006 9:38 am

Yes, a Google API is one option, except that I need to work with plenty more search engines.

Regards

John

BTW Have now purchased.

Post by **Marcus Tettmar** » Thu Jun 08, 2006 9:59 am

Hi,

Ok, I just spent the last couple of hours building the following script. This takes a keyword and a URL and then performs a search on Google and scans the results to find what position that URL is in. The URL can be a partial URL - so it does a partial match. As is it will see where mjtnet.com comes up for the keywords windows macro (#1 of course!).

Code: Select all

//Set number of pages to scan, query and URL to find
Let>PagesToScan=3
Let>Keywords=windows macro
Let>URLToFind=www.mjtnet.com

LibLoad>IEAuto.dll,hIE
If>hIE=0
  MessageModal>Could not load IEAuto.dll, make sure it is in the path or edit the LibLoad line.
  Goto>end_script
EndIf

//Move the mouse cursor out of harm's way to avoid causing mouseover events to interrupt
MouseMove>0,0
Let>delay=1

LibFunc>hIE,CreateIE,IE[0],0

//Go to Google.com
LibFunc>hIE,Navigate,r,%IE[0]%,www.google.com
LibFunc>hIE,WaitIE,r,%IE[0]%
Wait>delay

//Enter a search term
Let>FrameName={""}
Let>FormName={"f"}
Let>FieldName={"q"}
Let>FieldValue=Keywords
LibFunc>hIE,FormFill,r,%IE[0]%,str:FrameName,str:FormName,str:FieldName,str:FieldValue,0

//Click the Search button
Let>FrameName={""}
Let>FormName={"f"}
Let>TagValue={"btnG"}
LibFunc>hIE,ClickTag,r,%IE[0]%,str:FrameName,str:FormName,INPUT,NAME,str:TagValue

LibFunc>hIE,WaitIE,r,%IE[0]%
Wait>delay


//Scan through all texts looking for results ....
Let>rank=0

Let>PagesDone=0
Label>ScanPage

Let>p=0
Label>scan
  Let>P%p%_SIZE=4098
  Let>P%p%=DUMMY
  LibFunc>hIE,ExtractTag,r,%IE[0]%,,FONT,p,0,P%p%
  MidStr>r_6,1,r,this
  Pos>k -,this,1,IsResult
  Pos>Cached - Similar pages,this,1,BigText
  If>{(%IsResult%>0) AND (%BigText%=0)}
     Let>rank=rank+1
	 //Diagnostic showing results as scanned
     //MessageModal>Pos %rank%: %this%
	 Pos>URLToFind,this,1,IsFound
	 If>IsFound>0
	    MessageModal>Found %URLToFind% in Position: %rank%
	  	Goto>Finish
	 Endif
  Endif
  Let>p=p+1
  Pos>Result Page:,this,1,en
  If>en=0,scan
Label>end_scan

//Click the Next button
Let>FrameName={""}
Let>FormName={"f"}
Let>TagValue={%CRLF%+"Next"}
LibFunc>hIE,ClickTag,r,%IE[0]%,str:FrameName,str:FormName,A,TEXT,str:TagValue
LibFunc>hIE,WaitIE,r,%IE[0]%
Wait>delay

Let>PagesDone=PagesDone+1
//Loop back and do next page if required 
If>PagesDone<PagesToScan>Finish
//Comment next line out if you don't want IE to close
LibFunc>hIE,KillIE,r,IE[0]

LibFree>hIE
Label>end_script

This might not be quite what you want but I hope it demonstrates what you can do and the techniques you can use.

The layout of Google is dynamic as some result pages can have sponsored links at the top etc. This gets round that by looking through all FONT elements and using some simple rules that determine whether it's a result or not based on what is in the text. It stops when it finds "Result Page:" which appears at the end. And ignores anything that doesn't follow my simple rules. Be warned - if Google change the layout and format of their page the script will likely need recoding.

Snowycat · Post by **Snowycat** » Thu Jun 08, 2006 11:57 am

Sincere thanks.

John

Post by **Marcus Tettmar** » Mon Jun 12, 2006 9:19 am

I have made a few small tweaks to improve the accuracy of my example rank checker script and have also posted it to Scripts & Tips for the benefit of others. Please see:

http://www.mjtnet.com/forum/viewtopic.php?t=2983

Snowycat · Post by **Snowycat** » Sat Jun 17, 2006 10:24 pm

I have been using this macro and it is brilliant. Thanks again.

I tried to make some changes to be able to use it for Yahoo but to no avail. Any chance of some help here?

John