Pixel recognition

freaky2k · Post by **freaky2k** » Tue Jun 13, 2006 12:11 am

Alright, I'm pretty much completely new to this, but would really like to get it. I have looked around these forums and in tutorials, but can't seem to really find anything...or at least don't understand it. I would like to compare small pieces of a picture to what is on the screen. If that picture is found on the screen, I would like it to be clicked. It sounds very simple to me, but I can't seem to figure out how to use image recognition or anything of the like. Can anyone point out to me where I can find out a piece by piece tutorial or something that is already done that is very similar to this that I can take a look at? If you don't know of any places/examples, anyone mind explaining to me how I can go about this? Thanks a bundle and I appologize if this is already on here somewhere.

Me_again · Post by **Me_again** » Tue Jun 13, 2006 12:57 am

You need to download and install Macro Scheduler, and the Image Recognition Library (which you can find here).

With the Image Recognition Library there is an example "Google Button Example" which captures the screen, and then looks to match a bitmap image "googlebutton.bmp", and click on its location. If I interpret your post correctly this is exactly what you want. To make it work for you just subsitute your own bitmap for googlebutton,bmp.

Also be sure you set the correct directories for your system.

Let us know how you get on and we can help you further.

freaky2k · Post by **freaky2k** » Tue Jun 13, 2006 1:17 am

Alright, thanks, that gets me started.
Not sure if this is going to be a problem or not, but the background changes...is that going to screw up the screen.bmp?
Also, right off the bat I get a problem with "Invalid Numeric Value for MouseMove command." Would this have to do with the size of my window (its not full screen)?

also, what is imgs referring to? such "Let>y=imgs_6+y"

Me_again · Post by **Me_again** » Tue Jun 13, 2006 2:06 am

The script grabs a new screenshot each time it runs so the changing background shouldn't be a problem, but when you make your "googlebutton.bmp" you should be careful to only capture part of the button, like the text, that is always the same, otherwise it can't find a match.

What do you have for the first line of the script? When I installed the image library it defaulted to a different directory so you may need to change that line to something like;

Let>imglibDLL=c:\Program Files\Macro Scheduler Image Library\imglib.dlll,imglib

LibFunc>imglib,FindImgPos,imgs, etc. returns the x coordinate of where it finds the match in imgs_5, and the y coordinate in imgs_6

freaky2k · Post by **freaky2k** » Tue Jun 13, 2006 2:14 am

The only things I changed were the image locations:

IfFileExists>screen.bmp
DeleteFile>C:\Program Files\MJT Net Ltd\Macro Scheduler\screens\screen.bmp
EndIf
ScreenCapture>X,Y,W,H,C:\Program Files\MJT Net Ltd\Macro Scheduler\screens\screen.bmp

LibFunc>imglib,FindImgPos,imgs,C:\Program Files\MJT Net Ltd\Macro Scheduler\screens\neo_1.bmp,d:\screen.bmp,20,1,ref:0,ref:0
If>imgs>0
//click on button
//remember that x,y returned is relative the image and since we didn't use the
//full screen we will need add top left offset to get screen coordinates
Let>x=imgs_5+x
Let>y=imgs_6+y
MouseMove>x,y
LClick
Endif

LibFree>imglib

I still don't understand imgs. Is 'imgs' a command in Macro Scheduler? I dont see imgs as being defined as anything...how does it know the image is imgs_5 or imgs_6....doesn' t make sense to me.

And I am still wondering why it's coming up with "Invalid Numeric Value for MouseMove command."

Me_again · Post by **Me_again** » Tue Jun 13, 2006 2:33 am

freaky2k wrote: I still don't understand imgs. Is 'imgs' a command in Macro Scheduler? I dont see imgs as being defined as anything...how does it know the image is imgs_5 or imgs_6....doesn' t make sense to me.

That's just the way the FindImgPos function works, you give it information of which image you are searching in, which image you are searching for, and it tells you the x y coordinates where it finds the image. "imgs" is just a name that the person who wrote the example chose for the variable,

LibFunc>imglib,FindImgPos,imgs,d:\googlebutton.bmp,d:\screen.bmp,20,1,ref:0,ref:0

you can change it to anything you want, here I used "qqq", in which case the x coord is in qqq_5 and the y coord is in qqq_6

LibFunc>imglib,FindImgPos,qqq,c:\ftps\googlebutton.bmp,c:\ftps\screen.bmp,20,1,ref:0,ref:0
MessageModal>%qqq_5% %qqq_6%

And I am still wondering why it's coming up with "Invalid Numeric Value for MouseMove command."

My guess is that you didn't edit the first line to point to the correct location for the image recognition library. Here's mine again:

Let>imglibDLL=c:\Program Files\Macro Scheduler Image Library\imglib.dlll,imglib

freaky2k · Post by **freaky2k** » Tue Jun 13, 2006 2:48 am

Alright, yea, I didn't see that first line. Fixed the URL. It works, although I'll definitly need to change some things...like make it loop and make it so it scans for more than one image. I feel like it scans quite slowly...is there a way to make it faster and almost instant?

Edit:
Okay, although it works, the x,y cords are slightly off. I'd like to edit this, but the way I am doing it just brings back the old error.
"MouseMove>x+20,y+20" I added the +20. Apparently this isn't correct coding for this program? I just want to add 20 pixels to the cords that it is getting back.

Me_again · Post by **Me_again** » Tue Jun 13, 2006 3:00 am

That's good

If the button is usually in the same general area of the screen then you don't need to capture/search the whole screen and that would be faster.

Are you sure you need to do image recognition? MacroScheduler has many other ways to find buttons in webpages. I just re-read your original post and I see it's not necessarily a button so this may not apply.

freaky2k · Post by **freaky2k** » Tue Jun 13, 2006 3:42 am

yea, they aren't actually buttons. it's in Flash and the screen is small so I didn't think it'd take too long. There are 8 general areas to scan and images will continue to pop up in those locations. How can I delegate the scans to certain areas instead of the whole screen?

Also could you tell me how to add onto the mouse cords? Like I wanted to add 20 pixels to x and y cords. When I try it my way it just gives me that old error. I'm use to simple java coding.

Post by **Marcus Tettmar** » Tue Jun 13, 2006 7:19 am

freaky2k wrote:Okay, although it works, the x,y cords are slightly off. I'd like to edit this, but the way I am doing it just brings back the old error.
"MouseMove>x+20,y+20" I added the +20. Apparently this isn't correct coding for this program? I just want to add 20 pixels to the cords that it is getting back.

You can't do that. You need to do:

Let>x=x+20
Let>y=y+20
MouseMove>x,y

pgriffin · Post by **pgriffin** » Tue Jun 13, 2006 1:44 pm

If I may add a couple of notes to this thread,

Why would you add 20 pixels to the coords? You could set the parm in the DLL call to locate the middle of the image. Isn't that what you are attempting?

When you are getting the errors associated with MouseMove, it is becuase you are passing numeric values to the MouseMove> in your code with are not numeric, which means some variable is not being resolved correctly.

freaky2k · Post by **freaky2k** » Tue Jun 13, 2006 8:21 pm

I no longer have the mousemove problem now that I have the correct way to add on to the cords.

Thanks for all the help guys.
Unfortunitly, I will not be able to use this in the end. I did not expect the scanning to take so long. It takes at least a second scanning for a single image and what I want would take multiple scannings almost constantly which this is simple way too slow in reaction time. So unless there is a way to dramatically increase the scan time, I don't think this will work out.