Pure Text Recognition - FireFox

[macroman]

Hi Guys,

Please give it to me direct and the most easiest way just to find a simple sentence on a website and put a mouse over it...

I'm trying to use the wizard and I get no luck what so ever! I have firefox focused, and trying to find the words "skip guided setup" for the past 24 hours and it just doesn't find it... let alone clicking on it... when using image recognition then it's all good 90% of the time.

[CyberCitizen]

The GetText functions work with many windows apps - it will work with any windows app which uses the Windows TextOut set of functions to render text. But not all apps do so it won't work with all apps. E.g. it doesn't currently read text from Chrome which uses proprietary text rendering methods. I don't know about FireFox but it could be the same. If you use the wizard and point it at the e.g. the desktop, notepad and explorer you'll see that it returns text. So it's a case of suck it and see. Not all apps produce text as far as Windows is concerned. This topic has been dealt with on my blog a number of times:

http://www.mjtnet.com/blog/2009/01/23/s ... scheduler/

[macroman]

Thanks, read that about 10 times... iexplorer had better results thann firefox... i figured it's just basic text that you can copy and paste... it should be easy... it's not like i'm trying to get text from an image....

[Marcus Tettmar]

It makes no difference whether you can copy and paste it.

What matters is how the text is being rendered. The text capture functions can only work if Windows' OWN TextOut functions are used to render the text.

These text capture functions work by "hooking" the Windows TextOut functions. We basically get to peek into them and see the text out buffers when they are called and so we see the text.

But if an app chooses not to use these functions and decides to render the text another way - remember text is just pixels on a screen - and uses it's own functions *which we can't hook* then we are BLIND.

Unfortunately FireFox doesn't use the windows text rendering functions to generate the text, so as far as Macro Scheduler is concerned it isn't text - it's just a bunch of pixels. It doesn't get generated by the Windows TextOut functions. So our hooks don't see them.

As far as Windows is concerned the text is not text. Nothing ever goes near the text generation functions of Windows so Windows never sees the text.

I know this is a hard concept to grasp as I spend a lot of time trying to explain it. Bottom line is if you can see text in the text capture wizard then you can use the text capture commands to get it it. If you don't, you can't. It's not broken, it's just not able to see text that isn't produced by Windows - that isn't text as far as Windows is concerned.

Clipboard operation is irrelevant here. How an app puts text on to the clipboard has nothing to do with how the text is rendered, and it's how it is rendered that allows us to "snoop" on it to see what is being sent to the screen.

Of course you could USE the clipboard - but that would involve sending keystrokes/messages to select it and copy it. You would then be able to use GetClipboard to get the clipboard contents to a variable.

IE does use Windows TextOut functions, so with IE we're good to go.

IE is far and away the best way to automate a web page.

Yeh, I hear you, I hate IE too. I prefer Chrome. I don't *personally* use IE. But as IE is good for automation I don't mind if Macro Scheduler uses it, and when Macro Scheduler is automating it I don't have to watch it - that's the whole point of automation. IE is ubiquitous and has a great automation API and is also more "accessible" (we can use the UI functions). So from Macro Scheduler's point of view it's the best one to use for automation.

[macroman]

I mean, if we do CTRL+F to find a text, whether it's in notepad or firefox... the window system finding feature can pin point it so quickly and easily... perhaps it's so easy to use we may think it's a piece of cake doing this with MS? I mean... there has to be a more solid way to search for a text that is "selectable" with a cursor....

[Marcus Tettmar]

Yes, use IE, not FireFox, then use the built in IE functions which can extract the underlying text and html from the page.

Just because Firefox implements a search feature which highlights text on their app, doesn't mean anything useful has been EXPOSED to other processes.

Use IE for the automation. You'll be able to automate it much more easily.

[deloppoled]

I've found that using the Firefox add-on called IE Tab 2 (FF 3.6+) v5.12.12.1 lets you work within Firefox by opening an instance of Internet Explorer within Firefox. I don't know if the entire IE (edit: object model) is exposed but I tried something like this and it worked no problem.

Code: Select all

//Place the cursor over some word in IE Tab 2 and run it
GetTextInit
GetCursorPos>mouseX,mouseY,cursorType
GetTextAtPoint>mouseX,mouseY,strText,vCharPos
MessageModal>Macro Scheduler working within Firefox thanks to IE Tab 2, oh yeah, I almost forgot--here's your string -- %strText%

I know that it works for capturing text which is what I needed to do today. I've been using this addon for quite a long time, I'm not sure how many years, but it does work well when a website just doesn't want to display correctly in Firefox--I know the addon is stable. In fact it is used by 662,805 with 4/5 Stars with 379 reviews.

It does say "Fully Supported" on the add-on page, so who knows. Further testing is probably in order! This does seem to open up a nice door pretty widely for FireFox users.

Oh and by the way, IE Tab 2 works in Chrome too!

Does anyone want the source code? Sure, why not!
IE Tab v2 is GPL licensed and the source code is hosted here: http://code.google.com/p/ietabv2/

Seems like this could open up some new doors and that's always a good thing!

All in all, pretty darned neat, from a Firefox addict's point of view.

Del

[Marcus Tettmar]

IE Tab just wraps an IE document window inside a tab in Firefox/Chrome. The upshot being that for all intents and purposes you are using IE.

So why not just automate IE in the first place?

[deloppoled]

I am a home user so I've been able to live without IE, which I can understand that many business users could never get away with doing as businesses tend to use IE, which really is fine for them.

Personally, I have never run IE on a regular basis, it's never been my browser of choice. I was in college when the first web browser, Mosaic came out. From there I used Netscape never falling off the boat to this day. Now I use Firefox. Sure I have kept a copy of IE on every PC I own for the sites that I need to visit that don't support (proper HTML) Firefox. But before I found the IE Tab, years ago, I had to open a copy of IE by <Start><Programs><Internet><Rarely Used><Microsoft><IE>IE which is (still) rather cumbersome. I know I could move it up in the start menu tree but I don't need too, I prefer Firefox because of it's flexibility and it can run IE inside of it, so why would I open an additional program?

Anything that I want my browser to do, Firefox will do it itself or via one of the user submitted, Mozilla approved add-on. One configures Firefox into the exact browser that one wants. While I did not write Firefox or the thousands of add-ons, I've gathered them together in such a way that my Firefox setup is quite specific to me. I just can't do that with IE without taking the time to code something myself--why do that when someone else has kindly done it for me and the community?

I guess that I am just old school. IE appeared after the whole web scene got going, so to me it was always an afterthought and they (Microsoft) were seen as jumping on the bandwagon. I realize that IE does expose its' object model very nicely and I have had to use IE and it's object model when I worked for big business, but at home, rarely. if you want to automate IE and one is a Firefox user, why would one want to open a separate program/browser when Firefox can open IE Tab for me, via a simple context menu setting?

Preferences, everybody has them!

Best wishes.

[Marcus Tettmar]

I understand all that. I use Chrome. I almost never use IE. Myself.

But when I need to automate something I let Macro Scheduler use IE. IE is part of the system. It's intrinsic. The web browser control is an ActiveX object that other apps can use. So it is great for automation.

When people automate uploading data to a web form, they generally have it run over night and do not sit and watch it. They might even have a dedicated PC do the automation.

I totally get why *YOU* would prefer to use FireFox. But why prevent *Macro Scheduler* from using the IE browser for automation.

*YOU* would use copy and paste when you copy stuff out of Excel and into a web form. Macro Scheduler would be better off accessing the data in Excel directly via Excel's COM object.

When *YOU* want the latest file in a folder you'd open up Windows Explorer, navigate to the folder and sort it by newest first. But that is cumbersome for Macro Scheduler. Instead a macro would use the GetNewestFile command.

My point is that what is right for YOU isn't necessarily best for Macro Scheduler.

I wouldn't DREAM of telling YOU to use IE instead of Firefox. But my job here is to help you AUTOMATE stuff, and in terms of using Macro Scheduler to automate the web, *IT* prefers to automate IE. While it's working don't look at it. And when it's done have it kill IE and open Firefox up for you.

[deloppoled]

Thank you for your reply Marcus.

Google Chrome? Nooooooo! I can't even bring myself to go to google dot com! They seem to know too much about me already. I don't care for their auto-updating the browser without even asking, etc.. But, if one can get past all that, Chrome is nice. And that is of course individuals choice! Some may have a sour taste for Mozilla politics, or Microsoft's I don't know, richness?! It's all about preferences and if it's not business critical, what's important to an individual. If it is about business, what is best for the business.

I can completely understand your point and I agree with you that for automation, Macro Scheduler and IE are a perfect match. In fact if I were using Macro Scheduler in an office environment, I am sure that I would be using those tools together for many different things, everything that needed to be automated between the two for that matter.

But as a home user, I don't have much I need to automate over the Internet and I don't even have a home Intranet (anymore!). I use a password manager, so that takes care of that. It really left me only one thing that i wanted to automate at this time and for me the IE Tab 2 works well for that.

I would certainly not recommend that business users flock over and drop IE for IE Tab 2, but I would recommend home users who prefer Firefox and don't automate much over the web, to take a look at IE Tab 2 add-on since it works just fine. I am thinking more in a home-user (Firefox biased ) sort of way.