Dealing with Remote Desktop screens

[SandyDoss]

Hello,

We've written a number of scripts to interact with a Remote Desktop session, and have been pretty successful in accomplishing what we need to accomplish using MacroScheduler.

We've started down the path of automating repetitive data entry tasks to said RDP - our first attempt was so successful (yay!) that a second has been requested.

My issue in this second attempt is that a selection needs to be made from a window that displays my choice(s) based on entry into separate fields, but cannot be accessed using the same keystrokes (like a dropdown). In other words, most dropdowns will allow you to send text (or enough text) so that you can isolate the desired selection without having to 'point and click'. Not so in this case.

So, I'm looking for a way to compare a string of text (that, with some footwork can (probably) be formatted to look like the selections displayed) to the available selections in the window, identify where my string is located in the window, position the mouse, and 'click' to select it.

Since it is Remote Desktop, I'm thinking OCR (which isn't my first choice, but may be my only choice). I've attempted Tesseract. The amount of time in capturing the image and converting to text is significant in that it would have to be done literally hundreds of times (at an average of 10 seconds per attempt), and (more importantly) the text file I'm getting is not making sense. That may be due to ignorance on my part and I'll continue to explore that. Any other suggestions or methods I might have missed in going through the forums/blog?

Thanks so much for any help given.

Sandy

[Marcus Tettmar]

Sounds like a nightmare. I suppose you could OCR the remote desktop screen. You'd presumably need to OCR the screen. If a match it's the selected item, if not, press down, OCR ... rinse and repeat.

Can't you just install Macro Scheduler inside the remote machine and be done with all that?

RDP works by sending screen shots in one direction and mouse/keyboard in the other. The clipboard can also be sent but you can't normally copy listboxes to the clipboard (unless it's an editable combo box where you can copy the selected item from the editable area).

Could you create a simple macro, compiled to a .exe, which cycles through the drop down grabbing all the options and constructing a delimited string which it places on the clipboard? You could launch that locally (on the remote machine) to have that interrogate the drop down and send the info back that you need ?

[SandyDoss]

Nightmare doesn't even come close. Unfortunately, I live for challenges like this.

You asked, "Can't you just install Macro Scheduler inside the remote machine and be done with all that?"

No. The machine is owned by a third party, proprietary. We haven't had much success with the owners and sharing

I haven't tried placing an executable on the machine but will consider that.

You've touched on what I think will be the final solution, though. I've found that once I limit my selections based on input, I can arrow down to highlight the first selection, ctrlC to copy, paste to notepad, and see an internal 'key' (NOT the data displayed on the screen). I can use the key as access to a database we keep on the local side (they may not be cooperative but I have my ways) to look up the single piece of data I'm missing as input that would match it. I'm assuming that the ctrlC puts it on MY clipboard, as opposed to something on the remote, but I'm a little ignorant in this area. Worth a try, though.

Again, this would be a "rinse and repeat" process (but maybe not, if I simply recycle through the list and build that string) but would cut out the 10 seconds or so I get using OCR on each entry.

Thanks, Marcus. It's nice to have a sounding board! I'll post what I end up doing.

[SandyDoss]

So, just to follow up on this, I did end up copying the key code behind the display in the select box and pulling it from the clipboard, and bouncing that against a MySQL database to determine if this was, indeed, the selection that I wanted.

Thanks again for all your help.