Is there a way to detect if a sound occurred in an app

sailor dude · Post by **sailor dude** » Thu Jun 03, 2010 4:11 pm

I am automating a web based client application used to access a database.

It requires IE7 to be used as the client.

It uses Java script.

It is very difficult to determine when the Java script used in the client has completed a task.

VB script to detect IE busy does not work due to the Java script.

Text capture does not work due to the way the Java script creates text.

Mouse cursor recognition does not work for a variety of reasons.

I am currently using image recognition to determine if the task is complete.

However, every time the the application is upgraded, the images change slightly. It is time consuming to have to recapture the images.

I noticed the application makes a clicking sound after it completes a task.

Does anyone know if there is a way to detect if a sound is played?

Macro Scheduler is awsome.

Thanks in advance.

Post by **Marcus Tettmar** » Thu Jun 03, 2010 4:15 pm

I do not know of any way to detect a sound being played.

Text capture should work fine in IE with Javascript produced text. JAVA may be an issue depending on the framework in use. But JavaSCRIPT is fine. Perhaps you mean JAVA. They are very different.

sailor dude · Post by **sailor dude** » Thu Jun 03, 2010 4:38 pm

I just grabbed the text using GetWindowText.

Unfortunately, it appears the developers treated much of the text as images or there is something else going on.

The app uses Java, not Javascript.

Is there anything in VBscript that could detect if a sound is played?

The app is probably playing a .wav file.

adroege · Post by **adroege** » Thu Jun 03, 2010 5:28 pm

Since it's a web app, you should look into using WebRecorder, or doing your own parsing of the HTML -- which is much more reliable than other methods. Perhaps if you share the URL of the application you are using, we can make more specific recommendations.

However, every time the the application is upgraded, the images change slightly. It is time consuming to have to recapture the images.

Maybe you are capturing too large of a target image. Try looking for a VERY SMALL graphic area which would suit your purpose. Even down to just a few pixels -- i.e. maybe they tweak the images, but *something* on the image must remain relatively constant between versions. Remember the phrase -- you are looking for a needle in a haystack.

sailor dude · Post by **sailor dude** » Thu Jun 03, 2010 5:57 pm

Thanks adroege

The database is not public so I can't share a URL.

I am grabbing the smallest image I can for speed reasons.

The images seems to change color and size. Because of the size change I do not think color tolerance will help me.

I think my application is playing the Windows Nav Start.wav file when it is close to completing the task.

adroege · Post by **adroege** » Thu Jun 03, 2010 6:08 pm

here is something interesting....
http://www.zonetrigger.com/sound-detection/

The images seems to change color and size. Because of the size change I do not think color tolerance will help me.

I think you are still capturing a *complete* image.... what I meant was you need to think in terms of just a few pixels -- I'll be willing to bet that if you show me several different images you have been capturing (that have been changing from release to release) there will be some pixels that are constant. These are the pixel changes to focus on. Then it doesn't matter that the entire graphic as as a whole undergoes minor alterations.

sailor dude · Post by **sailor dude** » Tue Jun 15, 2010 4:01 pm

I really would like to detect a sound being played to automate this application rather than detect when an image or pixel has changed. I think it would be a much cleaner solution. Currently, my scripts are not supportable enough to distribute.

I confirmed Internet Explorer is playing Windows XP Start.wav upon Start Navigation. Also it plays Windows XP Start.wav at Complete Navigation.

Alternatively to detecting a sound, is there a way to know when .wav files are being used.

adroege · Post by **adroege** » Tue Jun 15, 2010 4:24 pm

here is something interesting....
http://www.zonetrigger.com/sound-detection/

Did you look at the link I mentioned in a previous post? It details software which can trigger events based upon sound.

sailor dude · Post by **sailor dude** » Tue Jun 15, 2010 4:35 pm

Thanks adroege

I did look at the link.

Unfortunately it cannot detect what sound is being played.

Also I would have to create an event of some kind that could be recognised by Macro Scheduler.

sailor dude · Post by **sailor dude** » Tue Jun 15, 2010 5:02 pm

I tried some other ideas.

I created a corrupt.wav file and selected it for IE Complete Navigation. IE did not play a sound but did not throw an error that I could see.

I renamed sndrec32.exe thinking this is the file being used to play the wav file. Windows replaced sndrec32.exe in C:Win/Sys32 confirming IE is using sndrec32.exe

Is there something else I can do to detect if a certain wav file is being played?