How to get rid of FindFailed Exception in SikuliX script using Selenium Webdriver!

SikuliX works on the principal of automating anything you see on the screen. It uses image recognition to access the web elements in a web page. One of its many advantages is that it can automate flash objects in the website.

One major hurdle you get in coding with SikuliX API in Selenium is the FindFailed exception. Whenever there is even the slightest mismatch between the image you are trying to find and the image on the screen, your script will abort throwing the FindFailed exception. Even though you will find both images perfectly matching but they differ in terms of RGB pixels and it's hard to tell for the naked eye. If your test case is not focused on GUI testing and you just want to access the web element using SikuliX script, this could be a tedious job to do. One very good suggested solution I found going through Sikuli docs is:

1. Create a Pattern from the image
2. Set the minimum Pattern similarity to be used to less than the default value of 0.7
3. Create a Region on the screen where you exactly want to locate the element
4. Finally look for the item in the newly created region instead of looking for it in the whole screen

Below I share a simple piece of code that 

• Opens the Disney World website using Chrome browser.
• Clicks on 'View Gallery'
• Clicks through images using next.png below

next.png

I took this image using qSnap on Windows 10 Pro on default screen resolution of (3840 × 2160). Note that this image doesn't match the default pattern similarity level 0.7. This is perfect since we want to locate the next link using suggested solution above.

	package disney;
	import java.awt.Rectangle;
	import java.util.List;
	import org.openqa.selenium.By;
	import org.openqa.selenium.WebDriver;
	import org.openqa.selenium.WebElement;
	import org.openqa.selenium.chrome.ChromeDriver;
	import org.sikuli.basics.Debug;
	import org.sikuli.script.FindFailed;
	import org.sikuli.script.Pattern;
	import org.sikuli.script.Region;
	public class viewGallery {
	public static void main(String[] args)
	{
	// Open web browser, maximize and browse Disney World webs
	WebDriver driver;
	System.setProperty("webdriver.chrome.driver","C:\\Users\\Downloads\\chromedriver_win32\\chromedriver.exe");
	driver = new ChromeDriver();
	driver.manage().window().maximize();
	driver.get("https://disneyworld.disney.go.com/destinations/magic-kingdom/");
	// find View Gallery and click
	driver.findElement(By.id("openMediaEngineButton")).click();
	// create object of screen to use Sikuili API commands
	Screen s= new Screen();
	// create a Pattern with image next.png
	Pattern p1 = new Pattern("next.png");
	// set the debug level to 3
	Debug.setDebugLevel(3);
	// Set the minimum Similarity of Pattern to use when searching
	p1.similar((float) 0.4);
	// Create a Rectangle with your screen co-ordinates and required size. This will be used later to create the Region to search in
	Rectangle r = new Rectangle(2700,950,118,92);
	try
	{
	// Get the list of all Images in the slide show and loop through them
	List<WebElement> list = driver.findElements(By.xpath(".//*[@class= 'thumbnailSliderHolder' ]//child::li"));
	for(WebElement i:list)
	// Create Region, Highlight it for 3 seconds, find the Pattern and click on it
	Region.create(r).highlight(3).find(p1).click(p1);
	} catch (FindFailed exception) {
	exception.printStackTrace();
	}
	// close the Browser instance
	driver.close();
	}
	}

view raw viewGallery.java hosted with ❤ by GitHub

Although all of the above hustle of creating the perfect region with desired coordinates could be avoided using findElement function as below in the for the loop. I chose to find the next link using SikuliX for demonstration purposes.

driver.findElement(By.xpath(".//*[@class='nextButton fullscreenFadable']//a")).click();

view raw WebElement.java hosted with ❤ by GitHub