Selenium Factory, an Idea

Before we get to Selenium Factory, let me preface…

I am currently an technology development intern working for Capital One. The department I was assigned to was QA or quality assurance. One of the things all Capital Interns are told, is that we have a personal “brand” which is how people view you. I wanted my brand to be entrepreneur, company builder, or future owner.

Probably a bit ambitious, but we only live once, right? Today I met that goal, someone introduced me to a new employee as, “This is Austin, he’s probably the future owner of this place. <laugh>” I may not achieve my goal of owning a multi-national corporation, but people believe (or at least joke) that it’s a possibility.

Alright. That’s all well and good (and sounds a bit like gloating), but what is Selenium Factory and what does that story have to do with it?

Short answer, everything.

Selenium Factory, Hatching an Idea

Selenium Factory is currently just an idea, with a working demo and a github. As I said, I wanted my brand to be “future owner.” How would you go about making yourself be perceived as a future owner? Gaining respect is just about the only way to perceived as a leader. My goals were straight forward enough, out perform every other of the 200 interns.

It may seem a tall order, to out perform 200 other individuals. However, when you take a step back and think, I was probably in the best position. I was in quality assurance, one of most overlooked aspects of software development. Better yet, of those 200 interns there are only ~50 technology related interns. The rest are business analysts, business management, statisticians, etc. Software is just about the most scalable thing in existence (might be some hyperbole there).

The first week of my internship it was announced that there would be a intern-only business case competition, essentially it was a competition to do a project, create a new piece of software, change a policy, etc. I was enthralled, a chance to make my brand! I had a few fun ideas I bummed around, I spoke with the other interns in the Rolling Meadows office, and my manager. Then the idea hit me. Why not try to replace my department!?

I should add at this point, that because I was using Selenium (an Open Source framework for Java), Qt (an Open Source framework for C++), and I was working on this project in my free time (at least until recently) I felt I had every right for the project to be Open Source. Up to this point I have never been told to “stop writing” about this project, better yet my managers tentatively alright with me writing about Selenium Factory (at least they saw me writing about it in this blog and didn’t reprimand me). Largely because it is not directly related to banking and accomplishes Capital One’s Technology goal of becoming “more Open Source.” That being said, I would like to give credit to Capital One for allowing me to work on an Open Source project, and although it is highly beneficial to them, allowing me to write about it as well.

Selenium Factory, Framing an Idea

An idea like “replacing a department” may seem a bit far fetched at first, but just hear me out. First, I don’t believe I can replace a full department, however I’m sure I can make a significant improvement. Hopefully I can replace the department, but I would settle for a 25% reduction in the workforce, maybe as high as 80%. Still seems a bit far fetched, but lets consider.. before search engines how did people search the web? Before virtual spreadsheets, it was done by hand. Software lets us take virtually any task and improve it in some measure.

My idea, replacing the QA department was just that an idea. There was no real material behind it, how was I going to do that? Well, the specific QA department I was in was related to testing websites. They were using QTP and were slowly transitioning to Selenium , along with no automated mobile QA testing (all was being done manually). Further, they were using Firebug (along with most of the industry) to determine web elements and XPaths. This situation seems ripe for some automation, why not merge Firebug and Selenium. After a bit of research, this has already been done, meet the Selenium IDE.

(I recommend: Selenium Testing Tools Cookbook for learning to use Selenium)
(I recommend not learning QTP, it’s not worth it)

However, there were a few glaring issues with the Selenium IDE:

1. There is no (or next to no) code optimization, meaning the code repeats everywhere
2. Generates Java/Python/Ruby/etc. functions, not XML code for Selenium Grid (of course you could add that via the API)
3. It is a FireFox extension

I therefore set out to fix that. First, I created a web browser in Qt (a C++ framework), removing the requirement of FireFox. I then built a way to track users selections (along with fellow intern Gaurav D. Kale), basically building the FireBug functionality required for the project. Alright, what about the two remaining Selenium IDE issues, it’s still a work in progress, but the idea is basically building a code optimizer.

By creating Selenium Factory from scratch, detached from any web browser, I have full control over the code (start to finish). This enables a couple of really cool features I intend to add:

1. Similar API plugin for Selenium Factory as Selenium IDE
2. Company specific framework is loaded into Selenium Factory (optional)
3. Generate – builds Java functions, send functions to (shared company wide) database
4. Database has a code optimizer, fixes any replicates and removes anything unnecessary
5. Sends back java function names, with order
6. Selenium Factory generates TestNG script ready to run a Selenium Grid on BrowserStack, SauceLabs, or similar service

If we were to group the process, half would be built into Selenium Factory (an Open Source project), the other half would be company specific. In the representation below, to the right of the blue line represents the company specific software/database:

The factory in the image above represents the core of Selenium Factory, it will take function names (with the company framework) and create tests for specific Browser, OS, Platform selections.

Selenium Factory, Justifying an Idea

These numbers are from industry averages.

Once the idea was framed properly, it became clear to me and other that the idea was both scalable and quite difficult. However, if we could pull it off would it be worth it? Although the idea was clearly a good, it was not clear if it was worth it. At this point I have several key components of the project, pretty much all the way to the blue-line in the graphic, or up to the company specific aspects. Now it is time to justify the cost of creating a code optimizer.

Before I begin I’m going to share a few assumptions:

• The average QA tester costs a company $95,000 a year
• The average C++ programmer costs a company $125,000 a year
• The average compiler programmer costs $180,000 a year
• Code optimization is difficult, and the expenses of someone who can perform code optimization will likely have a similar salary
• There are about 100 QA testers at large companies
• It takes a minimum of 30 minutes to program an automated QA test

Assuming 75 QA testers, at $95,000 a year, that is $7,125,000 a year on QA testing.

If Selenium Factory takes the average time it takes to program an automated QA test down to 2 minutes (the time for a manual QA test) from 30 minutes, it takes 15x less time (30 min / 2 min).

Further, BrowserStack and SauceLabs have ~300 possible combinations of device, platform, operating system, browser combinations. These services cost somewhere between $5000 and $125,000 a year, depending how many virtual machines. Let us assume that these costs are not going to be factored in because the company has already decided to build automated testing, so this is a cost already sunk. These costs will not be factored into our equations, but the cost of constructing ~300 different test scenarios is not sunk, because Selenium Factory can automate that too (just slide the bar to how many you want and it will auto generate). Making it a 300x increase in productivity (compounded with the 15x).

This makes each QA tester 4500x more effective. If we would like to look at “value added” it would be:

$32,062,500,000 = 4500 * $7,125,000

That’s right to achieve the same level of testing would require $32 Billion dollars. I should mention that because QA is not necessary and there is never “enough” testing, there is diminishing returns to more QA. However, at this point it becomes pretty clear that regardless the cost of development it is indeed worth development, because at the very least you could reallocate a large portion of the QA staff.

The estimated time to complete this project is 1 year with five “code optimizers” and five C++ programmers for the front end:

• $600,000 for five C++ programmers for one year
• $900,000 for five “Code Optimizers” for one year

In other words, for $1.5 million this project (per year) will have a “value added” of billions (per year). Seems like a pretty straight forward win. Even in the worst cast, you would be able to reduce the QA staff by 25% which would make up for development costs.

Selenium Factory, Selling an Idea

Building Selenium Factory is the real challenge. It is fair to say that it will be a “worth while” investment for a company. The only downside is convincing non-technical people that it indeed is worth the cost. Showing them figures is probably “good enough” in this instance, to say “go ahead and work on it.” Requesting a 1.5 million dollar budget to manage two five man teams, while I’m a 22 year old University student on the other hand…

The only way I can swing the development of Selenium Factory is through salesmanship. Will it work? To be honest I don’t know. I act as if I am 100% certain it will work. Certainly, there is money out there for a project which is exceptionally profitable and could be useful industry wide, but convincing someone… something else entirely. I have spoken with my manager, similar minded employees working with Selenium and I hear, “if it is possible, it is amazing.” The thin-blade of doubt in their voice, cutting deep.

I know this is possible, and I am presenting the idea to hopefully win the internship competition. Obviously, the most important aspects of this and how it functions at the company level are not open sourced. However, I feel this is a real benefit to the industry and will outperform all other versions of this product. My only hope is that I can sell the project and it receives the funding it needs to be the “best QA tool out there.”

Related Articles

• What I Look For In An Interview
• Building a Simple Web Browser in C++ using Qt
• Pancake Sort: Everyday Algorithms
• Tracking User Selected Web Elements from Qt webView
• Returning and storing user selected web elements using a basic C++ web browser