How To Automate a Web Browser Using Selenium & Python

Have you ever wondered if it is possible to let your computer perform actions on a web page automatically? Selenium is an excellent library we can use in Python that will enable you to do just that, plus it is much easier than you thought!

Video Tutorial - https://www.youtube.com/watch?v=AwAe1vbTmFk

To illustrate this, we will be going to the Tesla stock page on Yahoo Finance, go to the Historical Data Tab, select the last 5 years as the time period and download the .csv file.

The first thing we need to do is to install a few software. I have provided links below:

Python

Chrome Web Driver

PyCharm (Our IDE)

Google Chrome

These are pretty straight forward to install. Should you have any issues, a quick search on Google should give you a solution. 

First we need to set up Selenium in Python. We install it inside PyCharm by going to File > Settings > Project > Project Interpreter > Plus sign > Search & select Selenium > Install Package.

What we want to do is tell python how we want to interact with our webpage, which in our case will be https://finance.yahoo.com/quote/TSLA?p=TSLA.

To do this, we will need the XPath for all the elements on the webpage we want to interact with. An element is all the things that make up the website. Examples include images, buttons, search boxes, password fields and even things you can not see such as scripts. 

To obtain these xpaths, we simply press F12 or Shift+Ctrl+I inside Chrome. This will open developer tools. Then we click on the select an element tool:

Then you can simply hover directly over the element you want to interact with (click). In our case we will hover over Historical Data so that we see the word, span. Then click on it.

You should now see the the item being focused in the HTML on the right. Click on the element in the code, copy and copy XPath. In this case the XPath for the Historical Data element is:

//*[@id="quote-nav"]/ul/li[6]/a/span

We can now use this XPath in Python to create a link to that element and interact with it. We will also be doing the same with all the other XPaths. 

Now we can jump into the coding, where further explanation will be given: