Interactive chart visualizations using Python and bqplot: visualizing S&P500 returns

A couple of months ago, I stumbled upon this interesting presentation Jupyter Notebooks: interactive visualization approaches. The presentation showed how you can use bqplot to build interactive visualizations. 

Bqplot contains a set of 2D plotting widgets built on top of the ipywidgets framework for Jupyter notebooks. The bqplot package aims to bring d3.js visualizations to Python while retaining the flexibility and ease of use of ipywidgets and was developed by the quantitative research team at Bloomberg. You can install bqplot using conda or pip. 

One of the examples built by the team that you can find on Github is a Jupyter notebook which shows US equity market performance (using the S&P 500 index) where you can select an interval on a time series chart - for the selected area you get the total return as well as a histogram of the daily returns.

References:

• Interactive data visualization using Bqplot
• bqplot - Interactive plotting in Jupyter notebook
• bqplot: bringing more interactivity to data

Quick tip: troubleshooting Jupyter notebook not starting correctly

The other day my Jupyter notebooks did not start correctly from Anaconda navigator - luckily the Jupyter docs have a section  Jupyter - What to do when things go wrong. So I tried starting it from Anaconda prompt and it indeed gave me an exception that there was an invalid path in the Jupyter config file - to find out where to look for the config file check out Jupyter common directories and file locations

Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance

Yahoo Finance is one of the most popular sources of free financial data. It does not only contain historical data but also financial statements, dividend information and calculated metrics like e.g. 50 and 200 day moving average, beta, etc ... Yahoo Finance does not have an officially supported API anymore but pandas-datareader  still allows you to access the data from Yahoo Finance in Python (other alternatives are yfinance and yahoo_fin).

This post is part of a series on using Pandas datareader to retrieve financial data:

• Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)
• Using Python and Pandas Datareader to retrieve financial data - part 2: Fama & French library
• Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance

In this post I have used version 0.10.0 of pandas-datareader  (released July 13, 2021) which is currently working with Yahoo Finance - previous versions of pandas-datareader had to be updated after Yahoo made some changes on the underlying API.

Warning: Accessing Yahoo Finance using Python libraries is quite brittle so don't try to built production trading systems using this data source.

Accessing the Yahoo Finance API using pandas-datareader is very simple as shown in the screenshot below but I would also recommend implementing a cache mechanism for your queries using the requests-cache Python library to avoid having your IP address being banned. The full source of this Jupyter notebook is available at https://github.com/jorisp/tradingnotebooks/blob/master/YahooFinancesingle.ipynb

References:

• Yahoo Finance API - A complete guide
• yfinance 
• yahoo_fin

Using Python and Pandas Datareader to retrieve financial data - part 2: Fama & French data library

This post is part of a series on using Pandas datareader to retrieve financial data:

• Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)
• Using Python and Pandas Datareader to retrieve financial data - part 2: Fama & French library
• Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance

In this post we will look at the datasets made available by Eugene Fama and Kenneth French. Eugene Fama and Kenneth French did a lot of research on which factors drive security returns. In 1993, they published the Three Factor Model (see article "Common risk factors in returns of stocks and bonds", Journal of Financial Economics 33, 1993), which showed that their factors (size of the firm, book-to-market values and excess return) capture a statistically significant fraction of the variation of stock returns. In 2014, Fama and French adapted their model to include five factors.  Fama won the Nobel Prize for Economics in 2013 for his research. Fama also published a number of papers on the Efficient Market Hypothesis and random walk theory.

Fama and French still publish the returns of various investment factors analyzed by them on their homepage on a regular basis.  You can download this data using the pandas_datareader library - you can take a look at the official documentation,  Fama-French Data (Ken French's Data library) to get started or take a look at the Jupyter notebook that I shared on Github https://github.com/jorisp/tradingnotebooks/blob/master/FAMA.ipynb

References:

• Fama/French forum ideas and observations from Fama and French
• How to use the Fama French model
• Fama-French 5-factor model: five major concerns
• In pursuit of the perfect portfolio: Eugene F. Fama (YouTube - MIT Laboratory for Financial Engineering )

Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)

The pandas-datareader Python library covers a number of APIs with global fundamental macro- and industry data sources including the following (for a full list see  Pandas Datareader - data sources ):

• St. Louis FED (FRED): Federal Reserve data on the U.S. economy and financial markets
• Fama/French data library : market data on portfolios capturing returns on key risk factors like size, value, and momentum by industry
• Yahoo Finance : retrieve daily stock prices, historical corporate actions (dividends and stock splits) from Yahoo Finance 
• World Bank: global database with economic/social indicators and demographics.

This post is part of a series on using Pandas datareader to retrieve financial data:

• Using Python and Pandas Datareader to retrieve financial data - part 1: Federal Reserve Data (FRED)
• Using Python and Pandas Datareader to retrieve financial data - part2: Fama & French library
• Using Python and Pandas Datareader to retrieve financial data - part 3: Yahoo Finance

In this post I will focus on retrieving data from FRED using pandas-datareader.  Federal Reserve Economic Data (FRED) - https://fred.stlouisfed.org/  is a database maintained by the Federal Reserve Bank of St. Louis. It has more than 800.000 data time series covering categories such as Economic growth & employment, monetary & fiscal policy, demographics, industries, commodity prices at different frequencies (daily, monthly, annual).  One of the interesting time series you can find here are 3-month Treasury Bill Secondary Market rate (TB3MS) or 1-year US Treasury bills which are used a proxy for the risk free rate in financial modeling. 

There is however some missing data on the TB1YR - so I will be using the TB3MS (3 Month) in my next example. You will notice that all time series are identified by a short abbreviation that you can find by searching on the FRED website.

Full source code is available at https://github.com/jorisp/tradingnotebooks/blob/master/FRED.ipynb 

References:

• FRED API: get US economic data which drives the real estate market
• Pandas-datareader python library

Reading and writing files in Azure Blob Storage with Python

Azure Blob storage is Microsoft's object storage solution for the cloud and allows you to store massive amounts of unstructured data, such as text or binary data at low cost for every scale. If you are not familiar with it, I can recommend taking a look at the Store data in Azure learning path on Microsoft Learn

Using Python in combination with Azure Blob Storage is quite easy using the azure-storage-blob client library for Python . You can set up a container with private access meaning that you will need to provide credentials to access the containers and the blobs contained within. The easiest way to do this is using a shared access signature (SAS) token. You can generate a SAS token from the Azure Portal.

To interact with the different parts of Azure  Blob Storage you will typically use the BlobServiceClient to work with the Azure storage account itself, the ContainerClient to work with a specific container and the BlobClient to work with a specific blob.  Below is the sample code which uses these different clients in a Jupyter notebook (based on Quickstart: Manage blobs with Python v12 SDK)   - you can find the full Jupyter notebook at tradingnotebooks/AzureBlobStorage.ipynb at master · jorisp/tradingnotebooks (github.com) 

 References:

• Quickstart: Manage blobs with Python v12 SDK
• Using your data lake as a cheap time series database: do's and don'ts
• How to download blobs from Azure Storage using Python - sample code for multithreading

Alternatives to Azure Notebooks preview

Azure Notebooks Preview will be retired on January 15th, 2021 and all user data will be destroyed - don't forget to download your user data before then.  Microsoft alternatives to use are listed on Quickstart: Export a Jupyter Notebook project in Azure Notebooks Preview 

How to use secrets in Jupyter Notebooks with python-dotenv

It is not a good idea to embed the the secrets (credentials, keys, etc..) that you need in your Python code directly in your Jupyter Notebooks. You can store these secrets in environment variables but an alternative is using the python-dotenv library. Python-dotenv is a library that reads key-value pairs from  a .env file and adds them to environment variables.

To get started you need to install the latest version of python-dotevn and then you will need to create a .env file in the same folder as your .ipynb files.

Tip: You may not be able to create .env file directly in Windows Explorer - use the command prompt and the echo command to create this file.

You can read a key-value pair in just one line of code after you loaded the extension and the os module which provides functions to interact with the operating system.

Source code is available on https://github.com/jorisp/tradingnotebooks/blob/master/dotenv.ipynb

References:

• How to not embed credentials in Jupyter notebooks

Quick tip: reload Python modules in Jupyter with the autoreload magic method

If you are writing your own Python modules and you include them in your Jupyter notebooks you might notice that by default your code updates in the custom Python module are not reflected once the module has been load by Jupyter. You can fix this by adding %autoreload at the top of your notebook.

%load_ext autoreload
%autoreload 2

For the documentation take a look https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html

Quick tip: change Jupyter notebook startup folder in Anaconda

This excellent walk through explains how to change the Jupyter notebook startup folder in Anaconda - quick summary of the steps:

1. Generate config file by running jupyter notebook --generate-config in the Anaconda prompt
2. Open the c:\Users\\.jupyter\jupyter_notebook_config.py
3. Uncomment c.NotebookApp.notebook_dir = '' and fill in your startup folder e.g. 'c:\jupyter_workspace'  

For an overview of all of the configuration setting in the jupyter_notebook_config.py file take a look at Config file and command line options (Jupyter documentation)