In our today's article, we will talk about where to go for free or for a relatively small amount of money to download historical data on the US market, as well as about a universal way to download, store, analyze and use in your own algorithms all types of market data.
First, let's concern the main sources of market data on US securities with a brief description of them. In general, there are three types of sources:
1. Sources of historical data, such as exchanges, which provide trading history on their own sites (of course, we do not consider direct connections, which are of type 2).
2. Sources of market data, such as broker terminals, through which it is possible to download also some history, but the main interest is what is happening right now.
3. Multifunctional sources that combine the type 1 and type 2, and usually presented by specialized services.
The first type of sources includes sites such as
Google and
Yahoo Finance:
The apparent advantage of these services is their full free, but on the other hand, it is impossible to download intraday market data, as well as impossible to get anything other than candles. By this we mean the data such as Level1, Order Log, Market Depth, etc.
This practically eliminates the possibility of using the data for testing strategies involving the intraday trading. On the other hand, if your strategy assumes a medium-term trade, for example, based on the Turtle approach, or you practice portfolio investments without too frequent portfolio shake-up, then use of data from these sources to be very reasonable and appropriate.
The sources of market data as already described above, are, primarily, broker terminals or other connections to the broker, which every practitioner trader has. For example: Fusion/Blackwood, Rithmic, Gain Capital, OEC Trader, Sterling, etc.
Benefits from the use of this source can be seen almost immediately. First, it is free (of course excluding those fees that you pay to the broker). Second, it is large amount of data that can be obtained: some types of candles, ticks, Level1, DOM etc. On the downside, there are the lack of deep history and the need to collect the necessary data using cunning way, when it is nothing to do without specialized software.
With this approach your capabilities for testing are greatly enhanced. There is a possibility to create not only intraday strategies, but also high-frequency algorithms based on historical regularities found.
Multifunctional sources are mostly specialized services that provide a real-time market data as well as any requested history, for example
IQFeed.
The main advantage of this source is its versatility and fullness of the data, i.e. at any time it is possible to get any desired data by the user’s request such as ticks, candles, order books etc. The downside of this approach is the payment for this service, the price of which starts from $50 per month for the basic version. If there is a desire to get a few more, it is necessary to connect the additional functions which, as you already understood, will cost money too. But like the previous option, you will need a special program to collect and store the data. Since, at the end of your subscription, you will lose all data. Plus, though the depth of history is more than at the previous method, but it is still limited. Especially for tick data.
Now we can proceed to the most interesting, but how do we get the history optimally and do not spend a lot of money. In our opinion, the common approach is as follows:
- to download the daily candles from a free source, and to test your strategy preliminary on these data;
- to download intraday data through your broker, and to test the strategy in more detail;
- the purchase of a subscription to a paid service, and downloading the data array of interest.
In order to realize suchlike a specialized software will be necessary to download the required data from the required service for you first, and then it will continue to collect the data from your broker in a common format. Thus, the unity of data will not be lost and this data can be easily used for further analysis.
For such tasks we have created the
S#.Data (Hydra) program (
to read the instruction manual and have a look at examples of use here). This is a free program available for download. Hydra provides a variety of functions, but its main task is to download and store the data.
Hydra supports downloading not only candles of any timeframe, but also ticks, order log, level 1, order books on a variety of instruments. The program can not only download but also store the data incoming from the broker, for example from
OEC Trader, Sterling, etc.
Hydra stores data in the CSV or BIN formats (super compact storage format – 7 bytes per 1 order book image or 2 bytes per tick). The data are located locally, like files, and they can be accessed from any programming language, that eventually allows the user to store and use a vast array of market data on a home computer, a server or in the cloud (AWS supported).
Summing up the article, we hope that the techniques outlined in it will allow you to get the market data for adequate means and become a bit closer to the professional participants.
They have long been doing just that.
Good luck on the market!