Good afternoon. We continue to talk about StockSharp\u0027s products, and in past articles we have analyzed the S#.Designer trading robot constructor, as well as the S#.Shell graphical framework, which helps you create trading robots, including in C# using the S#.API library. The work of these programs is aimed at creating trading systems using programming languages or cubes, in which the lines of code are replaced with cubes for simplicity, performing trading operations or responsible for conditions and parameters. Creating trading robots would not be complete without testing the newly created trading strategy. To test and debug trading robots, before launching on real trading, you need to conduct testing on market data to make sure that they work correctly, consider possible \"thin spots\" or errors that occur during trading. Today, market data plays an important role not only for technical analysis of trading systems, but also for analysis of market behavior in General. Agree that having data on market behavior for the period of interest, under the appropriate conditions, you can accurately predict the behavior of the market, thus you can predict the development of events. Today, market data is the most valuable information for a trader, and its search is necessary for him, both for future trading and for analyzing the actions performed. The problem with getting market data is the small number of sources that provide information on favorable terms. There are always restrictions for teaching, either the data is paid, or it is limited to the period for which it is provided. Moreover, the trader receives data from only one source, and changing sources to get other data is quite an inconvenient procedure. Based on the needs of the trader in S#, we developed the S#.Data or just Hydra. What is she setting up? The S#.Data is a program for automatically loading market data. source-market-data.gif The user is provided with a huge Arsenal of functions: - Select the source of market data (more than a hundred sources to download). - Selection of the necessary financial instrument that is of interest to the trader. - Select data type: candles, tick deals and glasses - An impressive selection of timeframes for downloading market data. source-market-data-download.gif A huge advantage of Hydra is the ability to choose the data storage format. Thus, after downloading market data, the user saves it in a convenient format and can use it when testing their strategy on any platform. Data can be saved in two formats: in the special S#.Data (BIN) - binary format, which provides the maximum compression rate, or in CSV text format, which is convenient when analyzing data in other programs. market-data-download.gif All the user needs is to configure the source of market data, select the tool and the required period. In this case, you can configure the program to continuously save market data. The ability to use server mode allows you to save market data on the server and configure access from anywhere in the world, which allows you to create huge amounts of data for more detailed analysis or use in long-term auctions. At the same time, S#.Data can use both historical and real-time data sources (for example, connecting SmartCOM to get glasses). This is possible by using an extensible (plug-in) source model. Also, Hydra is a plugin that allows you to develop your own sources. The ability of the Analytics function allows you to analyze data according to various criteria without leaving the program. Plotting the data of stock quotes and indexes, make the downloaded data more visible and more convenient to use. A very important advantage that makes this program unique, in addition to a set of features that any other program can envy, is that the Hydra program is completely free. All these features are not the limit of functionality, only bright distinctive features that make Hydra an indispensable tool for any trader who is especially set up for profit.
In our today\u0027s article, we will talk about where to go for free or for a relatively small amount of money to download historical data on the US market, as well as about a universal way to download, store, analyze and use in your own algorithms all types of market data. First, let\u0027s concern the main sources of market data on US securities with a brief description of them. In general, there are three types of sources: 1. Sources of historical data, such as exchanges, which provide trading history on their own sites (of course, we do not consider direct connections, which are of type 2). 2. Sources of market data, such as broker terminals, through which it is possible to download also some history, but the main interest is what is happening right now. 3. Multifunctional sources that combine the type 1 and type 2, and usually presented by specialized services. The first type of sources includes sites such as Google and Yahoo Finance: image002.gif The apparent advantage of these services is their full free, but on the other hand, it is impossible to download intraday market data, as well as impossible to get anything other than candles. By this we mean the data such as Level1, Order Log, Market Depth, etc. This practically eliminates the possibility of using the data for testing strategies involving the intraday trading. On the other hand, if your strategy assumes a medium-term trade, for example, based on the Turtle approach, or you practice portfolio investments without too frequent portfolio shake-up, then use of data from these sources to be very reasonable and appropriate. The sources of market data as already described above, are, primarily, broker terminals or other connections to the broker, which every practitioner trader has. For example: Fusion/Blackwood, Rithmic, Gain Capital, OEC Trader, Sterling, etc. image003.png Benefits from the use of this source can be seen almost immediately. First, it is free (of course excluding those fees that you pay to the broker). Second, it is large amount of data that can be obtained: some types of candles, ticks, Level1, DOM etc. On the downside, there are the lack of deep history and the need to collect the necessary data using cunning way, when it is nothing to do without specialized software. With this approach your capabilities for testing are greatly enhanced. There is a possibility to create not only intraday strategies, but also high-frequency algorithms based on historical regularities found. Multifunctional sources are mostly specialized services that provide a real-time market data as well as any requested history, for example IQFeed. image006.gif The main advantage of this source is its versatility and fullness of the data, i.e. at any time it is possible to get any desired data by the user’s request such as ticks, candles, order books etc. The downside of this approach is the payment for this service, the price of which starts from $50 per month for the basic version. If there is a desire to get a few more, it is necessary to connect the additional functions which, as you already understood, will cost money too. But like the previous option, you will need a special program to collect and store the data. Since, at the end of your subscription, you will lose all data. Plus, though the depth of history is more than at the previous method, but it is still limited. Especially for tick data. Now we can proceed to the most interesting, but how do we get the history optimally and do not spend a lot of money. In our opinion, the common approach is as follows: - to download the daily candles from a free source, and to test your strategy preliminary on these data; - to download intraday data through your broker, and to test the strategy in more detail; - the purchase of a subscription to a paid service, and downloading the data array of interest. In order to realize suchlike a specialized software will be necessary to download the required data from the required service for you first, and then it will continue to collect the data from your broker in a common format. Thus, the unity of data will not be lost and this data can be easily used for further analysis. For such tasks we have created the S#.Data (Hydra) program (to read the instruction manual and have a look at examples of use here). This is a free program available for download. Hydra provides a variety of functions, but its main task is to download and store the data. hydra_new.png Hydra supports downloading not only candles of any timeframe, but also ticks, order log, level 1, order books on a variety of instruments. The program can not only download but also store the data incoming from the broker, for example from OEC Trader, Sterling, etc. Hydra stores data in the CSV or BIN formats (super compact storage format – 7 bytes per 1 order book image or 2 bytes per tick). The data are located locally, like files, and they can be accessed from any programming language, that eventually allows the user to store and use a vast array of market data on a home computer, a server or in the cloud (AWS supported). Summing up the article, we hope that the techniques outlined in it will allow you to get the market data for adequate means and become a bit closer to the professional participants. They have long been doing just that. Good luck on the market!