An automation algorithm for harvesting capital market information from the web

Purpose – The purpose of this paper is to develop an algorithm to harvest user specified information on finance portals and compile it into machine‐readable datasets for quantitative analysis. Design/methodology/approach – The Visual Basic macro language in Microsoft Excel is applied to develop code that is not constrained by the single‐query function of Excel. The core of the algorithm is built around the splitting of the URL connector line and the placement of a continuously updating variable into which are looped as many tickers as there are in the input list. The output is then written to non‐overlapping cells. Findings – Numerical information placed on major finance websites can be harvested into structured machine‐readable datasets by applying this algorithm. Research limitations/implications – One significant change in Microsoft Excel 2007 is that the worksheet is expanded from 224 to 234 cells, or to be more specific, from 256 (IV) columns × 65,536 rows (28 × 216) to 16,384 (XFD) × 1,048,576 (214 × 220). These new limits while allowing for a larger number of tickers, still constrain a single worksheet to 16,384 columns. For five fields per ticker that translates into roughly 3,200 ticker symbols. Practical implications – The algorithm extends user accessibility to websites that do not provide the facility of simultaneous downloading of information on multiple stock tickers. Furthermore, the procedure automates the downloading of multiple pieces of information (fields) and entire tables per ticker (record). Originality/value – An exhaustive literature search did not find any paper that discusses a multiple ticker algorithm for web harvesting.