Adaptive Modeler™

Version 1.1.0

User’s Guide

Copyright © 2003 - 2008 Jim Witkam. All rights reserved.

Brief Contents:

1. Introduction. 11

1.1 What are agent-based models and why use them for price forecasting? 11

1.2 How does Adaptive Modeler use agent-based models for price forecasting? 11

1.3 How does Adaptive Modeler differ from conventional trading software? 12

1.4 What are the possibilities of Adaptive Modeler? 12

1.5 Specifications 13

1.6 System requirements 14

1.7 Installation. 14

1.8 Conventions 14

2. Quick start 16

2.1 Using the samples 16

2.2 Creating a new model 18

2.3 Getting help. 18

3. How does Adaptive Modeler work? 19

3.1 Agent-based Model 19

3.2 Trading System. 24

4. Quotes 26

4.1 Quote retrieval 26

4.2 Supported quote file formats 27

5. Configuring a model 30

5.1 General parameters 30

5.2 Agent-based Model parameters 32

5.3 Genomes 35

5.4 Evolution parameters 37

5.5 Trading System parameters 39

5.6 Saving and restoring model configurations 40

6. User Interface. 42

6.1 Controlling model evolution. 42

6.2 Data series tree view. 42

6.3 Charts 43

6.4 Current Values 47

6.5 Trading Signals 47

6.6 Population Window. 47

6.7 Performance Overview. 49

6.8 Market Depth. 51

6.9 Agent Window. 52

6.10 Logger 53

6.11 Customizing the User Interface. 53

6.12 Styles 54

6.13 Computation performance issues 55

7. Trading. 56

7.1 Interpreting forecasting success 56

7.2 Using the Trading Simulator 56

7.3 Statistical Simulations 57

8. More about data series 58

8.1 Parameters 58

8.2 Setting the compounding period. 58

8.3 Moving Averages 59

8.4 Recomputable vs. non-recomputable data series 59

8.5 Memory limitations 59

9. Exporting data. 61

9.1 Export Settings 61

9.2 Other Export issues 62

10. Batch processing. 64

10.1 Creating a batch process 64

10.2 Starting a batch. 66

10.3 Saving and opening batch settings 66

10.4 Starting a batch from the command line. 66

Appendices 67

I. Explanation of data series 68

I.1 Security data series 68

I.2 Agent-based Model data series 70

I.3 Trading System data series 86

II. Command line syntax 94

III. Genetic programming in Adaptive Modeler 95

III.1 General introduction to genetic programming. 95

III.2 Genetic programming in Adaptive Modeler 96

Full Contents:

1. Introduction. 11

1.1 What are agent-based models and why use them for price forecasting? 11

1.2 How does Adaptive Modeler use agent-based models for price forecasting? 11

1.3 How does Adaptive Modeler differ from conventional trading software? 12

1.4 What are the possibilities of Adaptive Modeler? 12

1.5 Specifications 13

1.6 System requirements 14

1.7 Installation. 14

1.8 Conventions 14

1.8.1 Currency. 14

1.8.2 Dates 15

2. Quick start 16

2.1 Using the samples 16

2.1.1 Starting a new model based on the sample quote file and default model parameters 16

2.1.2 Opening an already evolved sample model 17

2.2 Creating a new model 18

2.3 Getting help. 18

3. How does Adaptive Modeler work? 19

3.1 Agent-based Model 19

3.1.1 Agent Population. 20

3.1.1.1 Trading rules 20

3.1.1.2 Order generation. 20

3.1.1.3 Margin maintenance. 21

3.1.1.4 Default management 21

3.1.2 Virtual Market 22

3.1.3 Breeding. 22

3.1.3.1 Selection of parents 23

3.1.3.2 Creating offspring. 23

3.1.3.3 Replacing agents 23

3.1.4 Model Evolution. 23

3.1.4.1 Running multiple model evolutions 24

3.2 Trading System. 24

3.2.1 Trading Signal Generator 24

3.2.2 Trading Simulator 25

3.2.3 Statistical Simulations 25

4. Quotes 26

4.1 Quote retrieval 26

4.1.1 Required quote data. 26

4.1.2 Optional quote data. 26

4.1.3 Accepted quote intervals 26

4.1.4 Quote bar timing convention. 26

4.1.5 Splits and dividends 26

4.1.6 Missing or irregular quotes 27

4.1.7 Decimal digits 27

4.1.8 Quote reading process 27

4.2 Supported quote file formats 27

4.2.1 General quote data formatting requirements (applies to all formats) 27

4.2.2 MetaStock ASCII files 28

4.2.3 Yahoo CSV files 28

4.2.4 MetaTrader4 CSV files 29

4.2.5 Other supported formats 29

5. Configuring a model 30

5.1 General parameters 30

5.1.1 Quote history file of security. 30

5.1.2 Security name/description. 30

5.1.3 Model name. 30

5.1.4 Model evolution start date and time. 30

5.1.5 Market Trading Hours 30

5.1.5.1 Handling changes in Market Trading Hours 31

5.1.6 Pause model after creation. 32

5.2 Agent-based Model parameters 32

5.2.1 Population Size. 32

5.2.2 Agent Initialization. 32

5.2.2.1 Wealth distribution. 32

5.2.2.2 Position distribution. 33

5.2.3 Minimum position unit 33

5.2.4 Broker commission (for agents) 34

5.2.5 Forecast 34

5.2.6 Rounding. 34

5.2.7 Random seed. 34

5.3 Genomes 35

5.3.1 Maximum genome size. 35

5.3.2 Maximum genome depth. 35

5.3.3 Minimum initial genome depth. 35

5.3.4 Maximum initial genome depth. 35

5.3.5 Genome creation gene selection. 35

5.3.5.1 Genome Creation and Mutation Tester 36

5.3.6 Create unique genomes 37

5.4 Evolution parameters 37

5.4.1 Breeding cycle frequency. 37

5.4.2 Initial selection. 37

5.4.3 Minimum breeding age. 38

5.4.4 Parent selection. 38

5.4.5 Mutation probability. 38

5.5 Trading System parameters 39

5.5.1 Allow Short Positions 39

5.5.2 Significant Forecast Range. 39

5.5.3 Generate Cash Signal when forecast is outside range. 39

5.5.4 Apply FDA Filter 39

5.5.5 Start Capital 39

5.5.6 Enable Trading Simulator 40

5.5.7 Auto start at bar 40

5.5.8 Fixed Broker Fee. 40

5.5.9 Variable Broker Fee. 40

5.5.10 Avg bid/ask spread. 40

5.5.11 Avg slippage or price improvement 40

5.6 Saving and restoring model configurations 40

5.6.1 Default configuration. 41

6. User Interface. 42

6.1 Controlling model evolution. 42

6.2 Data series tree view. 42

6.3 Charts 43

6.3.1 Adding charts 43

6.3.2 Adding data series to an existing chart 43

6.3.3 Removing charts 43

6.3.4 Removing data series from a chart 43

6.3.5 Scrolling through charts 43

6.3.6 Maximizing a chart 44

6.3.7 Data series names 44

6.3.8 Adding a moving average. 44

6.3.9 Changing parameters 44

6.3.10 Showing the data overlay. 44

6.3.11 Linking charts 45

6.3.12 X-Axis for time series charts 45

6.3.12.1 Chart period. 45

6.3.12.2 Positioning and meaning of X-Gridlines 46

6.3.12.3 X-Axis labels 46

6.3.13 X-Axis for distribution series charts 46

6.3.13.1 Histogram bin size. 46

6.3.13.2 Bin range shown. 46

6.3.13.3 Positioning and meaning of X-gridlines and labels 47

6.3.14 Y-Axis 47

6.3.14.1 Y-Axis scaling. 47

6.3.14.2 Y-Axis labels 47

6.4 Current Values 47

6.5 Trading Signals 47

6.6 Population Window. 47

6.6.1 Scatter plots 48

6.6.2 Density charts 48

6.6.3 Using the Z (color) dimension. 48

6.6.4 Changing the axes ranges 48

6.6.5 Changing the gridline intervals 49

6.6.6 Showing the data overlay. 49

6.6.7 Showing correlation and regression. 49

6.6.8 Setting the agent dot size. 49

6.7 Performance Overview. 49

6.7.1 Calculation settings 50

6.7.1.1 Compounding Period. 50

6.7.1.2 Calculate since Trading Simulator Start / number of periods 50

6.7.1.3 Risk Free Rate. 50

6.7.1.4 VaR Confidence Level 50

6.7.2 Performance calculation. 50

6.7.3 Sub period information. 51

6.8 Market Depth. 51

6.9 Agent Window. 52

6.9.1 Showing an agent’s genome. 52

6.10 Logger 53

6.11 Customizing the User Interface. 53

6.11.1 Showing a window. 53

6.11.2 Maximizing a window. 53

6.11.3 Hiding or closing a window. 53

6.11.4 Resizing windows 54

6.11.5 Moving windows 54

6.11.6 Creating window instances 54

6.11.7 Deleting a window instance. 54

6.11.8 Renaming a window instance. 54

6.12 Styles 54

6.12.1 Default style. 55

6.13 Computation performance issues 55

7. Trading. 56

7.1 Interpreting forecasting success 56

7.2 Using the Trading Simulator 56

7.3 Statistical Simulations 57

8. More about data series 58

8.1 Parameters 58

8.2 Setting the compounding period. 58

8.3 Moving Averages 59

8.4 Recomputable vs. non-recomputable data series 59

8.5 Memory limitations 59

9. Exporting data. 61

9.1 Export Settings 61

9.1.1 Selecting data series to export 61

9.1.2 Selecting the export file. 61

9.1.3 Export historical values 61

9.1.4 Auto Export 62

9.2 Other Export issues 62

9.2.1 Adding data series to the selection. 62

9.2.2 Removing data series from the selection. 62

9.2.3 Removing a data series from a chart or the Current Values grid while it is being exported 62

9.2.4 Exporting distribution data series 62

9.2.5 Styles 63

9.2.6 At what point in the Agent-based Model cycle are values being exported? 63

9.2.7 Date and time values in the export file. 63

10. Batch processing. 64

10.1 Creating a batch process 64

10.1.1 Batch name. 64

10.1.2 Quote file(s) 64

10.1.3 Models per security. 64

10.1.4 Configuration. 65

10.1.5 Style. 65

10.1.6 Run numbers start value. 65

10.1.7 Run models until end of quote file. 65

10.1.8 Run models for a given number of bars 65

10.1.9 Export data at end of run. 65

10.1.10 Save models at end of run. 65

10.1.11 Close models at end of run. 66

10.2 Starting a batch. 66

10.3 Saving and opening batch settings 66

10.4 Starting a batch from the command line. 66

Appendices 67

I. Explanation of data series 68

I.1 Security data series 68

I.1.1 Price. 68

I.1.2 Volume. 68

I.1.3 Return. 68

I.1.3.1 Total Return. 68

I.1.3.2 Compounded Average Return. 68

I.1.3.3 Trailing Return. 69

I.1.3.4 Return Distribution. 69

I.1.4 Volatility. 69

I.1.4.1 Weighted volatility. 69

I.1.4.2 Historical volatility. 70

I.2 Agent-based Model data series 70

I.2.1 Bars processed. 70

I.2.2 Orderbook 70

I.2.2.1 Buy Orders 70

I.2.2.2 Sell Orders 71

I.2.2.3 Buy Orders remaining. 71

I.2.2.4 Sell Orders remaining. 71

I.2.3 Price. 71

I.2.3.1 VM Price. 71

I.2.3.2 VM Bid and Ask 71

I.2.3.3 VM Spread. 71

I.2.3.4 Best Agents Price. 71

I.2.4 VM Volume. 72

I.2.5 VM Trades 72

I.2.6 Return. 72

I.2.6.1 VM Return Distribution. 72

I.2.7 Forecast 72

I.2.8 Forecast Accuracy. 73

I.2.8.1 Forecasted Price Change. 73

I.2.8.2 Forecast Error 73

I.2.8.3 Mean Absolute Error 74

I.2.8.4 Mean Squared Error 74

I.2.8.5 Root Mean Squared Error 74

I.2.8.6 Right / Wrong Forecasted Price Changes 75

I.2.8.7 Forecast Directional Accuracy. 75

I.2.8.8 Forecast Directional Significance. 77

I.2.8.9 Forecast Directional Area Under Curve (FD AUC) 78

I.2.9 Filtered Volatility. 78

I.2.10 Population. 79

I.2.10.1 Population Size. 79

I.2.10.2 Average Agent Age. 79

I.2.10.3 Age Distribution. 79

I.2.10.4 Average Agent Wealth. 80

I.2.10.5 Wealth Distribution. 80

I.2.10.6 Stdev Agent Wealth. 80

I.2.10.7 Population Cash. 80

I.2.10.8 Population Position. 80

I.2.10.9 Position Distribution. 81

I.2.10.10 Breeding fitness return distribution. 81

I.2.10.11 Breeding fitness excess return distribution. 81

I.2.10.12 Replacement fitness return distribution. 81

I.2.10.13 Replacement fitness excess return distribution. 81

I.2.10.14 Trade Duration Distribution. 81

I.2.10.15 Volatility Distribution. 81

I.2.10.16 Beta Distribution. 81

I.2.10.17 Generation Distribution. 82

I.2.10.18 Offspring Distribution. 82

I.2.10.19 Parents 82

I.2.10.20 Terminations 82

I.2.10.21 Creations 82

I.2.10.22 Immigrants 82

I.2.10.23 Emigrants 82

I.2.10.24 Defaults 82

I.2.10.25 Margin Calls 82

I.2.10.26 Genome Size. 82

I.2.10.27 Genome Depth. 83

I.2.10.28 Average Nodes Crossed. 83

I.2.10.29 Average Nodes Mutated. 83

I.2.10.30 Mutations 83

I.2.11 Agent 83

I.2.11.1 Wealth. 83

I.2.11.2 Position. 84

I.2.11.3 Total return. 84

I.2.11.4 Total excess return. 84

I.2.11.5 Breeding fitness return. 84

I.2.11.6 Breeding fitness excess return. 84

I.2.11.7 Replacement fitness return. 84

I.2.11.8 Replacement fitness excess return. 85

I.2.11.9 Trade Duration. 85

I.2.11.10 Volatility. 85

I.2.11.11 Beta. 85

I.2.11.12 Offspring. 85

I.3 Trading System data series 86

I.3.1 Signal 86

I.3.2 Trading Simulator 86

I.3.2.1 Wealth. 86

I.3.2.2 Position. 86

I.3.2.3 Trades 87

I.3.2.4 Return. 87

I.3.2.5 Volatility. 88

I.3.2.6 Beta. 88

I.3.2.7 Alpha. 88

I.3.2.8 Value at Risk (VaR) 89

I.3.2.9 Relative VaR. 89

I.3.2.10 Sharpe Ratio. 89

I.3.2.11 Risk-adjusted Return. 90

I.3.2.12 Maximum Drawdown. 90

I.3.2.13 MAR Ratio. 91

I.3.3 Statistical Simulations 91

I.3.3.1 Introduction. 91

I.3.3.2 Historical Simulation. 92

I.3.3.3 Monte Carlo Simulation. 92

II. Command line syntax 94

III. Genetic programming in Adaptive Modeler 95

III.1 General introduction to genetic programming. 95

III.2 Genetic programming in Adaptive Modeler 96

III.2.1 Differences between the way Adaptive Modeler uses GP and the conventional approach 96

III.2.2 Input of the trading rules 96

III.2.3 Output of the trading rules 97

III.2.4 Defined types 97

III.2.4.1 Advice. 97

III.2.4.2 Position. 97

III.2.4.3 Limit 97

III.2.4.4 Direction. 97

III.2.4.5 Leverage. 97

III.2.4.6 Quote. 98

III.2.4.7 Volume. 98

III.2.4.8 Market 98

III.2.4.9 Change. 98

III.2.4.10 Lag. 98

III.2.4.11 Boolean. 98

III.2.5 Function and terminal set 98

III.2.5.1 CurPos 100

III.2.5.2 LevUnit 100

III.2.5.3 FullLev. 100

III.2.5.4 Rmarket, Vmarket 100

III.2.5.5 Long, Short, Cash. 100

III.2.5.6 Bar 100

III.2.5.7 InvPos 100

III.2.5.8 RndPos 100

III.2.5.9 RndLim. 100

III.2.5.10 open, high, low, close. 101

III.2.5.11 bid, ask 101

III.2.5.12 average, min, max 101

III.2.5.13 volume. 101

III.2.5.14 avgvol, minvol, maxvol 101

III.2.5.15  >. 101

III.2.5.16 change. 102

III.2.5.17  +. 102

III.2.5.18 dir 102

III.2.5.19 isupbar 102

III.2.5.20 upbars 102

III.2.5.21 bsmin, bsmax 102

III.2.5.22 volat 102

III.2.5.23 rsi>=80, rsi<=20. 102

III.2.5.24 sk>sd, sk<sd. 103

III.2.5.25 ema. 103

III.2.5.26 mfi>=80, mfi<=20. 103

III.2.5.27 pos 103

III.2.5.28 addpos 103

III.2.5.29 lim. 103

III.2.5.30 advice. 104

III.2.5.31 and, or, not 104

III.2.5.32 if 104

1. Introduction

Adaptive Modeler is a tool for creating agent-based market simulation models for price forecasting of real world market-traded securities such as stocks, ETFs or forex currencies.

1.1 What are agent-based models and why use them for price forecasting?

An agent-based model is a dynamic system of interacting autonomous entities. An agent-based model of a financial market consists of a population of agents (representing investors with their own assets and trading strategy) and a price discovery mechanism (representing a market).

Agent-based models have shown to be able to simulate complex systems such as stock markets better than traditional mathematical finance. Conventionally, financial markets have been studied using analytical mathematics based on a generalization of market participants and other simplifications and idealizations. However, the behavior of financial markets as observed in reality can not be fully described by such mathematical models. In reality, market prices are established by a large diversity of investors with different decision making methods and different investment goals (such as risk preference and time horizon). The complex dynamics of these heterogeneous investors and the resulting price formation process require a simulation model of multiple heterogeneous agents and a virtual market.

Research has shown that complex behavior as seen in actual markets can emerge from simulations of agents with relatively simple decision rules. Furthermore, commonly observed “stylized facts” of financial time series (such as fat tails in return distributions and volatility clustering) that have confronted the Efficient Market Hypothesis, have been reproduced in agent-based market models.

1.2 How does Adaptive Modeler use agent-based models for price forecasting?

Adaptive Modeler creates an agent-based market model for a given real world security. The model consists of a population of thousands of agents each with their own technical trading rule (initially created randomly) and a virtual market. Adaptive Modeler then evolves this model step-by-step while feeding it with historical prices of the security. After every received price, the agents evaluate their trading rule and place buy or sell orders on the Virtual Market where an order matching and price formation process takes place. Agent and their trading rules evolve through adaptive genetic programming. Agents with poor performance are being replaced by new agents whose trading rules have been created through crossover of trading rules of agents with good performance.

Self-organization through the evolution of agents and the resulting price dynamics drives the model to learn to recognize and anticipate recurring price patterns while adapting to changing market behavior. Model evolution never ends. When all historical prices have been processed, the model waits for new price data to become available and then evolves further. The model thus evolves in parallel with the real world market. After every processed price bar the model generates a bar-ahead price forecast for the security based on the behavior of the Virtual Market. Trading signals are generated based on the forecasts and the user’s trading preferences.

More about the inner workings of Adaptive Modeler is explained in 3. How does Adaptive Modeler work?

1.3 How does Adaptive Modeler differ from conventional trading software?

Most conventional trading software based on technical trading rules supports the user in finding or creating a (mostly static) rule-based trading strategy by optimizing and back-testing on historical data. If one searches long enough, this approach will always produce a trading strategy that seems highly profitable on historical data. This however doesn't mean that this strategy will also perform well in the future when price behavior may be different. The apparent past success of the strategy has in fact merely been caused by repeatedly optimizing and back-testing on the same historical data. This tends to lead to overfitting (or curve fitting) and is likely to produce trading rules that fail when exposed to new price data.

More advanced software may provide adaptive trading rules that automatically adapt to price developments using neural networks, genetic algorithms or other techniques. However, one adaptive trading rule will still not be able to capture the complex price behavior of a financial market caused by the interaction of various heterogeneous investors, and this approach still carries the risk of overfitting.

In financial markets no single trading rule continues to beat the market for any long period of time. Financial markets are constantly changing and new trading strategies come and go, affecting price behavior and each other’s returns. As the market evolves, trading strategies need to evolve as well in order to stay profitable.

Instead of optimizing one or a few trading rules by bac