Rules for Backtesting

Discussion in 'General Discussion' started by Luke, Jul 17, 2017.

  1. Luke

    Luke Well-Known Member

    I'm starting a round of backtesting and wanted to ask the community for guidance on how to set up backtesting. Do you test how you trade or trade how you test (in theory, both should be the same, but Jim has a saying about theory and practice...)? Do you follow a complete, rule-based system for adjustments, leave room for user discretion, or how do you set up your standards?

    I'm starting to test a Road Trip clone (since I don't have the full guidelines except what's in the presentations, I'm assuming I won't be trading the actual RTT) and curious if this, or other BWBs have testing strategies that you guys follow. I'll be using OV and ToS for some analysis. My thoughts are the following:

    -Look to open 75-85 DTE and target entering on a down day of at least 10 points, since it is a Vega negative trade.
    -Trade anytime after 9:30am Eastern time and plan to close trades after 2:30pm Eastern
    -Use monthlies, weeklies, and quarterlies
    -Total risk (upper profit/lower loss) 3.5% or less
    -RH on down days when the RH will not bring the BE@ exp or 1/3 of max loss closer than 60 points.

    Any other considerations either for the RTT or similar BWB or flies? Suggestions appreciated.
     
  2. Paul Demers

    Paul Demers Well-Known Member

    Hi Luke
    I am not sure you will find very many trades that you can trade on a complete rules based system as this market is continually evolving. I think you need to have a few "arrows in your quiver" for different adjustment strategies for the same trade to give you some flexibility. With a rigid set of rules you may find that there will be periods of time or events which would lead to poor trade performance, resulting in giving up on a perfectly good trade.

    When I start developing a new trade I will look at some big market moves, both the downside and upside and see what happens to the trade without adjusting. This should give you a feel for its weaknesses and help develop an adjustment strategy. Then back test those times with that strategy to make sure the adjustments do what you are expecting. If that goes well then I would back test a few years. Rather than focus on how much the trade made it is more important for me to see how much the p/l swings and how big the daily draw downs are. There is so much more that can be discussed but I think that is enough to get you going.

    Hope this helps
    Paul
     
    Luke and PK like this.
  3. PK

    PK Well-Known Member

    My personal experience. I learned and profited a lot from backtesting after having gained experience with real money. I lost a lot of time and money with backtesting before having gained experience with real money. In other words, for me (personal opinon), it makes sense backtesting your own ideas and tweaks AFTER you have succeededed in consistently making profits with strategies that are known to work out well and have been backtested ad nauseam by clever people (Kevlar, RTT, Rhino, Jeep, Fruitfly etc. etc. etc.). The not very creative reason for me to backtest is to see how I can adopt a successful strategy to my capital, trading hours, risk tolerance etc. And, to dedicate some very, but very, small amount of fun-money to discover the holy grail of trading ;-)
     
    Luke and Paul Demers like this.
  4. Marcas

    Marcas Well-Known Member

    I put my personal experience too.
    I find backtesting extremely boring. Boring, boring, boring. You have to do a little of it, but only as much as to get yourself familiar with the trade in some market conditions, just to get a feel. Backtesting as a method to validate a trade... no, no and no. It takes so much time of your life. If you have nothing better to do - go for it. Spend every minute of your free time looking at screen and watching the same trade over and over again. You also creating nice surprise for yourself because life trading is different from backtesting.

    Better build yourself nice automatic backtester that can do all the work for you and show results in nice table or graph. Only problem with this is that you will spend every minute of your free time watching screen and trying to make everything work. But it is another story...
     
    Luke likes this.
  5. status1

    status1 Well-Known Member

    The back testing is ok for maybe narrowing down to a few strategies that looks like it will work for you but there is no substitute for real live
    Especially in this slow grind up low vol market
    You may have to adapt or change your rules for different type of markets I don't think one set of rules will work in all market conditions
    You said "entering on a down day of at least 10 points" That may be hard to find in this market so you would have to wait days or weeks for an opportunity to get in and than probably have to adjust just a few days or weeks after you get in
    I understand you are cautious and I don't blame you I am just saying don't rely too much on back testing Don't expect to find a perfect strategy that works in all market conditions
     
    Luke likes this.
  6. Paul Demers

    Paul Demers Well-Known Member

    I have always promoted the idea of backtesting. I have done a lot of it. All you have to do is ask Jim Riggio as he can attest to that fact. I completely agree with Markus and status1, back testing is not the same as live trading and it is boring. It is my opinion that its merits go beyond just developing a trade it also serves other purposes. If you live test you really need to trade small in the beginning until you have seen enough different environments so you can develop a complete adjustment strategy. If you are trading trades that are a monthly trade you will only have 12 experiences in a year to analyze your strategy. Backtesting accelerates that process by allowing you to pick and choose timeframes and starting DTE’s to see how the trade reacts. The other thing it helps with is the stress. Because you can go back and pick some challenging timeframes you can see how ugly the trade can get and test to see how well your adjustment strategy protects the trade. After enough times of doing that you will become more confident that you will be able to handle an event that is similar. I am not sure that I know of any contemporary military leader that went into battle without a comprehensive set of plans for as many contingencies as they could think of. For me trading is a battle, they want my money and it’s my job not to give it to them.
    Automated backtesting is fine and I use it now but only on trades that have been fully vetted and I am just trying to tweak adjustment points or original setups. To make the automated back testing useful you need some sort of benchmark to start with and the manual testing gives you that.
     
    Last edited: Jul 18, 2017
    PK, SVL and Luke like this.
  7. Luke

    Luke Well-Known Member

    Funny reading the replies from @Marcas and @status1 - I agree, backtesting is incredibly boring and can be tough to slog through. I also agree that live trading is a great teacher, like PK says. I am certainly doing both and I routinely model and screenshot 7 day projections and adjustments and try to see how they would work out. The hard part for the few sessions of backtesting is trying to analyze how I would respond with a drawdown, barely touching an adjustment point, or fast moves, so that's part of the reason I'm curious about "rules".

    I am live practicing some rules based trades and keeping them small. I do like them for the relatively low stress and potential to get in and get out (see Hot Sauce trades in my trade journal and even the parking trade is pretty much set and forget). But for the larger trades, like the M3 and RTT clone, I'm doing some testing to see how they respond to increases in vol as well as moves to both directions. Entering on a 10 point down day isn't too difficult as there have been 11 since March and I think I saw 14 if including down days that came back and closed off of the lows. I guess what I'm seeing in low vol is that any sort of down move tends to help the entry price and set up the trade a little better when the market keeps grinding higher. Of course, I haven't traded through much, but it just seems to have helped me get into profit zones with the M3 and the prices on the RTT can get the ratio a little lower, possibly helping hit TP a few days earlier.

    @Paul Demers Excellent points and I appreciate your comments and direction. One thing I'm planning to do is go back and back test my live trades. In each circumstance, I've set up notes with screen shots of adjustments I thought of making (and in most cases have not made them) and I'm curious to see how they would have worked. Backtesting will allow for that and like you've pointed out, help me evaluate and speed up the learning curve in different environments. I realize no trade will work in all circumstances and that no set of circumstances will exactly mirror another, but time in the market is a great teacher and helps provide a level of experience that I need to get.

    My problem is that I'm not very cautious and I tend to take on more risk than I probably should starting out with new trade ideas. Trading "small" is relative, but I'm continuing to work to refine my risk management, use some hedges, and accept calculated risks. This forum and the exchange of ideas and live traders sharing what they're doing is tremendously helpful. My equity curve is pointing northwest and I've made enough to do some dinners out, add trades, and even pay for mentoring. I don't plan on stopping anytime soon, but want to keep adding to my skillset.

    I was out surfing the other day and was trying to teach a friend how to do it. We watched the waves, went over some board features, talked about positioning and timing, but when it came time to get in the water, I realized how tough it really is to learn. I have no problem seeing the water moving, seeing the currents, using the rips, and getting myself to the right place and timing the wave. It is a reaction to reading the waves, tides, and currents and I can handle powerful waves and what scares some people is what gives me a huge rush and is my therapy, but it doesn't mean I can describe each scenario and teach someone else how to read the water the same way. I realized that there are traders here that know way more about how the markets move, how to handle the steep drops, punchy waves, peeling reef breaks and shore pound and there is no real teacher other than the markets. So I'm encouraged to just keep getting time in the water, watching, observing, and putting money to work. I'm definitely a "ready, fire, aim" kind of guy, but I also realize that getting "ready" is more than just watching a webinar and funding an account!
     
  8. Marcas

    Marcas Well-Known Member

    I see that we're throwing different meanings into one word backtesting (or two: back testing).
    What I had in mind was backtesting existing strategy, in most likely scenario, provided by someone else; with entry points, adjustment rules etc. - as per opening post.
    Other approach should be taken when we use backtesting software for other purposes like developing new strategy, modeling adjustments for existing position, analyzing past trade or even monitoring existing positions.
    I only shared personal opinion, I know that many traders spend a lot of time on this 'combat simulators'' with good results. If it works for you - fine, just be aware what this procedure does and how it differs from live trading.
     
    Luke likes this.
  9. Kevin CK

    Kevin CK Active Member

    Great discussion! I only started back testing in the last month or so since trying and now purchasing the ONE software. I started out just following a strategy that was taught and works and only decided to begin backtesting afterwards in order to learn more about how to better handle difficult market environments. I would have to agree with PK, that backtesting after you've traded a system that works like is very helpful.

    I think it's also important not to over backtest because while backtesting can give you a better feel for the trade and increase your confidence, it could also give you a false confidence that is highly dependent on your ability to execute perfectly (which probably never happens even for the best traders) and get the fill you want when you want (which rarely happens). The challenge with hypothetical trading is that it fails to capture the extent of the decision making tree and domino effect of poor decision making. When you make an error on a trade (adjusting too soon, too late, not at all etc.) that you're taking heat on, the chances of compounding that problem are much higher with actual money on the line than in simulation. Backtesting is very helpful and I believe all traders should do some version of it, but having enough live trading experience to understand how to best interpret the backtested results is vital.

    My own opinion is that if the reason you're backtesting is to strengthen your trading discipline for a given strategy, it's really important to backtest what you plan to trade. I think it's more valuable to test the most general conditions as opposed to very specific ones as the general conditions will give you a better feel of how your strategy will do in "most" environments. For example, with the RTT back test I completed, I avoided conditions such as adding the trade on down days only, and entries/exits around specific times of the day. The same applies with the RH adjustment. The reason being, your conditions may not hit for a while (or you may not be able to put on the trade when they do) which could cause you to miss a cycle and with a trade like the RTT that tends to favor longer DTE entries, missing one or two trades can make a big difference in your annual performance. If the backtest does well in general conditions, then it should do even better when I try to capitalize on entries in favorable conditions (adding and RH'ing on down days etc.).

    Hope that helps!
     
    PK and Luke like this.
  10. Kevin CK

    Kevin CK Active Member

    And one other point I just remembered that was a really important lesson for me is that backtesting a strategy that requires a lot of fine adjustments (like the M3) can be a bit misleading as my own limited experience trading the M3 is that I realized all the adding and removing of spreads/rolling spreads complicated the trade and made it difficult to adjust when markets were moving fast. I traded it live only after backtesting it and at the time I was backtesting it, I didn't realize that the ease with which I could add or remove spreads in ONE was not replicable on my trading platform. The ease of adjustments is the main reason I don't trade the M3 and that's something that I did not realize until I traded it live.
     
    Luke likes this.
  11. SVL

    SVL Well-Known Member

    Hi Luke,
    I still do a lot of debriefing of my past live trades.
    For example you take July expiration cycle and back test the same cycle by starting the trade 65 DTE, 60 DTE, 55 DTE, 50 DTE, 45 DTE.
    Quite often I copy my T-log to the new account and trade the same campaign by using different adjustments to fix the delta.
    So you might back trade the same cycle 5-10 times . After doing the same exercise again and again, you will build your mechanical skills and gain a much better understanding of nuances of each specific adjustment techniques .
    You will realize that everything has plusses and minuses. For example, rolling shorts up will stop being effective the closer you get to the expiration.
    All this knowledge you will get from back-testing much faster than in real live trading.
    The execution is a completely different subject which also needs to be practiced . Anything could be executed in 99 % of market environments and it just a matter of slippage.
    The rule of thumb, in the fast rising market, the price of butterflies below the money is dropping with each point of rise in the index price and vice versa.
    If you want to sell butterflies, you have to give the market-makers a discount of 0.10-0.20 cents below the mid, because they know those flies might get much cheaper 2 hours later.
    John Locke had 2 enriching sessions about the proper back testing in 2016.
    Both sessions are available for purchase on his website:
    https://www.lockeinyoursuccess.com/...g-results-through-proper-back-testing-part-1/
     
    Paul Demers and Luke like this.
  12. Luke

    Luke Well-Known Member

    Great advice here. Thanks for sharing because this is more practical for me since I am not yet developing any trades. I think what you outlined here is better than what I put down for rules and this is a clear way to compare results from one live cycle and speed up my learning curve.

     
  13. SVL

    SVL Well-Known Member

    Luke,
    I am glad you found it useful.
    I would also recommend to find in the CD digital library and listen to the recording of Trading Group 2 Session from May 26, 2015.
    The main topic of that session was the importance of proper back-testing and debriefing of your trades.
    In my personal opinion, the recording of TG2 sessions for the period between Nov 2014 and May 2015 where Tom Hughes and Curtis Allen shared and discussed in detail almost every week their live M3 trades is the crown jewel of the whole Capital Discussion’s digital library.
     
    Luke likes this.
  14. Luke

    Luke Well-Known Member

    Awesome, thanks for the tip. Downloading them now and working through them.

     

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice