Fooled by Randomness

Discussion in 'General Discussion' started by Kevin Lee, Jan 6, 2017.

  1. Kevin Lee

    Kevin Lee Well-Known Member

    Many years ago, I read a book called Fooled by Randomness by Nassim Taleb, the author who also wrote the book The Black Swan. The main theme of the book is that our brains evolved to be great at pattern recognition, a skill critical to survival in the ancient savanna. This skill lives on to modern times but it fools us to see patterns where there is none. We see patterns in random objects, such as in the clouds as well as in stock charts. It's difficult to "un-see" those patterns even if we rationally understand they are purely random. That's why it's so easy for us to be fooled by randomness.

    Motivated by this book, I created a spreadsheet to generate a time series to mimic daily stock prices, purely by random number. I was floored when I saw the chart patterns were essentially indistinguishable from a real stock chart. Trendlines and support/resistance are clearly visible and one can perform technical analysis on the time series to find some that works. But these are randomly generated values !

    This is an example -


    Yesterday, I had a discussion with a friend on that subject. He asked me to re-create the excel, which I already lost. After finding a copy online, I modified it to fit more closely to a stock index time series. I'm sharing it here for those interested. Play with it to see the kind of chart patterns you can generate and then take a look at the formula to confirm that it is indeed generated by random numbers.


    Attached Files:

    Last edited: Jan 6, 2017
    ktingle, max.shi, ooo|ooo and 7 others like this.
  2. Trader G

    Trader G Well-Known Member

    I always send people this link when they tell me they trade based on patterns. It picks a random stock and you have to pick direction.

    Edit: I will say that a chart of my skill at picking direction is not random. It slopes sharply downward. :)
  3. AKJ

    AKJ Well-Known Member

    There would seem to be an easy solution to fixing this problem; just fade your instinct and the slope will turn sharply upwards.
  4. ACS

    ACS Well-Known Member

    Adam Grimes has covered this issue quite well. The webinar he did for IB in February 2013, for example, shows how moving averages and Fibonacci have no statistical edge. If you really want to continue your Taleb inspired education then read Antifragile.
  5. Harry

    Harry Well-Known Member

    Despite what you say, people make living (and lose living but they are not the best) reading charts. Of course not every one of your chart readings will be correct, but if you can get the same edge as in options trading: more right than wrong OR bigger rights than wrongs, it does not matter what the public says.

    The challenge is that our markets change, and we say the same in terms of options - low vol, high vol, skew this, skew that - and the chartist has to understand the phase of market they are in - pure line graph is sometimes enough, but not always.

    I am not saying he is a prophet, but track bulkowski. I go to his site when in doubt about some pattern.

    The benefit with the options is that you shoot first adjust later, whereas in charts it is mostly take a loss and move on. As long as you have solid adjustment / hedging techniques and do not screw up money management, you can survive options even if you enter at the wrong time. Not so with charts - try picking bottom of a falling stock, it may work most of the time but when you are wrong with 2-3 gaps ... amazing vote of confidence to your mindset ... just like price rolling through your long strikes.
  6. Kevin Lee

    Kevin Lee Well-Known Member

    This is a never ending debate and I have no intention to stir up one here. Intention is just to show something I thought was interesting.

    Yes - people do make a living reading charts, but so do fortune tellers. ;)

    Agree - Bulkowski is great. I like his books because they are fact based. Bulkowski does provide long-term back test results. His works are the most scientific among all the TA I have seen. What I did find in his books is that many of the popular patterns are not much better than random chance but some less know patterns did provide some edge.
    TOPS likes this.
  7. SVL

    SVL Well-Known Member

    TA works for sure as all algorithmic trading is based on some form of TA.
    I remember from the recent interview of Emanuel Derman that he said that over 50 % of all trading volume is done by computers nowadays.
    He also said in the same interview that a retail trader has almost zero chance to beat these computers.
    That’s why there are so many people try to sell mentoring programs for TA or some kind of magic indicators instead of just making money in the market like you do.
    This is my favorite curve-fitted chart for long term investing. The only good thing about it is that it was perfect predicting what happened in the past. But in the future, who knows?

    Sergei Bull-Bear Market Indicator.png
  8. Kevin Lee

    Kevin Lee Well-Known Member


    I actually agree with what you said but with a twist. Algorithmic trading is some kind of TA but it's not the TA we are talking about. Algo trading happens to be something that I'm very interested in and I'm spending quite a lot of my time researching and doing it.

    Fact is the bulk of the algo trading volume is based on HFT, which is competing on speed and visibility of order flow. That's definitely not traditional TA. LFT, on the other hand, is based on analysis, but it's not the type of TA we are talking about here. If an algo is based on the type of simple visual analysis and correlation that human is able to process, it will be wiped out in micro seconds by competition.

    OK - let me stop here before I invite barrage of other responses. If TA is your thing, please continue to do what you feel is right. I have no intention to prove otherwise. :)
  9. ACS

    ACS Well-Known Member

    I'm no expert but my understanding is that what is commonly called HFT trading is divided into predator or prey. The institutions (prey) that need to move stock try to evade the predators and the predators try to front run them. Real computerized market making with bids and offers you can actually trade with is something different as is black box type trading such as trend following or mean reversion systems.
  10. DGH

    DGH Administrator

    I wish I had some of the money back that I spent on indicators and systems for Ninjatrader. I am not a believer in indicators, but I am convinced that the algos monitor double top/bottoms and triple top/bottoms as well as breakouts from Donchian channels or similar setups AND highs/lows from previous session(s) as well as intra-day highs and lows. Also, their programs tend to "kick in" at certain times of the day (TOD) with relatively predictable regularity. They will beat you on any tick-based, tick volume-based, or moving average type of indicator, e.g. SMA, EMA, CCI, money flow, etc. They "see" your orders, and lie in wait for you. They know your STOPs (and will "run" them) as well as your most likely entry points. If you doubt this, just watch a 1 minute or 250 tick chart for a while and note how quickly the market will move at certain points and then slow down or reverse. Been there, done that. I try to catch these quick moves with a Donchian channel breakout. I usually make my "latte money" for fun using Donchian channel breakouts plus monitoring the intraday A/D line and $TICK line, but I don't think I could make a living day trading. Heck, I'm not a very good swing trader, either. Good thing we have options.
    PK, Kevin Lee and KiwiDon like this.
  11. KiwiDon

    KiwiDon Well-Known Member

    I agree with Kevin and Dan here.

    Interestingly, on the topic that Kevin is illustrating with randomly generated series, Edward Thorpe's views as discussed in "Hedge Fund Market Wizards" concur:

    (his original fund, Princeton Newport Partners, achieved an annualized gross return of 19.1 percent (15.1 percent after fees) over a 19-year period.
    Even more impressive was the extraordinary consistency of return: 227 out of 230 winning months and a worst monthly loss under 1 percent.)

    His quote on traditional technical analysis:

    We got off on a tangent. I had asked you about how you made the transition from casino games to markets.

    After my successful casino games—I also developed a system for beating Wheel of Fortune—I got to thinking about games in general and thought, The biggest game in the world is Wall Street. Why don’t I look at and learn about that?

    I knew almost nothing about the market. In 1964, I decided to spend the summer learning about the stock market. I read everything from Barron’s to books such as the Random Character of Stock Prices. After a summer of reading, I had a lot of thoughts about what to do and what to analyze.

    Were any books particularly helpful?

    Most of them were helpful in the negative. For example, Technical Analysis by Edwards and McGee was very helpful in the negative.

    What do you mean by “helpful in the negative?”

    I didn’t believe it. The book convinced me that technical analysis was a road not to go down. In that sense, it saved me a lot of time.But one could come up with a rational explanation of why chart analysis might work—namely that the charts reflect the net impact of all the fundamentals and the psychology of all the market participants.

    You can’t prove a negative. I can’t prove it doesn’t work. All I can say is that I did not see enough substance there to pursue it. I didn’t want to take time to try things unless I thought they were pretty good.
    Interestingly his views on trend following are more favorable

    (refer ):

    Do you have thoughts on futures trend-following as a strategy?

    I believe there are versions of it that have a Sharpe ratio of about 1.0 or more, but that is risky enough so that it is hard to stay in a business because you can get shaken out.

    I take it then that you believe there are trends inherent in the markets.

    Yes. Ten years ago, I wouldn’t have believed it. But a few years ago, I spent a fair amount of time looking at the strategy. My conclusion was that it works, but that it was risky enough so that it was hard to stay with it.

    Did you ever use trend-following as a strategy?

    I did.

    When did you trade futures?

    We began the research project in 2006, and launched the trading program in late 2007. It was promising, and we were thinking of bringing it up in large-scale with institutional money. But in early 2010 my wife was diagnosed with brain cancer, and my heart wasn’t in it. Life is too short. I didn’t want to launch another major activity, so we gradually wound the program down.

    So the program had worked well while you’re using it.

    Reasonably well. It wasn’t as compelling as the Princeton Newport strategies or statistical arbitrage, but it would have been a good product, and it seemed to be better than most of the other trend-following programs out there that were managing a lot of money.

    What kind of Sharpe ratio was your program running?

    It was a little better than 1.0 annualized.


    ...and at the risk of making this post WAY too long....this recent list by Jeff Miller illustrates the complexity that exists out in the non-charting world (points 5 and 6 are pertinent to this thread's topic):

    As part of my preparation for 2017, I asked how I could be most helpful for individual investors. The suggested resolutions are a combination of expert investment methods and avoiding the most common investor mistakes. They may be difficult to follow. If you can, you will find them profitable.
    1. Make a fresh start. We should do this constantly, but the calendar is a good reminder. One of my former bosses took a vacation at the end of each year. His instructions included a list of stocks that he “did not want to see on his sheets” when he returned. He knew the right thing to do, but it was still difficult for this top professional to do it himself.
      1. Review your losing positions. Is the thesis intact?
      2. Review you winners. Is it time to trim? Have you reconsidered your price target? Are there new ideas that are better?
    2. Look past the headlines. Read the actual story. The writer may have one message, while the headline represents another. If you care enough to read financial news, you can spend a few more seconds on each article or post.
    3. Save time by dumping sources. Conduct a review. If a source has not provided anything helpful in the last two or three years, ignore it! You need the time for more important matters. If the information has led you astray, that is even more reason to move on.
    4. Do not blame others for your own poor decisions. If you have not enjoyed the market rally, it is not the fault of the Fed, the Congress, the President, the Plunge Protection Team, high frequency traders, or anyone else except you. Unlike casinos, the odds for investments are in your favor. Accept the reality that government officials, all over the world, are attempting to block, fix, or postpone problems. You may not like the solutions, but why not profit anyway?
    5. Do not be mesmerized by charts including commentary and big, colored lines. Ask yourself whether the underlying argument makes sense. Should you sell your long-held position because a guy on TV says there is “technical damage?” Be sure you know what that means and compare it to your own analysis.
    6. Beware of misleading charts.
      1. Ignore charts that “prove” that current markets are just like some prior time period. With modern software, it is easy to cherry-pick some prior period, adjust the scales, and scare people witless (TM OldProf euphemism). Do not be bamboozled by this cheap trick.
      2. Ignore charts that are too good to be true. They are. Usually the researcher has used too many variables on too little data. If you do not understand what that means, it is even more reason to be skeptical. You will have a high susceptibility to confirmation bias.
    7. Do not blindly accept “headwinds” or “tailwinds” stories. These are popular, easy to write, and require little research. Just ask yourself a few questions.
      1. Is this really likely? Can I even estimate the probability?
      2. How big are the consequences?
      3. Can we expect a policy reaction?
    8. Learn to use “business cycle peak.” Ignore all talk about “recessions.” The term has been corrupted in a way that is beyond repair. My educational effort is a lost cause, since most continue to treat below-trend economic growth as a recession. This makes them live in a state of constant fear. Bad consequences for stocks occur right after a business cycle peak. So far, so good, on that front.
    9. Turn the page when you see “ageing bull” or the like. While it is true that this economic and stock cycle is longer than the norm, it has been gradual and started from a very low point. Suppose we had flipped six heads in a row. Does that change the odds of the next coin toss?
    10. Do not use your conclusions to go “all in” or “all out.”Everything you see urges you to believe that you are a great market timer. Get real! No one does that consistently. The best we can do is to control risk.
    So most importantly —
    Evaluate the risk of your portfolio and make sure that it is appropriate to your plans and circumstances.

    Last edited: Jan 7, 2017
  12. Boomer34

    Boomer34 Well-Known Member

    Thanks for that post. Big insights in there!
  13. Joe A

    Joe A Active Member

    I wrote a Python program a while back to convince myself of the truth of what Kevin is saying. It is an odd sensation to see yourself creating lines of 'resistance' and 'support' with your own code!
  14. max.shi

    max.shi Active Member

    I am probabilistic advocator, but I believe Charting techniques still have their own purpose and usefulness. The main reason of this is that individual traders, who see similar 'patterns', make charts. Another reason is that diffusing information requires time, which could help to build up momentum (reflected in the drift rate simulation). The support line or MA (basically they are low-pass filter) actually pick out your drift-rate, one useful parameter for momentum trading.
  15. KiwiDon

    KiwiDon Well-Known Member

    Another interesting way of interacting with probability and it's short term and long term effects is to study the statistical similarities of the stock market with the casino game of blackjack (where the casino edge can be reduced to 0.5% playing a basic strategy....think of it as the drain on returns caused by the "annual expense fee" in stockmarket terms)....a quote from this article is fascinating:

    "Blackjack can be a surrogate for more sophisticated financial instruments. The first graph in Figure 5 shows the performance of the Standard & Poor's stock market index during 2011. The second shows the performance of our blackjack simulation playing 100 hands a day for each of the 252 days the stock market was open that year. The S&P dropped 14.27 points. Our blackjack simulation, which bet $10 per hand, lost $3860 over the same period. More important than these final results is the fact that both instruments show large fluctuations about their mean behavior. Over the short term, stock market daily behavior and card shuffling luck are more influential than any long-term trend."
    Some of the graphs from the article are quite startling in their similarity to different time frames in the stock market (this first one is 25,000 hands played of blackjack for four different "players"):
    four hands.png

    This second one is illustrating the points made near the end of the earlier quote:

    Last edited: Jan 12, 2017
  16. DGH

    DGH Administrator

    Interesting. To continue the analogy, I would point out that a player's edge is determined by: 1. his/her ability to either "count cards" or at least have a heightened sense of "table awareness", assuming only one or two decks are in the shoe. This skill can mitigate, to some degree, the effects of randomness (i.e. luck) and 2. his/her willingness to take the "daily max loss" in order to maintain positive expectancy. Professional blackjack players always know exactly how much they can "safely" lose on any given day in order to prevent blowing up their "account".
    Andrew C likes this.
  17. KiwiDon

    KiwiDon Well-Known Member

    Very true, points 1. and 2. I must admit, I found the authors personal view honest and amusing:

    My personal interest in blackjack began with a 1973 book by John Archer. But I can attest that card counting is boring, error-prone, and not very lucrative. It is nowhere near as glamorous—or as dangerous—as the recent Hollywood film "21" portrays it. And many venues now have machines that continuously shuffle the cards after each hand, making card counting impossible.​

    On a different note, this table of the various counting strategies is fascinating....a lot of work has gone in over the years by many smart people:
  18. DGH

    DGH Administrator

    In the Reno-Tahoe area where we lived for many years (in the 90's) before moving to Santa Cruz some of the smaller casinos would purposely use just one deck and give extra benefits to players (mainly tourists) who usually lost anyway. Larger casinos have their own Monte Carlo simulation programs, of course, and do everything possible to eliminate any player's edge on all casino games. They are also very sophisticated at spotting card counters if fewer decks and non-continuous shuffle play is used. I prefer to "let luck be a lady" with the options game. That's why I like the RTT. I don't consider it gambling, particularly when my expectancy and Sortino ratio can beat those of any card sharp. Plus, I never really got a thrill from blackjack unless there was a lovely lady sitting at the table. That being said, one takeaway point which I consider as important to a successful trader as it is to a professional blackjack player is the ability to take a loss and walk away. A bottle of LAPHROAG helps to ease the sting of a small loss, but there is no healing balm for a blown account.
    Karthnv, Gabor Maly, KiwiDon and 2 others like this.
  19. stevegee58

    stevegee58 Well-Known Member

    The dealers are well trained to spot an "off" bet and will call in the pit boss the second they see one. I was playing BJ using basic strategy (i.e. no counting, all legal) and got busted for doubling down on a weird hand. I had doubled my stake by then so I guess they'd decided I'd made enough money for 1 day. :D

    Traders sometimes bristle at casino analogies but really it's still a useful one. The whole idea is to learn to think in probabilities. This is in contrast to common trading pitfall of worrying about being "right" and "wrong".
    Last edited: Jan 13, 2017
  20. KiwiDon

    KiwiDon Well-Known Member

    Max, I forgot to post a link to this book .... I stumbled upon it while reading up on vol strategies ( ). Might be worth a read for anyone interested in this general topic.

    Here's a sample from the Tony Cooper article:

    The XIV price series is either a random walk without drift (in which case it cannot be traded), a random walk with drift (in which case buy-and-hold is the optimal strategy), or not a random walk (in which case it may be possible to devise a profitable trading rule).

    We tried various random walk statistical tests consisting of variations, with and without drift, of the variance ratio test, the Augmented Dickey-Fuller test, and the Phillips-Perron test. None of these tests could reject a random walk. Therefore the safest approach is to assume that XIV is a random walk and that technical analysis may not work for trading – excess returns may be illusory.

    It is a grim fact of backtesting that every time we optimize a trading parameter by looking at past data, even though that parameter value may be optimal going forward, future returns will generally not be as high as past returns even if market conditions stay the same. That is because when we do the optimization we invariably get some market noise into the return calculation for the optimization. Going forward, the noise will be different and so will no longer contribute to the returns. This is called regression to the mean. Francis Galton demonstrated it clearly when he showed that if you take the tallest fathers in a group (the “optimal” fathers) their sons will not be as tall as the fathers even though those sons will still be the tallest sons on average. For a complete discussion see the chapter “Fool’s Gold” in Aronson (2007).

    So every time we optimize a parameter in a trading rule we put a Grim Reaper icon in the margin to indicate that we expect some reversion to zero in mean returns to follow. We can ameliorate regression to the mean by choosing a “typical” rather than optimal value for the parameter. For example, in the momentum case below we will choose 83 rather than the optimal value of 88. But we still expect some regression to the mean anyway.
    Last edited: Feb 2, 2017

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice