Baseball America released their Top 100 prospects list today and two Tigers made the list. Nick Castellanos, the Tigers expected starting third baseman for 2014, came in at #25 in this year's rankings. Castellanos was ranked #21 last year, #45 in 2012, and #65 in 2011. Devon Travis, the Tigers 23-year-old (as of tomorrow - happy birthday Devon!) #2 prospect who absolutely dominated last season in West Michigan and Lakeland was also featured on the list, and came in at #84.
Baseball America has been ranking the Top 100 prospects in baseball since 1990, and has earned a reputation for being one of the most accurate prospect ranking systems available. Since they've been doing this for so long, we've got a very significant sample size of players and rankings to draw conclusions from and determine how accurate their rankings really are.
I took a look at the career of every player that was featured on the Top 100 list through the first ten years of it's existence. From 1990 to 1999 over 550 players were featured on the list. 96% of those careers are completed and the players who remain active are not likely to change the data much moving forward, so we should be able to draw fair conclusions from this sample. As you might expect, some of the players became stars while others never even saw a major league dugout (Mike Drumright, anyone?). But how good was Baseball America at predicting which was which?
The graph below shows the percentage of players within a segment of the list against the number of WAR those players compiled throughout their careers. For instance, 83% of players who topped out between #1 and #10 on the list went on to produce at least 2 WAR in the major leagues. Similarly, 14% of those players produced at least 50 WAR.
So Baseball America has been "correct" in the sense that, in general, higher ranked players became more productive players. With the exception of a bit of overlap in the low-ranked-players-that-became-good area (blame Javier Vasquez and Kevin Appier for that), each fairly arbitrarily chosen grouping of players performed better than the the groups ranked below them. That alone is pretty impressive to me.
However, the extent to which Baseball America was correct is up for debate. Practically speaking, every player ranked outside of the top 50 is indistinguishable, and even every player outside of the top 20 has about an equal chance of producing 15 or more WAR. There isn't a whole lot of difference between the prospects ranked #25 and those ranked #85. Perhaps there isn't even much difference between #25 and an unranked prospect, but that is a study for another day and another person.
Just for fun, here's another look at the data. Each players career WAR is marked on a scatter plot against their highest ranking.
That really high dot is Alex Rodriguez.
There is clearly some accuracy involved in the upper echelon of Baseball America's rankings. An average player ranked in the top 20 put up 21.6 WAR during their careers (think Jim Northrup or Jeff Weaver). During the 90's Baseball America correctly placed stars such as Rodriguez, Chipper Jones, and Pedro Martinez in the top 10, and Mike Mussina, Scott Rolen, and Roy Halladay in the top 20. Yes, they missed a few too - like Andy Pettitte and Jim Thome, who topped out at #49 and #51 respectively - but overall the number of productive players featured in the top 20 shows some very obvious accuracy at the top of their talent evaluation system.
The Tigers haven't had a top 20 ranked prospect since Cameron Maybin (#6) and Andrew Miller (#10) in 2007. You can attribute this fact to a number of things, but one main component is that the Tigers have only had one top 20 draft pick since 2006.
Castellanos topped out at #21 last year (coincidentally the same ranking that both Rick Porcello and Jacob Turner reached with the Tigers), which places him just outside of the group represented by the red line in the graph above. Since he's right on the border of two groupings, let's take a closer look at that area of the rankings. WARNING: reducing the groupings like this results in some fairly small sample sizes.
As you can see, there is a pretty clear dropoff after the top 15 or 20, not just a gradual decline (the groupings I chose weren't entirely arbitrary). Also, the 26-35 ranked players (orange line) are actually above the 16-25 playesr (red line). This is mostly because of a few outliers in that group, like Kenny Lofton, Frank Thomas, and Jeff Bagwell. Such is life with small sample sizes. But the point here is that there was a big difference between guys ranked 15 and guys ranked 20.
Ultimately, we can probably assume that Castellanos is more similar to the players historically ranked 21-50 than the ones ranked 11-20, but feel free to put him somewhere in the middle. You could use this data to argue that Castellanos has roughly a 50% chance of becoming a 4- or 5-WAR player, and about a 10% chance of becoming a star.
Travis' prospects are surprisingly similar, yet still a bit less optimistic. His current ranking gives him about a 5% chance of becoming a star, and it's less than a coinflip as to whether or not he'll even stick in the majors. But that is really only comparing him to players that topped out in the bottom of the rankings, his ranking could easily improve next year.
But of course I should mention the elephant-sized caveat in the room - this data is based on the rankings from two decades ago. Baseball America undoubtedly uses a different system for their rankings now than they did back then. I'd like to believe that they are more accurate nowadays, but we'll have to wait to find out.
I performed this study for my own benefit. I was curious about what a prospect ranking actually meant, in a practical sense, about a player's odds of success. As with many of these types of endeavors, I probably created more questions than I answered. This is far from a comprehensive analysis, but I think it does shed a little bit of light on what prospect rankings actually mean.
If anyone has any questions I'd be happy to attempt an answer. I'd like to stretch this data as far as possible since I already compiled it. There could very well be more to learn from this project, and I wouldn't mind analyzing it further.