From an interview with Baseball Prospectus' Nate Silver in the Baltimore Sun's blogging branch:
For the record, Wieters is projected to hit .311/.395/.546 with 31 homers and 100 RBI this year, despite the facts that he has only one year of pro experience and it's all been at Single and Double-A. For reference, that's about as well as Kevin Youkilis or Mark Teixeira did last year, and is easily the most optimistic projection for a rookie player that Baseball Prospectus' PECOTA system has ever made.
I saw this over at Rob Neyer's newly-formatted blog on ESPN.com, and I thought that rather than waste all my thoughts commenting over there, where someone might actually see them, I would lay something out here. This way nobody will ever read it, and it can't fall into the wrong hands, er... eyes. Whatever.
Anyway, I understood some of why PECOTA would suggest that a young player with so little pro experience would play at such a high level in his rookie year. After all, such a feat is not unprecedented. Mike Piazza hit the cover off the ball at AA and AAA in 1992, and then easily won the NL Rookie of the Year in 1993 at age 24, hitting .318/35/112 for the Dodgers.
He was, it should be noted, a year older, and put up his gaudy minor league minor league numbers at a higher level than Wieters did last year, but the analogy is there. Also, as Neyer pointed out, Johnny Bench won an MVP award at age 22, though he did so after having already played a few seasons in the majors, not as a rookie. Joe Torre, Yogi Berra, Carlton Fisk, Ted Simmons, Gary Carter and other catchers have hit very well at age 23 or so, but most of those had some seasoning in MLB before that.
And of course those were all before PECOTA. Back in the day, we used to have to rely on scouts and guesswork to figure out how well someone would perform, but today there is an array of different methods, all scientific in some form or another, to do this job. Bill James, ZiPS, PECOTA, and a half dozen others all have some proprietary methods for guessing at how well players will do in the future, with varying degrees of success.
Once in a while you get to peek inside a little, as when Baseball Prospectus introduced and explained PECOTA a decade ago. For the most part, though, these systems remain hidden, and the specific formulae are always closely guarded secrets. But Silver let the cat out of the bag in this interview, and I can't believe nobody has yet picked up on it:
Signing bonuses??!?
I can understand using players with comparable stats, comparable positions, comparable body types, ages, and etc. to try to figure out what someone might do, but signing bonuses? That essentially means that PECOTA is putting some semblance of trust in what the drafting teams think of their draftees' worth, right?
I'm not saying the teams don't know what they're doing, but there is an inherent danger in this approach. Talent and potential aren't the only thing to go into this process. Teams who have more money to spend can afford to give bigger signing bonuses, though I doubt a structured settlement company would figure their payouts like this.
The Yankees, for example, routinely give their picks more than "slot money" because, well, they're the Yankees. Or because they're trying to convince a player to sign with them instead of going to college, or playing another sport. Similarly, Scott Boras clients get more than anybody else because he's the best agent in the business.
Theoretically then, a Yankees' draftee, who wanted to go to college, affiliated with Scott Boras could easily get a bigger bonus than someone drafted 10-15 spots ahead of him. This, despite the fact that he's presumably not as talented, and yet, somehow, PECOTA would increase his projection because of it.
How far down the draft order does this consideration go? Just the first round? the first two, or five? The whole draft? Probably not. And for how long is this a factor? Just with rookies, or after a year or two of minor league experience?
In any case, most of the time, this is probably irrelevant, as I doubt that PECOTA makes a distinction between a bonus of $220,000 and $250,000, or between $19,000 and $30,000, or whatever. But when you start getting into the millions, well, I have to wonder.
Wieters got $6 million from the Orioles, almost as much as 2005 #1 pick Justin Upton got from the Diamondbacks. These are Upton's PECOTA projections for 2007 and 2008 as well as the performances on which those were based and the actual performances in the following years, for comparison:
2006 Performance: .263/.343/.413 in 501 PA in Class A, at age 18
2007 PECOTA: .255/.318/.413 (12HR, 10 SB, 62 Runs and 53 RBI in 529 PA in MLB)
2007 Actual: .221/.283/.364 in 152 PA in MLB, after hitting .319/.410/.551 in 100+ games at High A and AA
Looks like PECOTA missed high and wide with this one. Upton was only 19 in 2007, but BP thought he would be just a hair below the average NL hitter that year (which was .266/.334/.423, according to Baseball-reference.com). Instead, he hit more like the average NL pinch hitter (.228/.311/.358) which is to say, like a guy who does not deserve to play regularly.
2007 Performance: .319/.410/.551 in 456 PA in A and AA, at age 19
2008 PECOTA: .271/.349/.471 (20 HR, 92 Runs, 78 RBIs, 18 SB in 642 PA in MLB)
2008 Actual: .250/.353/.463 (15 HR, 42 RBIs, 52 Runs, 1 SB in 417 PA in MLB, after 15 decent games in AAA)
PECOTA got pretty close to the mark here, though. Despite the 21-point gap in batting average and the lack of steals, Upton's actual OPS was within 0.004 of his projection, albeit in a lot less playing time, which is difficult to project. So perhaps the signing bonus thing doesn't factor in once they've got a couple of years' experience to use for evaluation purposes.
Other 2007 draftees who got several million in signing bonuses either don't have a PECOTA projection for this year (Andrew Brackman) or have projections that seem pretty reasonable (Mike Moustakas, David Price, Josh Vitters) given what we have seen of them in the minors.
I also checked Jeff Samardzija, the Cubs' pick in 2005 who got a record $7.25 million so that he would pitch instead of playing football. His projection for last year indicated that he would stink very much bad in the majors, which he surprisingly did not, though their projection for him in 2008 says much the same (6.60 ERA).
I can't check everyone, of course, but my suspicion is that the signing bonus thing doesn't play a very big role. I'm just not sure I get why it should play any role at all. Silver's explanation for using the signing bonus is that "it has some predictive value" but of course, generally, the guys who get more money in the draft are the ones the teams think will be good major leaguers anyway, so there ought to be some correlation there.
This winds up as a self-fulfilling prophecy, where the the players who are expected to be good are given a lot of money, and then the players given a lot of money are expected to be good. Seems to me there must be a better way.
TD: I did want to ask you about [Orioles catching prospect Matt] Wieters, because PECOTA seems to have an almost unprecedented crush on him. I was wondering what you made of that?The bold emphasis is mine.
Nate Silver: [...] The Double-A team Wieters was playing for, when you look at park effects and league difficulty, it was a really tough year for hitters in Double-A. And so that gets ratcheted up quite a bit. The Eastern League was very competitive. He did about as well as any player can do down at that level. He's a big guy. That translates pretty well. And we look at the size of a guy's signing bonus because that has some predictive value. The fact that he has a very big pedigree in college and that more often than not, guys who are drafted that high tend to pan out. That combination of things led to a really aggressive forecast where, if he played for a whole year at that level, he could be an MVP contender.
For the record, Wieters is projected to hit .311/.395/.546 with 31 homers and 100 RBI this year, despite the facts that he has only one year of pro experience and it's all been at Single and Double-A. For reference, that's about as well as Kevin Youkilis or Mark Teixeira did last year, and is easily the most optimistic projection for a rookie player that Baseball Prospectus' PECOTA system has ever made.
I saw this over at Rob Neyer's newly-formatted blog on ESPN.com, and I thought that rather than waste all my thoughts commenting over there, where someone might actually see them, I would lay something out here. This way nobody will ever read it, and it can't fall into the wrong hands, er... eyes. Whatever.
Anyway, I understood some of why PECOTA would suggest that a young player with so little pro experience would play at such a high level in his rookie year. After all, such a feat is not unprecedented. Mike Piazza hit the cover off the ball at AA and AAA in 1992, and then easily won the NL Rookie of the Year in 1993 at age 24, hitting .318/35/112 for the Dodgers.
He was, it should be noted, a year older, and put up his gaudy minor league minor league numbers at a higher level than Wieters did last year, but the analogy is there. Also, as Neyer pointed out, Johnny Bench won an MVP award at age 22, though he did so after having already played a few seasons in the majors, not as a rookie. Joe Torre, Yogi Berra, Carlton Fisk, Ted Simmons, Gary Carter and other catchers have hit very well at age 23 or so, but most of those had some seasoning in MLB before that.
And of course those were all before PECOTA. Back in the day, we used to have to rely on scouts and guesswork to figure out how well someone would perform, but today there is an array of different methods, all scientific in some form or another, to do this job. Bill James, ZiPS, PECOTA, and a half dozen others all have some proprietary methods for guessing at how well players will do in the future, with varying degrees of success.
Once in a while you get to peek inside a little, as when Baseball Prospectus introduced and explained PECOTA a decade ago. For the most part, though, these systems remain hidden, and the specific formulae are always closely guarded secrets. But Silver let the cat out of the bag in this interview, and I can't believe nobody has yet picked up on it:
Signing bonuses??!?
I can understand using players with comparable stats, comparable positions, comparable body types, ages, and etc. to try to figure out what someone might do, but signing bonuses? That essentially means that PECOTA is putting some semblance of trust in what the drafting teams think of their draftees' worth, right?
I'm not saying the teams don't know what they're doing, but there is an inherent danger in this approach. Talent and potential aren't the only thing to go into this process. Teams who have more money to spend can afford to give bigger signing bonuses, though I doubt a structured settlement company would figure their payouts like this.
The Yankees, for example, routinely give their picks more than "slot money" because, well, they're the Yankees. Or because they're trying to convince a player to sign with them instead of going to college, or playing another sport. Similarly, Scott Boras clients get more than anybody else because he's the best agent in the business.
Theoretically then, a Yankees' draftee, who wanted to go to college, affiliated with Scott Boras could easily get a bigger bonus than someone drafted 10-15 spots ahead of him. This, despite the fact that he's presumably not as talented, and yet, somehow, PECOTA would increase his projection because of it.
How far down the draft order does this consideration go? Just the first round? the first two, or five? The whole draft? Probably not. And for how long is this a factor? Just with rookies, or after a year or two of minor league experience?
In any case, most of the time, this is probably irrelevant, as I doubt that PECOTA makes a distinction between a bonus of $220,000 and $250,000, or between $19,000 and $30,000, or whatever. But when you start getting into the millions, well, I have to wonder.
Wieters got $6 million from the Orioles, almost as much as 2005 #1 pick Justin Upton got from the Diamondbacks. These are Upton's PECOTA projections for 2007 and 2008 as well as the performances on which those were based and the actual performances in the following years, for comparison:
2006 Performance: .263/.343/.413 in 501 PA in Class A, at age 18
2007 PECOTA: .255/.318/.413 (12HR, 10 SB, 62 Runs and 53 RBI in 529 PA in MLB)
2007 Actual: .221/.283/.364 in 152 PA in MLB, after hitting .319/.410/.551 in 100+ games at High A and AA
Looks like PECOTA missed high and wide with this one. Upton was only 19 in 2007, but BP thought he would be just a hair below the average NL hitter that year (which was .266/.334/.423, according to Baseball-reference.com). Instead, he hit more like the average NL pinch hitter (.228/.311/.358) which is to say, like a guy who does not deserve to play regularly.
2007 Performance: .319/.410/.551 in 456 PA in A and AA, at age 19
2008 PECOTA: .271/.349/.471 (20 HR, 92 Runs, 78 RBIs, 18 SB in 642 PA in MLB)
2008 Actual: .250/.353/.463 (15 HR, 42 RBIs, 52 Runs, 1 SB in 417 PA in MLB, after 15 decent games in AAA)
PECOTA got pretty close to the mark here, though. Despite the 21-point gap in batting average and the lack of steals, Upton's actual OPS was within 0.004 of his projection, albeit in a lot less playing time, which is difficult to project. So perhaps the signing bonus thing doesn't factor in once they've got a couple of years' experience to use for evaluation purposes.
Other 2007 draftees who got several million in signing bonuses either don't have a PECOTA projection for this year (Andrew Brackman) or have projections that seem pretty reasonable (Mike Moustakas, David Price, Josh Vitters) given what we have seen of them in the minors.
I also checked Jeff Samardzija, the Cubs' pick in 2005 who got a record $7.25 million so that he would pitch instead of playing football. His projection for last year indicated that he would stink very much bad in the majors, which he surprisingly did not, though their projection for him in 2008 says much the same (6.60 ERA).
I can't check everyone, of course, but my suspicion is that the signing bonus thing doesn't play a very big role. I'm just not sure I get why it should play any role at all. Silver's explanation for using the signing bonus is that "it has some predictive value" but of course, generally, the guys who get more money in the draft are the ones the teams think will be good major leaguers anyway, so there ought to be some correlation there.
This winds up as a self-fulfilling prophecy, where the the players who are expected to be good are given a lot of money, and then the players given a lot of money are expected to be good. Seems to me there must be a better way.
Posts
1 comment:
I think you have the direction of causation wrong for Boras's effect on bonuses. Instead of granting Boras some superb skill in fleecing owners out of a couple extra million for draftees, Boras may get higher bonuses for draft picks because the most talented players sign with Boras. I mean he's not getting a gigantic bonus for the Bill Pecotas in the draft. His clients are mostly the studs, and that makes a big difference.
Also, while you certainly can find exceptions to the rule that higher bonuses correlate with higher projections or better performance, picking a handful of cases doesn't disprove Silver's contention that bonuses have predictive value. That's like looking at a handful of games where Albert Pujols goes 0 for 4 and concluding he's no good: Invariably those games are going to happen, but we know not to judge based on a few games. What we need to look at is the larger trend based on a much larger sample of picks. That type of investigation is much more solid than looking at a couple of data points. I'd be interested to see what you conclude after looking at more data!
Post a Comment