Simple: while you’re not able to extract carbon fiber’s small performance benefits, you are in an ideal position to experience its many drawbacks—and for normal people, carbon fiber bicycles have only drawbacks.
Probably the biggest drawback to carbon fiber bikes is that they’re like eggs. You know how eggs are almost impossible to break if you squeeze them from the pointy ends, but if you squeeze them any other way you’ll quickly wind up with a handful of yolk? Similarly, while carbon bikes are quite strong when used as designed, what they’re designed for is racing. They’re not designed for crashing, falling off a hitch rack, getting knocked over in the garage by your young children, or any of the other little mishaps that befall normal people’s bicycles as part of the messy business of day-to-day living.