Because we're large. In nature, crocodilians prey on small/mid-size mammals and birds. Humans are larger than their normal prey, and we're also tall, so we appear more massive to them. And most predators actively avoid animals that are too large to be their prey.
That's why it gets so much more dangerous if you're swimming. If a crocodile is above the water, then it can only see your head. And it's just the right size to be its prey.
And if the crocodile is underwater, then it may be even worse. Humans usually look clumsy when swimming, just like an animal in distress. In other words, an easy prey.