Fishing – Trash? Or am I just doing it wrong?

First off – fuck you if you're about to reply "DURR I JUST LET THEM HOP IN MY BOAT" – for some reason they don't like me like that)

So I believe it was before the fishing update I suddenly had 40 fish or so in the oceans in front of my docks…it was a wonderful time…
I should've taken advantage…

Then came the time of overabundant fish just jumping all over the place – every morning I'd have 10 fish or so on my dock, who needs to fish?!
I should've taken advantage….

Now, what seems like the last fishing update has come – and I HATE fishing now.
Trollfish are #2 tier right? I can't pull one in without 2 stamina foods and it being RIGHT on the coast and me just dragging his giant ass up on shore and snatching him.

Fishing skill is only like 11 – but how the hell am I supposed to rank it up when all I can really catch is small basic bitch fish.
Just grind the hell out of it?

Only slightly annoying, not to mention half the time I try to catch fish they just sit there like statues and don't take the fucking bait….
Then again – I seem to have no problem catching Tuna or Coral Cod in the Ocean when I go out….so I have no idea how Fishing works now lol

When does the skill start making a noticeable difference? Or are different fish just harder to catch? (I assumed BF fish would be easier to catch than Ocean)