Thoughts on agile and scrum

When I started working with development teams 3 years ago, I started learning all about agile and scrum. I'm currently a product manager working with two dev teams that loosely follow scrum. Since we don't have an official scrum master, I somewhat fill that role too. I've done my best to learn as much as I can about it and so far I'm a big fan. However I feel like most of the things I hear about it are from the scrum and agile community, which means I don't hear too many negatives. What's your experience with these been and if you were in the right position, how would you try to structure people to produce software?