Are you sure its not just excess skin?
I understand what you say by looking manly or stocky. A lot of people in here don't see athletic thick as stocky but I personally do. If you don't want that, start your resistance/strength training with body weight and resistance bands only. This will prep you for heavier weight lifting anyway if you choose to do so. If you do pick up weights later on your plan will most likely be lifting more days in the week at a moderate weight. You will have to do trial and error to find your perfect weight for your strength level. If your plan moves into wanting to gain a lot of muscle in a scrawny area you are going to have heavier weights with less frequency during the week.
Do only low intensity minimal cardio. Walking is perfect. "Lots of cardio" can backfire on you because it can take lean tissue with the fat if it is intense or if it cuts your calorie deficit too low. Its better to be cutting the excess through dietary control.
I personally would just try to maintain your weight but change your body composition by exercise and diet. Then once you get leaner, start to try to make gains. Some fat can come on with gaining so that could snowball if you just tried to gain now. And the higher fat you are the harder it is for your body to shed fat. Yeh...lovely...i hate that is true. LOL
Use this site for food and exercise guidance: (free)
SparkPeople: SparkDiet
Just wanted to add a little tip that when you make your calorie goals to try to eat no lower than your BMR ever. Fat comes off best when the body is not stressed out.
To speed it a long it is ok to throw a lower cal day or two in the week.
This is zig-zagging:
Calorie Calculator - Daily Caloric Needs
Under the advanced option.
Use those two tips along with spark's site and you will see a difference. Remember its going to take at least 3 months of dedication to see big results. Don't get discouraged the first month or two.
And if you do just go with weights and get too big go down a level. The muscle will shrink up. You won't blow up overnight so you won't be stuck at thick muscles unless you keep training them.
Do not blame your mom for the skinny fat. That comes from a history of poor diet and lack of physical activity. Which...maybe you can blame mom for the bad eating habits passed down.
Good luck!