You will need at least a modicum of mathematical knowledge to be a petroleum engineer. Estimating oil and gas field potentials is calculus-intensive, although most of that is done with programs these days. Even drilling a well uses a bit of math to determine Time to TD, length and placement of casing, volume of cement, all of the component volumes for frac jobs, etc.
I would suggest that you start learning math now at the level you are comfortable with, and then move into the areas that you aren't comfortable with. Get a tutor if necessary. If you don't have the foundation to build your degree upon, it will be a waste of your time and money.