Second Mortgage and 401(k) Loan Baldwinsville NY

I have thought about taking out $50,000 out of my 401(k) to pay off my second mortgage. My thought behind this is that I will be paying myself the interest rather than the bank. I have calculated what the payment would be and can afford this. Why does this sound too good to be true? You may be able to get a better interest rate, but that's the only good thing about the idea, says the Debt Adviser.