How important is this conference to your career? Do you have a very good chance of getting the kind of job you want there?

My advice is -- don't walk! Not unless you have basically a 100% certainty that you will get another job at the conference AND they won't hold you quitting your last job against you.