First, you want to make sure you are using the correct size nasal pillows. Nasal pillows should have a secure fit. Using nasal pillows that are too large can flare the nostrils causing irritation. Using nasal pillows that are too small can create friction in the nostrils, also causing irritation. Not using heated humidification can dry your nasal passages, causing discomfort.
Recommended solutions would be using heated humidification, or using a nasal moist gel.