Legal Question in Landlord & Tenant Law in Oregon

Generally yes. You need to make sure your refusal is actually communicated to your landlord, either in writing sent to your landlord or conspicuously posted on your door. Your denial must be reasonable, so if you're never going to let your landlord inspect the property, your landlord may be able to get a court order allowing an inspection.