The simple solution is to post your land. You can still offer permission to those that ask.

When you post, remember that you need to post signs at all corners of the property as well as every 400 feet. Additionally signs need to be updated annually and you need to register your land with your town clerk’s office and pay a $5 fee. The town clerk registration needs to happen annually too. It’s an onerous process here in VT, but it is what it is.